152. 领读Kimi K3技术报告:从架构创新聊起,注意力美学、多教师蒸馏和开源MoE
152. 领读Kimi K3技术报告:从架构创新聊起,注意力美学、多教师蒸馏和开源MoE
Podcast2 hr 4 min
Listen to Episode
Note: AI-generated summary based on third-party content. Not financial advice. Read more.
Quick Insights

Sustained multi-trillion parameter model scaling reinforces strong demand for NVIDIA Corporation (NVDA) and its next-generation Blackwell architecture, solidifying the chipmaker's long-term competitive moat. Investors should also look at Xiaomi Corporation (1810.HK), which is well-positioned to commercialize consumer-facing AI by achieving ultra-fast inference speeds across its vast smartphone and smart device ecosystem. Downstream AI software builders present an attractive margin-expansion opportunity as low-cost structural innovations from DeepSeek continue to drive API pricing lower. Meanwhile, monitor the progress of frontier models like Moonshot AI (Kimi), as their technical breakthroughs serve as a key leading indicator for high-margin Enterprise AI adoption.

Detailed Analysis

Moonshot AI / Kimi (Private)

  • Successfully launched the Kimi K3 model, expanding to 2.8T total parameters with over 100B activated parameters and a 1M token context window.
    • Achieved effective model scaling, demonstrating that scaling total parameter size remains the foundational driver of true model intelligence.
    • Introduced key architectural innovations including Hybrid Delta Attention, enhanced Multi-Head Latent Attention (MLA), and Latent MoE (Mixture of Experts) to lower communication overhead while maintaining stability.
    • Positions its commercial offerings at a premium price point, focusing on top-tier frontier capability rather than competing purely on low API prices.

Takeaways

  • Kimi represents the high-end frontier scaling path among Chinese AI labs, making its technical breakthroughs a key indicator for high-margin enterprise AI adoption.
  • Investors evaluating AI software and infrastructure should monitor Kimi's architecture choices, as their hybrid and latent MoE designs set new efficiency benchmarks for multi-trillion parameter models.

DeepSeek (Private)

  • Featured as the leading counter-strategy to high-parameter scaling, emphasizing extreme cost efficiency and high-throughput inference.
    • Evolving from DeepSeek-V3 toward DeepSeek-V4, shifting structural focus toward Sparse Attention and Multi-Token Prediction (MTP) to optimize memory and compute balance for agent-driven tasks.
    • Continues to drive structural innovations like Dual-Pipe communication overlap and Distributed Pipeline Parallelism (DPEP), setting the benchmark for low-cost open-source foundation models.

Takeaways

  • DeepSeek's architectural paradigm exerts ongoing downward price pressure on commercial AI APIs, making it a critical driver of AI application affordability.
  • Serves as the industry benchmark for low-cost inference, benefiting downstream software builders seeking cost-effective API integration.

NVIDIA Corporation (NVDA)

  • Highlighted regarding the interaction between advanced frontier AI architectures and next-generation Blackwell GPU hardware.
    • Advanced hardware architectures like Blackwell face unique overheads in sparse indexing, making kernel-level hardware-software co-design essential for unlocking genuine acceleration.
    • Multi-trillion parameter training across distributed clusters requires specialized communication frameworks and GPU memory optimization (such as FP8 offloading and custom chunk algorithms) to prevent idle hardware waste.

Takeaways

  • Frontier model scaling to the 2.8T+ parameter range continues to drive massive demand for high-bandwidth compute and advanced interconnects, supporting sustained demand for NVDA hardware.
  • The increasing necessity of deep kernel optimization and infrastructure co-design reinforces NVIDIA's software and hardware competitive moat.

Xiaomi Corporation (1810.HK)

  • Mentioned for significant engineering breakthroughs in model inference speed, achieving speeds up to 1,000 TPS (tokens per second).
    • Integrated TensorRT optimizations with draft models and D-Flash speculative decoding to drastically reduce latency in small-batch and user-facing workloads.

Takeaways

  • Xiaomi's ability to achieve ultra-fast inference speeds positions it well to deploy responsive, low-latency AI features across its vast smartphone and AIoT hardware ecosystem.
  • Highlights growing domestic capability in production-level inference acceleration, improving the commercial viability of edge and consumer AI services.
Ask about this postAnswers are grounded in this post's content.
Episode Description
今天的节目是一集学习播客,学习Kimi K3技术报告,希望和大家一起领略“技术之美”。 K3是有效扩展到2.8T总参数并全量开源的MoE模型。大家可能注意到,这集技术报告的领读距离模型发布已经过去一段时间。期间,我们试图寻找一位合适的嘉宾,我们希望这位嘉宾的学术和工作背景非常适合讲K3。最后找到孙宇涛。 宇涛目前是清华大学计算机系博士候选人,上海创智学院璞锐学者。他从博士开始的研究方向是LLM架构、预训练,架构创新一直是他的兴趣点,这正好是K3亮点之一。 宇涛透过领读K3技术报告也串联讲解了十多篇相关论文。他的语速非常快——前方语速预警。 OUTLINE: 02:00 宇涛的自我介绍和研究经历,为什么对架构创新最感兴趣? 16:05 从Kimi K3论文出发,论文讲解的框架与脉络 19:02 宇涛开始领读论文: 1 导读 Kimi K3将设计概括为沿sequence、depth和width三个维度扩展信息流;其中depth与width本质上仍是模型容量扩展的两种组织⽅式,这⼀“三维 scaling”更多是贯穿论⽂的叙事框架。 2 Model Architecture 2.1 线性注意⼒的前世今⽣ [Microsoft Research] RetNet:最初的 data-independent decay 与 chunk-recurrent 递归形式 [NVIDIA] Gated DeltaNet:引⼊ gated delta rule [Moonshot AI] Kimi Linear:fine-grained decay及其带来的infra变化 2.2 Gated MLA [Alibaba Qwen] Gated Attention for Large Language Models:attention gating与训练稳定性 2.3 Attention Residuals [Microsoft Research] On Layer Normalization in the Transformer Architecture:Pre-LN、Post-LN与训练稳定性 [ByteDance Seed] Hyper-Connections:残差连接的新维度 [Moonshot AI] Attention Residuals:以跨深度attention取代固定残差累积,使各层选择性聚合此前表⽰ 2.4 Stable LatentMoE与SiTU-GLU [NVIDIA] LatentMoE:通过latent expert space降低MoE通信与权重访问成本 [OpenAI] GPT-OSS:以clamped SwiGLU控制激活值 2.5 Muon [Moonshot AI] Moonlight / Muon is Scalable for LLM Training:更好的优化表现及随之突出的 activation outlier问题 2.6 Quantile Balancing [科学空间] 《MoE环游记:6、最优分配促均衡》:从最优分配推导Quantile Balancing 2.7 Native Vision 3 Pre-Training 3.1 Scaling Law [OpenBMB] MiniCPM:WSD的训练预算扩展、稳定阶段复⽤、继续训练与提前停⽌ 3.2 Long-Context Extension [Cohere] RNoPE:交替使⽤RoPE与NoPE,兼顾位置建模与长上下⽂检索 4 Post-Training 4.1 Post-Training Pipeline 4.2 Reinforcement Learning [Moonshot AI] Kimi K1.5:提出partial rollout,通过复⽤未完成轨迹降低长CoT rollout开销 [Moonshot AI] Kimi K2.5:reasoning-effort budget control 与 Agentic Generative Reward Model 4.3 Multi-Teacher On-Policy Distillation [Microsoft Research] MiniLLM:基于student-generated samples的on-policy distillation 4.4 Deployment-Aware Post-Training 4.5 Draft Model Fine-Tuning 5 Infrastructure 5.1 Pre-Training 5.1.1 KDA Kernel [FLA] Flash Linear Attention:KDA kernel与线性注意⼒⾼效实现 5.1.2 Distributed Training [DeepSeek-AI] DeepSeek-V3 Technical Report:DualPipe与细粒度MoE communication–computation overlap MoE overlap的资源trade-off与cross-PP activation transfer 5.1.3 Perfectly Balanced Expert-Parallel MoE Training 5.1.4 Memory-Efficient Training [Moonshot AI] Mooncake:Mooncake Transfer Engine与cross-PP activation remote offload 5.1.5 Multimodal Encoder Optimization 5.2 RL 5.2.1 Long-Context RL Infrastructure 5.2.2 Sandbox Infrastructure 5.3 Inference 5.3.1 KDA-Aware Prefix Cache Management 5.3.2 High-Performance Kernels 5.3.3 Fleet-Level Scheduling LINKS: 我们的播客在小宇宙、Apple Podcast、Spotify等全音频平台播出; 我们的视频播客在小宇宙、Bilibili、小红书、视频号、抖音等全视频平台播出; 如果你想服用文字版,请搜索我们工作室的公众号:语言即世界language is world。 DISCLAIMER: 本内容不作为投资建议。 CONTACT: xiaojunzhang@lisw.ai Jump into the new world-and explore with us!😉
About 张小珺Jùn|商业访谈录
张小珺Jùn|商业访谈录

张小珺Jùn|商业访谈录

By 张小珺

努力做中国最优质的科技、商业访谈。 张小珺:财经作者,写作中国商业深度报道,范围包括AI、科技巨头、风险投资和知名人物,也是播客《张小珺Jùn | 商业访谈录》制作人。 如果我的访谈能陪你走一段孤独的未知的路,也许有一天可以离目的地更近一点,我就很温暖:)