DeepSeek AI 发布 DeepSeek-V4.1-Flash：1M 上下文、FP4 KV 缓存与跨层注意力复用

文章拆解了 KV 缓存压缩和预精简架构的具体做法，读者可以对照其工程路径评估长上下文部署成本。

DeepSeek AI 发布多模态 MoE 模型 DeepSeek-V4.1-Flash，552B 主干加 196B Engram 参数，上下文窗口 1M，prefill 激活 8B 参数、decode 激活 16B，全局 KV 缓存降至每 token 890 字节，约为 DeepSeek-V4-Flash 的 1/4、DeepSeek-V1 的 1/437。

长时程智能体已经把 LLM 服务变成了一种输入密集型负载。反复的预填充和百万级 token 上下文留下的 KV cache 给 HBM、SSD 容量和带宽带来了巨大压力。DeepSeek AI围绕这一瓶颈打造了其最新发布。DeepSeek-V4.1-Flash是一款多模态混合专家模型，拥有 552B 骨干参数、196B 额外 Engram 参数，以及 1M token 上下文窗口。它在预填充阶段每个 token 激活 8B 参数，在解码阶段激活 16B。核心数字是每 token 全局 KV cache 占用仅 890 字节，约为 DeepSeek-V4-Flash 的 1/4，比 DeepSeek-V1 小约 437 倍。

它能部署吗？能。开放权重以MIT 许可证发布，在 Hugging Face 上提供 vLLM、SGLang 和 Transformers 路径，研究团队还描述了一个公开 API，设有低、高、最高三档推理层级。

40 层骨干被拆分为 20 层因果编码器和 20 层解码器。受YOCO启发，解码器不计算自身的全局 KV。相反，逐层投影权重从编码器最终隐藏状态中推导出它。因此提示词 token 止步于编码器，这几乎将预填充计算量减半。128 token 窗口的滑动窗口注意力（SWA）仍在每一层运行，因此解码器 SWA 状态只需重放最后 128 个提示词 token 即可重建。研究团队称之为解码器 SWA 有界重放。

DeepSeek-V4 将 CSA 与高度压缩注意力（Heavily Compressed Attention）混合使用。V4.1-Flash 采用纯 CSA2，并沿层轴压缩缓存大小。每个 CSA2 层被静态分配为以下 3 种模式之一：

每一层都保留自己的主 Q 和 SWA KV。18 个 CSA2 编码器层采用压缩比 2，分为 3 组、每组 6 层（1 个 Full，5 个 Reuse）。20 个解码器层采用压缩比 1，分为 5 组、每组 4 层：第一组为 Full 加 3 个 Reuse，其余为 Reindex 加 3 个 Reuse。解码器中的分层稀疏索引器（Hierarchical Sparse Indexer）让 Full 层构建一个最多 16,384 个位置（2,048 个块，每块 8 个）的候选池，从而使后续的 Reindex 层只需对有限集合打分，而非整个上下文。

主 KV 缓存被量化为 E2M1，每 16 个通道配一个 E4M3 缩放因子，遵循 NVFP4 但不含其全局缩放因子。这通过后训练阶段的量化感知训练引入，相比 V4 的 FP8 缓存几乎将存储减半。

在部署层面，SWA KV 不再持久化到 SSD。它驻留在一个分布式池中，该池从主机 DRAM 的 10% 中划分而来，TTL 为分钟级，而全局 KV 则保证 72 小时的生命周期。未命中时，Encoder SWA Bounded Replay 只重新计算 128 个 token，而非层数乘以窗口大小。

其他改动包括 Single-Pass mHC，它将输入混合系数偏移一个 block，使融合的 Mega-mHC kernel 能将激活内存流量减半；位于第 1 层和第 14 层的 Engram 条件记忆模块；在预训练后以冻结主干训练的 DSpark 投机解码；以及 head-wise Muon。当上下文从 4K 增长到 1M 时，单 token 解码 FLOPs 仅增加 1/4。

预训练覆盖 45T 多模态 token，文本与多模态比例为 7:1。稀疏注意力从零开始以 64K 序列长度训练，无稠密预热，并在 34T token 时将上下文扩展至 1M。基础模型在世界知识和编程方面与 DeepSeek-V4-Pro-Base 相当，而总参数量仅为其 1/3，激活参数量仅为其 1/4。

后训练未引入新算法。收益来自大规模合成可验证的智能体任务、跨异构脚手架（Claude Code、Codex、OpenCode、Pi、mini-SWE、DeepSeek Harness）的 RL，以及来自 40 多个教师模型的 on-policy 蒸馏。精选的最大努力结果如下：

Long-horizon agents have turned LLM serving into an input-heavy workload. Repeated prefills and million-token contexts leave KV caches that strain HBM, SSD capacity, and bandwidth. DeepSeek AI built its newest release around that exact bottleneck. DeepSeek-V4.1-Flash is a multimodal Mixture-of-Experts model with 552B backbone parameters, 196B additional Engram parameters, and a 1M-token context window. It activates 8B parameters per token during prefill and 16B during decode. The main number is a global KV cache footprint of 890 bytes per token, about 1/4 of DeepSeek-V4-Flash and roughly 437x smaller than DeepSeek-V1.

Is it deployable? Yes. Open weights ship under an MIT license with vLLM, SGLang, and Transformers paths on Hugging Face, and the research team describes a public API with low, high, and max reasoning tiers.

The 40-layer backbone is split into a 20-layer causal encoder and a 20-layer decoder. Inspired by YOCO, the decoder does not compute its own global KV. Instead, per-layer projection weights derive it from the final encoder hidden state. Prompt tokens therefore stop at the encoder, which nearly halves prefill compute. Sliding-window attention (SWA) with a 128-token window still runs in every layer, so decoder SWA states are rebuilt by replaying only the last 128 prompt tokens. The research team calls this Decoder SWA Bounded Replay.

DeepSeek-V4 mixed CSA with Heavily Compressed Attention. V4.1-Flash uses pure CSA2 and attacks cache size along the layer axis. Each CSA2 layer is statically assigned one of 3 modes:

Every layer keeps its own main Q and SWA KV. The 18 CSA2 encoder layers use a compression ratio of 2 in 3 groups of 6 (1 Full, 5 Reuse). The 20 decoder layers use ratio 1 in 5 groups of 4: the first is Full plus 3 Reuse, the rest Reindex plus 3 Reuse. A Hierarchical Sparse Indexer in the decoder lets the Full layer build a candidate pool of up to 16,384 positions (2,048 blocks of 8), so later Reindex layers score a bounded set instead of the entire context.

The main KV cache is quantized to E2M1 with one E4M3 scale per 16 channels, following NVFP4 without its global scale. This is introduced through quantization-aware training in post-training and nearly halves storage against V4’s FP8 cache.

At the deployment level, SWA KV is no longer persisted to SSD. It lives in a distributed pool carved from 10% of host DRAM with a TTL of minutes, while global KV keeps a guaranteed 72-hour lifetime. On a miss, Encoder SWA Bounded Replay recomputes only 128 tokens instead of layers times window.

Other changes include Single-Pass mHC, which shifts input-mixing coefficients by one block so a fused Mega-mHC kernel can halve activation memory traffic, the Engram conditional memory module at layers 1 and 14, DSpark speculative decoding trained after pre-training with the backbone frozen, and head-wise Muon. Single-token decode FLOPs rise by only 1/4 when context grows from 4K to 1M.

Pre-training covers 45T multimodal tokens at a 7:1 text-to-multimodal ratio. Sparse attention is trained from scratch at 64K sequence length with no dense warmup, and context is extended to 1M at 34T tokens. The base model matches DeepSeek-V4-Pro-Base on world knowledge and coding while using 1/3 of the total and 1/4 of the activated parameters.

Post-training introduces no new algorithms. Gains come from large-scale synthesis of verifiable agent tasks, RL across heterogeneous scaffolds (Claude Code, Codex, OpenCode, Pi, mini-SWE, DeepSeek Harness), and on-policy distillation from over 40 teachers. Selected max-effort results:

来源：MarkTechPost（RSS）· marktechpost.com