DeepSeek V4 Pro

Overview

DeepSeek-V4-Pro is the flagship open-weights Mixture-of-Experts (MoE) language model of the DeepSeek-V4 series from DeepSeek-AI, with 1.6T total parameters and 49B activated per token and a native 1M-token context length (DeepSeek-V4-Pro model card). It was announced as "DeepSeek-V4 Preview … officially live & open-sourced" alongside the smaller DeepSeek-V4-Flash (284B total / 13B active) on DeepSeek's own API news page dated April 24, 2026, which also states that 1M context is the default across all official DeepSeek services and that both models support dual Thinking / Non-Thinking modes (DeepSeek V4 Preview Release). The accompanying technical report, "DeepSeek-V4: Towards Highly Efficient Million-Token …", was submitted to arXiv on 26 Apr 2026 and presents the two MoE models as a preview of the V4 series (arXiv:2606.19348). The weights are released under the MIT License (DeepSeek-V4-Pro model card).

Positioning is explicitly agentic and coding-first: DeepSeek describes V4-Pro as having enhanced agentic capabilities, as open-source SOTA on agentic-coding benchmarks, as leading all current open models while trailing only Gemini-3.1-Pro, and as being used for in-house agentic coding at DeepSeek, with integrations for Claude Code, OpenClaw and OpenCode (DeepSeek V4 Preview Release). The maximum reasoning-effort mode, DeepSeek-V4-Pro-Max, is described in the technical report as redefining the state of the art for open models (arXiv:2606.19348).

The headline techniques are efficiency-oriented. V4-Pro uses a hybrid attention architecture combining Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA), Manifold-Constrained Hyper-Connections (mHC) in place of conventional residual connections, and the Muon optimizer, pre-trained on more than 32T tokens; in the 1M-token setting it needs 27% of the single-token inference FLOPs and 10% of the KV cache of DeepSeek-V3.2 (DeepSeek-V4-Pro model card, arXiv:2606.19348). A production checkpoint, DeepSeek-V4-Pro-0813, superseded the preview: it is built on the preview structure with a DSpark speculative decoding module attached and reports 1.7T params, with "greatly enhanced agentic capabilities" (DeepSeek-V4-Pro-0813 model card); DeepSeek's changelog dated 2026-08-13 states the GA release rolled out on app, web and API with low/high/max reasoning effort and native OpenAI Responses API support (DeepSeek-V4-Pro GA Release, DeepSeek API Change Log).

Architecture

• Total parameters: 1.6T (preview checkpoint) (model card); the GA card lists 1.7T params for DeepSeek-V4-Pro-0813 (0813 model card)
• Activated parameters per token: 49B (model card, arXiv:2606.19348)
• Architecture family: Mixture-of-Experts decoder LM, DeepseekV4ForCausalLM, model_type: deepseek_v4 (config.json)
• Layers: num_hidden_layers: 61, plus num_nextn_predict_layers: 1 (MTP block) (config.json); the 61-layer stack is described as layers 0–1 HCA and layers 2–60 alternating CSA/HCA, with the final MTP block using sliding-window attention only (Hugging Face DeepSeek-V4 blog)
• Attention: hybrid architecture combining Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA) (model card, arXiv:2606.19348)
• CSA details: compresses KV 4× along the sequence, softmax-gated pooling with learned positional bias, an FP4 "lightning indexer" selecting top-k compressed blocks, plus a sliding-window branch for recent tokens (Hugging Face DeepSeek-V4 blog)
• HCA details: compresses KV 128×, dense (no sparse selection) attention over compressed blocks with a sliding-window recency branch (Hugging Face DeepSeek-V4 blog); the config's compress_ratios list alternates 128 and 4 across layers, with index_topk: 1024, index_n_heads: 64, index_head_dim: 128, sliding_window: 128 (config.json)
• Attention shape: num_attention_heads: 128, num_key_value_heads: 1, head_dim: 512, q_lora_rank: 1536, o_lora_rank: 1024, qk_rope_head_dim: 64, hidden_size: 7168 (config.json)
• MoE experts: n_routed_experts: 384, n_shared_experts: 1, num_experts_per_tok: 6, moe_intermediate_size: 3072, routing scoring_func: sqrtsoftplus with topk_method: noaux_tc (config.json); the model card itself does not state expert counts (model card)
• Residual connections: replaced by Manifold-Constrained Hyper-Connections (mHC) (model card, arXiv:2606.19348); config exposes hc_mult: 4, hc_sinkhorn_iters: 20, hc_eps: 1e-06, num_hash_layers: 3 (config.json)
• Context window: 1M tokens; max_position_embeddings: 1048576 with YaRN rope scaling factor 16 over original_max_position_embeddings: 65536 (model card, config.json)
• Precision / quantization: distributed as FP4 + FP8 Mixed — MoE expert parameters in FP4, most other parameters in FP8 (Base checkpoints are FP8 Mixed); HF tensor types listed as BF16, I64, F32, F8_E4M3, I8 (model card). Config shows quant_method: fp8, fmt: e4m3, scale_fmt: ue8m0, weight_block_size: [128,128], and the 0813 config adds expert_dtype: fp4 (config.json, 0813 config.json)
• KV/FLOP efficiency: at 1M context, 27% of the single-token inference FLOPs and 10% of the KV cache of DeepSeek-V3.2 (model card, arXiv:2606.19348)
• Optimizer and pre-training data: Muon optimizer; more than 32T diverse, high-quality tokens (model card, arXiv:2606.19348)
• Vocabulary: vocab_size: 129280, hidden_act: silu, swiglu_limit: 10.0 (config.json)
• Speculative decoding (GA checkpoint only): DSpark module attached, dspark_target_layer_ids: [58,59,60], dspark_block_size: 5, dspark_markov_rank: 512 (0813 model card, 0813 config.json)
• License: MIT (model card, 0813 model card)
• Modalities: text in / text out; the preview API announcement lists no image/audio input for Pro (a separate experimental deepseek-v4-flash-vision-exp accepts images) (DeepSeek API docs)

Official vendor eval scores

All rows below are vendor-reported. Unless noted, values are DeepSeek-V4-Pro-Max (maximum reasoning effort) from the official Hugging Face model card.

Additional vendor-reported values on the same cards, for reference: Apex 38.3, Apex Shortlist 90.2, SimpleQA-Verified 57.9, Chinese-SimpleQA 84.4, CorpusQA 1M 62.0, MCPAtlas Public 73.6, GDPval-AA Elo 1554 (HF model card); and for V4-Pro-0813: NL2Repo 61.5, Cybergym 83.3, Agents' Last Exam 25.7, AutomationBench (Public) 31.8, plus internal DSBench-FullStack 71.1 and DSBench-Hard 67.2 (0813 model card).

Benchmark explanations

• MMLU-Pro — a harder, cleaned multiple-choice exam suite across many academic and professional subjects, measuring broad knowledge and reasoning; scored as exact match (HF model card).
• GPQA Diamond — the hardest subset of graduate-level, "Google-proof" physics, chemistry and biology questions written by domain PhDs; measures expert scientific reasoning (HF model card).
• HLE (Humanity's Last Exam) — an extremely difficult, expert-authored exam spanning many fields, used as a frontier-difficulty knowledge and reasoning ceiling; DeepSeek also reports a tool-augmented variant (HF model card).
• LiveCodeBench — competitive-programming problems collected continuously from live contests to limit contamination; measures code generation correctness at pass@1 (HF model card).
• Codeforces — an Elo-style rating estimated from competitive-programming contest performance rather than a percentage accuracy (HF model card).
• HMMT 2026 Feb — the February 2026 Harvard-MIT Math Tournament problem set, a hard olympiad-style short-answer math benchmark (HF model card).
• IMOAnswerBench — International Mathematical Olympiad-style problems graded on final answers, measuring olympiad math ability (HF model card).
• SWE-bench Verified — human-validated real GitHub issues where the model must produce a patch that passes the repository's tests; measures agentic software-engineering ability (HF model card).
• SWE-bench Pro — a harder, more realistic variant of the SWE-bench issue-resolution task (HF model card).
• SWE-bench Multilingual — the same issue-resolution setup extended beyond Python to repositories in multiple programming languages (HF model card).
• Terminal-Bench 2.0 / 2.1 — multi-step shell and systems tasks executed in a real terminal environment, measuring end-to-end agentic tool use (HF model card, 0813 model card).
• BrowseComp — hard-to-find information-seeking questions that require an agent to browse the web persistently to locate the answer (HF model card).
• Toolathlon / Toolathlon-Verified — broad tool-calling agent tasks across many tools and APIs; the Verified split is a human-checked subset (HF model card, 0813 model card).
• MRCR 1M — multi-round coreference resolution at one-million-token context, measuring long-context retrieval and tracking (HF model card).
• DeepSWE — a software-engineering agent evaluation reported by DeepSeek for the 0813 production checkpoint (0813 model card).
• AIME — the American Invitational Mathematics Examination, a short-answer competition math benchmark; not reported by DeepSeek for V4-Pro (HF model card).
• tau2-bench — a customer-service-style tool-use and policy-following agent benchmark; not reported by DeepSeek for V4-Pro (HF model card).