Kimi K3

Overview

Kimi K3 is Moonshot AI's flagship open-weights model, described on its Hugging Face model card as "an open-weight, native multimodal agentic model and our most capable model to date" — a 2.8T-parameter Mixture-of-Experts model built on Kimi Delta Attention (KDA) and Attention Residuals (AttnRes), with native vision and a 1-million-token context window, billed as "the world's first open 3T-class model" (Kimi-K3 model card). Moonshot's own tech blog frames the release the same way: 2.8 trillion parameters, a 1M-token context window, native vision, Stable LatentMoE routing that effectively activates 16 of 896 experts, and roughly a 2.5× improvement in overall scaling efficiency versus Kimi K2 (Kimi K3 Tech Blog).

Moonshot's official blog index dates the Kimi K3 post to 2026-07-16 (Kimi Research & Tech Blog), and the launch post on Moonshot's official X account, published July 16, 2026, announced K3 as live on Kimi.com, Kimi Work, Kimi Code and the Kimi API, with "Open Weights by July 27, 2026" (@Kimi_Moonshot). The tech blog confirms the same commitment: "the full Kimi K3 model weights will be released by July 27, 2026," with further architecture, training and evaluation detail promised in a forthcoming Kimi K3 technical report (Kimi K3 Tech Blog). The Hugging Face repository (moonshotai/Kimi-K3) carries model-card metadata license: other / license_name: kimi-k3, and the card states that "both the code repository and the model weights are released under the Kimi K3 License" — i.e. a custom Moonshot license, not MIT or Apache (Kimi-K3 model card).

Positioning is explicitly agentic and coding-first. The card lists long-horizon coding ("sustains long engineering sessions, navigates massive repositories, and orchestrates terminal tools — from GPU kernel optimization and compiler development to vision-in-the-loop game dev, CAD, and even chip design") and agentic knowledge work (deep research, interactive visualizations, dashboards, motion design and video editing) as headline capabilities, and states that K3 "always has thinking enabled" with a reasoning_effort field supporting low, high and max (default max), plus a preserved-thinking-history requirement for multi-turn and tool-call usage (Kimi-K3 model card). Notable techniques disclosed by Moonshot include KDA and AttnRes, Stable LatentMoE with Quantile Balancing routing, Per-Head Muon optimization, the Sigmoid Tanh Unit (SiTU) activation, and Gated MLA, alongside native MXFP4 quantization-aware training applied from the SFT stage onward (Kimi K3 Tech Blog; Kimi-K3 model card). Serving is recommended on vLLM, SGLang and TokenSpeed, with the hosted API exposed as model id kimi-k3 at platform.kimi.ai (Kimi-K3 model card); vendor API pricing on the tech blog is $3.00/MTok cache-miss input, $0.30/MTok cache-hit input and $15.00/MTok output (Kimi K3 Tech Blog).

Architecture

• Architecture family: Mixture-of-Experts (MoE), autoregressive, native multimodal (text + image) (Kimi-K3 model card)
• Total parameters: 2.8T (Kimi-K3 model card)
• Activated parameters per token: 104B (Kimi-K3 model card)
• Layers: 93 total, of which 1 dense layer (Kimi-K3 model card); num_hidden_layers: 93, first_k_dense_replace: 1 (config.json)
• Attention-layer composition: 69 KDA layers + 24 Gated MLA layers (Kimi-K3 model card). config.json lists full_attn_layers at layers 4, 8, 12, … 88, 92, 93 — i.e. full (Gated MLA) attention roughly every 4th layer, with the remaining 69 indices in kda_layers (config.json)
• Attention mechanism: KDA (Kimi Delta Attention, hybrid linear attention) & Gated MLA; MLA configured with mla_use_nope: true, mla_use_output_gate: true, q_lora_rank: 1536, kv_lora_rank: 512, qk_nope_head_dim: 128, qk_rope_head_dim: 64, v_head_dim: 128 (Kimi-K3 model card; config.json)
• KDA configuration: 96 heads, head_dim: 128, short_conv_kernel_size: 4, gate_lower_bound: -5.0, use_full_rank_gate: true (config.json)
• Attention Residuals (AttnRes): attn_res_block_size: 12 (config.json); described by Moonshot as selectively retrieving representations across model depth rather than accumulating them uniformly, and as delivering ~25% higher training efficiency at <2% additional cost (Kimi K3 Tech Blog; @Kimi_Moonshot)
• Attention hidden dimension / heads: 7168 hidden, 96 attention heads (Kimi-K3 model card); hidden_size: 7168, num_attention_heads: 96, num_key_value_heads: 96 (config.json)
• Experts: 896 routed experts, 16 selected per token, 2 shared experts (Kimi-K3 model card); num_experts: 896, num_experts_per_token: 16, num_shared_experts: 2, moe_layer_freq: 1 (config.json)
• LatentMoE dimensions: Latent MoE dimension 3584, MoE hidden dimension per expert 3072 (Kimi-K3 model card); routed_expert_hidden_size: 3584, moe_intermediate_size: 3072, latent_moe_use_norm: true (config.json)
• Routing: Stable LatentMoE framework; sigmoid router with noaux_tc top-k method and renormalization (moe_router_activation_func: sigmoid, topk_method: noaux_tc, moe_renormalize: true) (config.json); Moonshot adds Quantile Balancing, which derives expert allocation from router-score quantiles and removes heuristic updates and the balancing hyperparameter (Kimi K3 Tech Blog)
• Activation function: SiTU-GLU per the model card (Kimi-K3 model card); hidden_act: "situ" with activation_situ_beta: 4.0 and activation_situ_linear_beta: 25.0 (config.json). The tech blog names the component Sigmoid Tanh Unit (SiTU) (Kimi K3 Tech Blog)
• Dense FFN intermediate size: 33792 (config.json)
• Context window: 1,048,576 tokens (Kimi-K3 model card); max_position_embeddings: 1048576 (config.json)
• Vocabulary: 160K per the card; vocab_size: 163840 in config (Kimi-K3 model card; config.json)
• Vision encoder: MoonViT-V2, 401M parameters (Kimi-K3 model card); vision tower config: 27 layers, 12 heads, vt_hidden_size: 1024, vt_intermediate_size: 4096, patch_size: 14, mm_projector_type: patchmergerv2 (config.json)
• Precision / quantization: MXFP4 weights with MXFP8 activations, via quantization-aware training from the SFT stage onward (Kimi-K3 model card); config declares format: mxfp4-pack-quantized, num_bits: 4, group_size: 32, quant_method: compressed-tensors, with attention, shared experts, dense MLP, lm_head, vision tower and mm projector excluded from quantization (config.json)
• Model class / implementation: KimiK3ForConditionalGeneration over a KimiLinearForCausalLM text backbone (model_type: kimi_linear) (config.json)
• Multi-token prediction: none configured (num_nextn_predict_layers: 0) (config.json)
• Optimizer technique: Per-Head Muon, extending Muon by optimizing attention heads independently (Kimi K3 Tech Blog)
• Decoding efficiency claim: KDA enables up to 6.3× faster decoding in million-token contexts (@Kimi_Moonshot)
• Recommended deployment scale: supernode configurations with 64 or more accelerators; vLLM, SGLang and TokenSpeed recipes (Kimi K3 Tech Blog; Kimi-K3 model card)
• Modality: Text + Image input per the card's model summary; the introduction also states K3 "understands text, images, and video within the same model" (Kimi-K3 model card)

Official vendor eval scores

All values are the Kimi K3 (max reasoning effort, temperature 1.0) column of Moonshot's own model-card evaluation table (Kimi-K3 model card).

Benchmark explanations

• GPQA Diamond — hardest subset of graduate-level, Google-proof multiple-choice science questions in physics, chemistry and biology; measures expert-level scientific reasoning.
• HLE-Full — Humanity's Last Exam, full set (text and image questions): very hard expert-written questions across many disciplines; Moonshot reports scores without and with tool augmentation (Kimi-K3 model card).
• AA-LCR — Artificial Analysis Long Context Reasoning: reasoning over very long inputs; K3's figure is cited from Artificial Analysis as of July 23, 2026 (Kimi-K3 model card).
• CritPt — hard research-level physics reasoning benchmark; K3's figure is cited from Artificial Analysis as of July 23, 2026 (Kimi-K3 model card).
• Terminal-Bench 2.1 — end-to-end terminal/command-line agent tasks: can the model drive a shell to complete real system and engineering jobs.
• DeepSWE — agentic software-engineering task suite (v1.1 tasks) measuring repository-level issue resolution with an agent harness (Kimi-K3 model card).
• ProgramBench — asks a coding agent to recreate a program's behavior from only a compiled binary plus documentation, judged against fuzz-generated behavioral tests (Kimi-K2.7-Code model card).
• FrontierSWE — frontier-difficulty software-engineering tasks scored by a dominance metric recomputed with the official evaluation script (Kimi-K3 model card).
• SWE-Marathon — very long-horizon software-engineering tasks, including GPU tasks with performance gates and anti-cheat validators (Kimi-K3 model card).
• Kimi Code Bench 2.0 — Moonshot's in-house coding-agent benchmark spanning realistic engineering tasks across 10+ languages and a full production stack, including cybersecurity/safety tasks (Kimi-K3 model card; Kimi-K2.7-Code model card).
• MLS-Bench-Lite — 30-task official subset of MLS-Bench, testing whether an agent can invent generalizable, scalable ML methods within a 5-hour budget (Kimi-K2.7-Code model card).
• SciCode — scientific code-generation tasks derived from research problems; K3's figure is cited from Artificial Analysis (Kimi-K3 model card).
• BrowseComp — hard web-browsing/search agent benchmark: finding hard-to-locate facts online across many tool calls.
• DeepSearchQA (F1) — deep research question answering with search tools, scored by F1 against reference answers.
• MCPMark-Verified — human-verified edition of MCPMark, testing MCP tool use across Notion, GitHub, Filesystem, Postgres and Playwright servers (Kimi-K2.7-Code model card).
• MCP-Atlas — realistic tool-use tasks over scalable MCP servers, run here on the 500-task public subset with a 100-turn limit (Kimi-K3 model card).
• OSWorld-Verified — computer-use agent benchmark: completing real desktop/OS tasks through GUI interaction.
• τ³-Banking — tool-agent benchmark in a banking domain testing multi-turn policy-following and tool calls; K3's figure is cited from Artificial Analysis (Kimi-K3 model card).
• GDPval-AA v2 (Elo) — Artificial Analysis Elo rating for economically valuable, real-world professional work tasks (Kimi-K3 model card).
• MMMU-Pro — massive multi-discipline multimodal understanding, "Pro" (harder) version: college-level reasoning over images.
• Video-MME — video understanding across short, medium and long videos; reported here with subtitles.
• OmniDocBench — document parsing/OCR accuracy, scored as (1 − normalized Levenshtein distance) × 100 (Kimi-K2.5 model card).
• SWE-bench Verified — human-validated subset of real GitHub issues requiring a patch that passes hidden tests (not reported for K3).
• LiveCodeBench — contamination-resistant competitive-programming benchmark using recently published contest problems (not reported for K3).
• AIME / HMMT — high-school olympiad-level competition math exams (not reported for K3).
• MMLU-Pro — harder, reasoning-focused version of MMLU multiple-choice knowledge testing (not reported for K3).
• tau2-bench — customer-service-style tool-use agent benchmark with domain policies (not reported for K3).