gpt-oss-120b

Overview

gpt-oss-120b is an open-weight reasoning language model released by OpenAI on August 5, 2025, alongside the smaller gpt-oss-20b, both distributed under the Apache 2.0 license (OpenAI, "Introducing gpt-oss"). The Hugging Face model card confirms the Apache 2.0 license and lists the model at 117B total parameters with 5.1B active parameters (Hugging Face: openai/gpt-oss-120b). OpenAI's accompanying model card describes both models as open-weight reasoning models trained with large-scale distillation and reinforcement learning, and states that weights, inference implementations, tool environments, and tokenizers are all released under Apache 2.0 (arXiv:2508.10925 abstract).

Positioning: OpenAI reports that gpt-oss-120b reaches near-parity with OpenAI o4-mini on core reasoning benchmarks, outperforms o3-mini and matches or exceeds o4-mini on Codeforces competition coding, MMLU, HLE, and TauBench tool calling, and beats o4-mini on AIME 2024/2025 competition mathematics and HealthBench (OpenAI, "Introducing gpt-oss"). The models are optimized for deep research browsing, Python tool use, and developer-provided functions, and use a rendered chat format designed for clear instruction following and role delineation (arXiv:2508.10925 abstract).

Notable techniques: the model is an autoregressive mixture-of-experts transformer that alternates banded-window and fully dense attention, uses Grouped Query Attention with learned attention-sink biases in the softmax denominator, RoPE positional embeddings with YaRN context extension, and gated SwiGLU MoE activations (OpenAI gpt-oss model card PDF). Its MoE weights — more than 90% of all parameters — are natively quantized to MXFP4 at 4.25 bits per parameter, which lets the 120b model fit on a single 80 GB GPU such as an NVIDIA H100 or AMD MI300X (OpenAI gpt-oss model card PDF; Hugging Face model card).

Architecture

• Model type: autoregressive Mixture-of-Experts transformer; HF architecture class GptOssForCausalLM, model type gpt_oss (model card PDF; config.json)
• Total parameters: 116.83B (≈117B as listed on the HF card) (model card PDF; Hugging Face)
• Active parameters per token: 5.13B (≈5.1B) (model card PDF; Hugging Face)
• Parameter breakdown: MLP 114.71B, attention 0.96B, embed + unembed 1.16B; checkpoint size 60.8 GiB (model card PDF)
• Layers: 36 (num_hidden_layers: 36) (OpenAI blog; config.json)
• Residual stream / hidden size: 2,880 (model card PDF; config.json)
• Experts: 128 MoE experts per block, top-4 selected per token (softmax-weighted over selected experts via a linear router); num_local_experts: 128, num_experts_per_tok: 4 (model card PDF; config.json)
• MoE activation: gated SwiGLU (SwiGLU limit 7.0 in config) (model card PDF; config.json)
• Attention pattern: alternates banded-window (sliding) and fully dense attention; config lists a strict sliding_attention / full_attention alternation across all 36 layers (model card PDF; config.json)
• Sliding-window bandwidth: 128 tokens (model card PDF; config.json)
• Attention heads: 64 query heads per layer, head dim 64, 8 key-value heads — Grouped Query Attention (group size 8) (model card PDF; config.json; OpenAI blog)
• Attention sinks: each attention head has a learned bias in the softmax denominator (off-by-one attention), letting heads attend to nothing (model card PDF)
• Positional embeddings: RoPE with YaRN scaling (rope_theta: 150000, scaling factor 32.0, original max positions 4096) (model card PDF; config.json)
• Context window: 131,072 tokens for dense layers (128k native) (model card PDF; config.json; OpenAI blog)
• Vocabulary size: 201,088 (config.json)
• Quantization: MXFP4 on MoE weights at 4.25 bits/parameter (>90% of parameters); config quant_method: mxfp4 with self-attention, router, embeddings and lm_head left unquantized; tensor types BF16 · U8 (model card PDF; config.json; Hugging Face)
• Hardware footprint: runs within 80 GB of memory / a single 80GB GPU (OpenAI blog; Hugging Face)

Official vendor eval scores

All values are OpenAI-reported for gpt-oss-120b. Scores vary by reasoning effort (low / medium / high); the table reports high reasoning effort, with medium shown where the HF card also lists it.

Nemotron 3 Ultra figures are the vendor-reported numbers from the NVIDIA Nemotron 3 Ultra model card, shown as a common reference point across model pages. n.a. means NVIDIA does not report that benchmark, or reports a different variant of it that would not be a like-for-like comparison. % better is (score − Nemotron) / Nemotron, computed only where both sides are a plain number on the same metric; a rating scale, a range, or a prose cell shows — instead. Neither column is a head-to-head evaluation: each score is self-reported by its own vendor under its own harness.

Benchmark explanations

• AIME 2024 / AIME 2025 — American Invitational Mathematics Examination problem sets; measures competition-level mathematical reasoning, scored as accuracy with and without tool access (OpenAI blog).
• GPQA Diamond — the hardest subset of graduate-level, Google-proof science multiple-choice questions (biology, chemistry, physics); measures expert-level scientific reasoning (Hugging Face card links the GPQA dataset).
• MMLU — Massive Multitask Language Understanding: broad multiple-choice knowledge and general problem solving across academic and professional subjects (OpenAI blog).
• Humanity's Last Exam (HLE) — extremely difficult expert-written questions spanning many disciplines; measures frontier general problem solving (OpenAI blog).
• SWE-bench Verified — human-validated real GitHub issues; measures whether a model can produce patches that resolve software bugs in actual repositories (Hugging Face card links the SWE-bench Verified dataset).
• Codeforces (Elo) — competitive programming rating derived from Codeforces contest problems; measures algorithmic coding skill on a rating scale rather than a percentage (OpenAI blog).
• τ-Bench Retail (Tau-Bench) — agentic tool-calling evaluation in a retail customer-service environment; measures multi-turn function calling and policy-following (OpenAI blog describes TauBench as a tool-calling / agentic evaluation suite).
• HealthBench / Hard / Consensus — health-conversation evaluations scoring the quality and safety of medical responses, with a harder subset and a physician-consensus variant (OpenAI blog).