Llama 3.3 70B Instruct

Overview

Llama 3.3 70B Instruct is Meta's instruction-tuned, text-only multilingual large language model, released on December 6, 2024 (Hugging Face: meta-llama/Llama-3.3-70B-Instruct; Meta Llama 3.3 model card on GitHub). It is an auto-regressive model built on an optimized transformer architecture with 70B parameters (the Hugging Face repo lists 71B params for the checkpoint), a 128k context window, and pretraining on more than 15 trillion tokens with a knowledge cutoff of December 2023 (Hugging Face model card). It takes multilingual text as input and produces multilingual text and code, with official support for English, German, French, Italian, Portuguese, Hindi, Spanish, and Thai (Hugging Face model card).

Positioning: Meta describes Llama 3.3 70B as offering performance similar to Llama 3.1 405B at a fraction of the serving cost, released "to close out the year" 2024 (Meta AI blog, "The future of AI: Built with Llama"). Meta's own instruction-tuned comparison table supports this: Llama 3.3 70B Instruct matches Llama 3.1 70B on MMLU (86.0) while beating both 3.1 70B and, on several tasks, the 405B model — for example IFEval 92.1 vs. 88.6 for 405B and MATH 77.0 vs. 73.8 for 405B (Hugging Face model card).

Licensing and techniques: the model is distributed under the custom commercial Llama 3.3 Community License Agreement, which grants royalty-free use, reproduction, modification and distribution but requires "Built with Llama" attribution, prefixing derivative model names with "Llama", a NOTICE file attribution, compliance with the Acceptable Use Policy, and a separate Meta-granted license for licensees exceeding 700 million monthly active users as of the release date (Hugging Face model card / license text; license file). Architecturally, all Llama 3 versions use Grouped-Query Attention for improved inference scalability, and the released BF16 checkpoints can additionally be loaded in 8-bit or 4-bit via bitsandbytes/transformers (Hugging Face model card).

Architecture

• Model type: auto-regressive language model using an optimized transformer architecture; dense (no mixture-of-experts) (Hugging Face model card)
• Total parameters: 70B per Meta's model information table (Hugging Face lists 71B params for the checkpoint); all parameters active per token since the model is dense (Meta model card; Hugging Face model card)
• Active parameters: n.a. as a separately reported figure — Meta reports only the 70B dense parameter count (Meta model card)
• Layers: 80 (num_hidden_layers: 80; the Meta/HF cards do not state layer count — value from a mirrored copy of the official config.json, as the meta-llama repo is gated) (config.json mirror)
• Hidden size: 8192 (config.json mirror)
• Attention: Grouped-Query Attention (GQA), 64 query heads and 8 key-value heads (8 query heads per KV group) (Hugging Face model card; config.json mirror)
• Experts / MoE: none — dense model, no expert routing reported (Meta model card)
• Context window: 128k tokens (max_position_embeddings: 131072) (Meta model card; config.json mirror)
• Positional embeddings: RoPE with rope_theta: 500000.0 (config.json mirror)
• Vocabulary size: 128,256 (config.json mirror)
• Precision / quantization: released in BF16 (torch_dtype: bfloat16); checkpoints can be run in 8-bit or 4-bit with bitsandbytes (load_in_8bit=True / load_in_4bit=True) (Hugging Face model card; config.json mirror)
• Training data: 15T+ pretraining tokens, knowledge cutoff December 2023 (Meta model card)

Official vendor eval scores

Meta-reported results for Llama 3.3 70B Instruct (English text, instruction-tuned evaluation table).

Benchmark explanations

• MMLU (CoT) — Massive Multitask Language Understanding: multiple-choice questions across 57 academic and professional subjects, evaluated with chain-of-thought prompting; measures broad knowledge and reasoning (Meta model card).
• MMLU Pro (CoT) — a harder, more reasoning-intensive refresh of MMLU with more answer options and cleaner items; measures robust general reasoning (Hugging Face card links the MMLU-Pro dataset).
• IFEval — instruction-following evaluation with verifiable constraints (format, length, keywords); Meta classifies it under "Steerability" (Meta model card).
• GPQA Diamond (CoT) — hardest subset of graduate-level, Google-proof science questions; Meta classifies it under "Reasoning" and scores accuracy (Meta model card).
• HumanEval — Python function-completion problems scored pass@1 by unit tests; measures basic code generation correctness (Meta model card, "Code" category).
• MBPP EvalPlus (base) — Mostly Basic Python Problems with the EvalPlus expanded test suite, pass@1; measures code correctness under stricter tests (Meta model card).
• MATH (CoT) — competition math word problems graded by symbolic equivalence (sympy); measures multi-step mathematical reasoning (Meta model card, "Math" category).
• BFCL v2 — Berkeley Function-Calling Leaderboard v2, scored on abstract-syntax-tree validity of generated function calls; measures tool-use/function-calling accuracy (Meta model card, "Tool Use" category).
• MGSM — Multilingual Grade School Math, exact-match accuracy on GSM8K-style problems translated into multiple languages; measures multilingual arithmetic reasoning (Meta model card, "Multilingual" category).