Phi-3 Mini is a 3.8B model that's shockingly good. Small model benchmarks.

I just ran a 3.8 billion parameter model on my MacBook and it outperformed models 2-18x its size on multiple benchmarks.

Microsoft Research released Phi-3 Mini on April 22nd. It has 3.8B parameters. That's smaller than GPT-2 XL (1.5B was GPT-2's largest, so Phi-3 is about 2.5x GPT-2). And it beats Mistral 7B and Llama 3 8B on several benchmarks.

The small model revolution isn't coming. It's already here and I almost missed it because I was too busy benchmarking the big ones.

The numbers that made me stop scrolling

| Benchmark | Phi-3 Mini (3.8B) | Mistral 7B | Llama 3 8B | Mixtral 8x7B (46.7B) | GPT-3.5-turbo | |-----------|-------------------|------------|-----------|----------------------|---------------| | MMLU (5-shot) | 68.8% | 60.1% | 66.6% | 70.6% | 70.0% | | HumanEval | 58.5% | 30.5% | 62.2% | 34.2% | 48.1% | | GSM8K | 82.5% | 35.4% | 79.6% | 58.4% | 57.1% | | MATH | 31.0% | 13.1% | 30.0% | 22.7% | 23.5% | | ARC-Challenge | 84.9% | 55.5% | 78.6% | 65.7% | 85.2% | | BIG-Bench-Hard | 71.7% | 57.3% | 61.1% | 69.7% | 66.6% |

Sources: Microsoft Research Phi-3 technical report, model cards for compared models, Hugging Face evaluation data.

A 3.8B model scoring 68.8% on MMLU. That's higher than Mistral 7B (60.1%), higher than Llama 3 8B (66.6%), and within 2 points of GPT-3.5-turbo (70.0%).

On GSM8K (math), Phi-3 Mini scores 82.5%. That beats every model in this table including Mixtral 8x7B (58.4%) and GPT-3.5-turbo (57.1%). A 3.8B model. Beating a 46.7B model on math by 24 points.

I triple-checked these numbers. They're real.

Running it on a MacBook

I installed Phi-3 Mini via Ollama and LM Studio. Here are my local performance numbers:

| Hardware | Quantization | Model size on disk | RAM used | Tokens/sec | |----------|-------------|-------------------|----------|-----------| | MacBook Pro M2 (16GB) | Q4_K_M | 2.3GB | ~4.5GB | 52 | | MacBook Pro M2 (16GB) | Q8_0 | 4.1GB | ~6.2GB | 34 | | MacBook Pro M2 (16GB) | FP16 | 7.6GB | ~9.8GB | 18 | | RTX 4090 (24GB) | FP16 | 7.6GB | ~9.1GB VRAM | 186 |

Source: My measurements, April 2024. Ollama runtime.

52 tokens per second on a laptop in 4-bit quantization. That's genuinely usable. The 2.3GB model file downloads in seconds. From "I want to try this model" to "I'm having a conversation with it" takes under 60 seconds on consumer hardware.

The Phi approach: it's all about the training data

Microsoft's Phi models have been making noise since Phi-1 in June 2023. Their thesis: you can compensate for small model size with extremely high-quality training data, especially synthetic data.

| Model | Parameters | Training data size | Training data strategy | |-------|-----------|-------------------|----------------------| | Llama 2 7B | 7B | 2T tokens | Web crawl + curation | | Mistral 7B | 7.2B | Unknown | Unknown (likely mixed) | | Llama 3 8B | 8B | 15T tokens | Web crawl + heavy filtering | | Phi-3 Mini | 3.8B | 3.3T tokens | Heavily filtered web + synthetic |

Sources: Microsoft Research Phi-3 technical report, respective model papers.

Phi-3 Mini uses 3.3T tokens, but the composition matters more than the count. Microsoft's approach emphasizes:

1. Synthetic data generated by larger models (filtered for quality)
2. Carefully curated web data (aggressively filtered)
3. Heavy emphasis on math and reasoning data

This is the same thesis as Llama 3 (data quality over model size) but pushed further. Phi-3 is roughly half the parameters of Llama 3 8B but achieves similar results because the training data is even more concentrated on quality.

The caveat: benchmarks aren't everything

I should temper the excitement. I also ran Phi-3 Mini through my standard evaluation, and the results are more mixed:

| Task | Phi-3 Mini (3.8B) | Llama 3 8B | Mistral 7B | |------|-------------------|-----------|------------| | Factual Q&A (25 prompts) | 58% correct | 71% correct | 62% correct | | Creative writing (25 prompts) | 3.12/5 | 3.72/5 | 3.54/5 | | Multi-step reasoning (25 prompts) | 60% correct | 64% correct | 48% correct | | Code generation (25 prompts) | 64% pass rate | 72% pass rate | 52% pass rate | | Summarization (25 prompts) | 3.28/5 | 3.84/5 | 3.62/5 | | Instruction following (25 prompts) | 56% fully correct | 68% fully correct | 60% fully correct | | Overall | 3.21 avg | 3.68 avg | 3.38 avg |

Source: My evaluation, 150 prompts, blind rating, April 2024.

On my real-world evaluation, Llama 3 8B beats Phi-3 Mini in every category. The benchmark scores and my evaluation tell different stories.

Why? I think Phi-3 Mini is optimized for benchmark-style tasks (multiple choice, math word problems, code completion). On open-ended generation, summarization, and creative work, the larger model's extra parameters provide more nuance and range.

This is an important distinction. If your use case maps closely to benchmark-style tasks (structured Q&A, math, code problems), Phi-3 Mini is extraordinary for its size. If you need general-purpose generation quality, Llama 3 8B is still the better choice, even at 2x the parameters.

The small model trajectory

I made a chart of the best MMLU score achievable at different parameter counts over time:

| Parameter count | Best MMLU (Jan 2023) | Best MMLU (Jan 2024) | Best MMLU (Apr 2024) | Improvement | |----------------|---------------------|---------------------|---------------------|------------| | Under 5B | ~35% | ~52% | 68.8% (Phi-3 Mini) | +33.8 pts | | 5-10B | ~46% | ~60% | 66.6% (Llama 3 8B) | +20.6 pts | | 10-20B | ~52% | ~65% | ~71% (various) | +19 pts | | 60-70B | ~68% | ~69% | 79.5% (Llama 3 70B) | +11.5 pts |

Sources: Hugging Face Open LLM Leaderboard history, model papers, my tracking data.

The smallest models are improving the fastest. Under-5B jumped 33.8 MMLU points in 16 months. The 60-70B tier only jumped 11.5 points. Small models are catching up because they had the most room to gain from better training data. The large models already saturated what good data could offer.

This is incredibly exciting for edge deployment, mobile AI, and any scenario where you can't send data to a cloud API. A 3.8B model that scores 68.8% on MMLU runs on a phone. That wasn't possible a year ago.

My prediction: by the end of 2024, we'll see a sub-3B model score 70%+ on MMLU. The small model revolution is just getting started.

---

If you found this interesting, you might also like:

• Every AI benchmark from 2020, ranked by how much they actually tell you
• DALL-E 2 is out. I ran 200 prompts and measured the results.
• InstructGPT and RLHF: what the training data tells us
• The Chinchilla scaling laws changed everything. Let me show you why.
• I ran GPT-3 on the same 50 questions every month for a year. Here's the drift.

-- dataku