Claude 4 Sonnet vs GPT-4o vs Gemini 2.5 Flash: the mid-tier model war

Most production AI applications don't use flagship models. They use the mid-tier: good enough quality, reasonable cost, fast response times.

The three dominant mid-tier models right now are Claude 4 Sonnet (Anthropic), GPT-4o (OpenAI), and Gemini 2.5 Flash (Google). I tested all three on the tasks that matter most in production.

The models

| Spec | Claude 4 Sonnet | GPT-4o | Gemini 2.5 Flash | |------|----------------|--------|-----------------| | Input/M tokens | $3.00 | $2.50 | $0.15 | | Output/M tokens | $15.00 | $10.00 | $0.60 | | Context window | 200K | 128K | 1M | | Speed (tokens/sec) | ~90 | ~85 | ~320 |

Sources: Anthropic, OpenAI, Google.

The pricing spread: Gemini 2.5 Flash is 20x cheaper than Claude 4 Sonnet on input and 25x cheaper on output. GPT-4o falls in between.

Task 1: Summarization (50 documents)

I gave each model 50 business documents (reports, articles, legal briefs) and asked for 200-word summaries.

| Metric | Claude 4 Sonnet | GPT-4o | Gemini 2.5 Flash | |--------|----------------|--------|-----------------| | Key point coverage | 92% | 88% | 85% | | Factual accuracy | 96% | 93% | 91% | | Conciseness | Good | Good | Tends to over-include | | Cost for 50 docs | $0.42 | $0.31 | $0.018 |

Claude wins on accuracy and coverage. Gemini Flash is 23x cheaper but misses some key points. GPT-4o is in the middle on both quality and cost.

Task 2: Data extraction (100 invoices)

100 sample invoices. Extract: vendor name, amount, date, line items.

| Metric | Claude 4 Sonnet | GPT-4o | Gemini 2.5 Flash | |--------|----------------|--------|-----------------| | Field accuracy | 97.2% | 95.8% | 94.1% | | Structure consistency | 98% | 96% | 93% | | Edge case handling | 92% | 88% | 82% | | Cost for 100 invoices | $0.84 | $0.62 | $0.036 |

Claude 4 Sonnet leads on all quality metrics. The edge case handling gap (92% vs 82%) matters: unusual invoice formats trip up Gemini Flash more often.

But: Gemini Flash at $0.036 for 100 invoices vs $0.84 for Claude. If 94% accuracy is sufficient, the cost savings are massive.

Task 3: Classification (500 emails)

Classify 500 emails into 8 categories: support, sales, billing, feature request, bug report, spam, internal, other.

| Metric | Claude 4 Sonnet | GPT-4o | Gemini 2.5 Flash | |--------|----------------|--------|-----------------| | Classification accuracy | 94.6% | 93.2% | 92.8% | | Multi-label handling | 91% | 89% | 86% | | Cost for 500 emails | $0.38 | $0.28 | $0.016 |

The gap is small here. All three are above 92%. For classification, the cheapest model that crosses your accuracy threshold wins, and for most applications that's Gemini Flash.

Task 4: Code generation (30 tasks)

30 code generation tasks: API endpoints, data processing functions, database queries.

| Metric | Claude 4 Sonnet | GPT-4o | Gemini 2.5 Flash | |--------|----------------|--------|-----------------| | Correct on first try | 83% | 73% | 67% | | Correct after one fix | 93% | 87% | 80% | | Code quality score | 8.4/10 | 7.8/10 | 7.1/10 | | Cost for 30 tasks | $1.24 | $0.91 | $0.054 |

Coding is where Claude 4 Sonnet's advantage is clearest. 83% first-try accuracy vs 67% for Gemini Flash. The gap widens on harder tasks.

Overall scorecard

| Category | Winner | Runner-up | Cost winner | |----------|--------|-----------|------------| | Summarization | Claude 4 Sonnet | GPT-4o | Gemini 2.5 Flash | | Data extraction | Claude 4 Sonnet | GPT-4o | Gemini 2.5 Flash | | Classification | Claude 4 Sonnet | GPT-4o | Gemini 2.5 Flash | | Code generation | Claude 4 Sonnet | GPT-4o | Gemini 2.5 Flash |

Claude 4 Sonnet wins quality in all four categories. Gemini 2.5 Flash wins cost in all four categories. GPT-4o is always second on both metrics.

The real decision framework

| Your priority | Best choice | Why | |--------------|------------|-----| | Maximum accuracy | Claude 4 Sonnet | Wins every quality metric | | Cost efficiency | Gemini 2.5 Flash | 20-25x cheaper, 90%+ quality | | Balanced | GPT-4o | Middle on cost and quality | | High volume, good enough | Gemini 2.5 Flash | Classification/extraction at scale | | Coding | Claude 4 Sonnet | 16-point advantage on first-try |

For most production use cases, I'd start with Gemini 2.5 Flash and switch to Claude 4 Sonnet only for tasks where the accuracy gap actually impacts business outcomes.

The mid-tier model war is the real war. This is where the volume is, where the revenue is, and where provider choice makes the biggest economic difference.

---

If you found this interesting, you might also like:

• Claude vs GPT-4: my first head-to-head data comparison
• Mistral Large vs GPT-4 vs Claude 3 Opus: the three-way benchmark
• [Gemini 2.5 Pro and ](/blog/gemini-2-5-pro-thinking-models-google-answer-o1)
• GPT-3 vs GPT-J: the first real open source challenger, in data
• Google's PaLM has 540 billion parameters. Let me put that number in context.

-- dataku