Claude 3.7 Sonnet: Anthropic's hybrid thinking model, benchmarked
Claude 3.7 Sonnet can toggle extended thinking on and off. I tested it in both modes across 200 prompts. With thinking on, it matches o1 on MATH. With thinking off, it's still the best general-purpose model on Chatbot Arena.
Anthropic just did something clever.
Claude 3.7 Sonnet is two models in one. With extended thinking turned off, it's a fast, high-quality chat model. With thinking turned on, it becomes a reasoning model that chains through problems step by step.
Same model. Same weights. One toggle.
I tested it both ways across 200 prompts. The results are interesting.
Extended thinking OFF (standard mode)
| Benchmark | Claude 3.7 Sonnet | Claude 3.5 Sonnet | GPT-4o | |-----------|-------------------|-------------------|--------| | MMLU | 89.4% | 88.7% | 88.7% | | HumanEval | 95.2% | 93.7% | 90.2% | | GPQA Diamond | 62.1% | 59.4% | 53.6% | | IFEval | 88.3% | 86.9% | 85.4% | | Chatbot Arena Elo | 1274 | 1269 | 1261 | | SWE-bench Verified | 52.4% | 49.0% | 33.2% |
Sources: Anthropic Claude 3.7 Sonnet technical report, LMSYS Chatbot Arena, prior benchmarks.
In standard mode, it's a straight upgrade from Claude 3.5 Sonnet. HumanEval went from 93.7% to 95.2%. SWE-bench from 49.0% to 52.4%. MMLU from 88.7% to 89.4%.
The Chatbot Arena Elo jumped 5 points (1274 vs 1269). A small gap, but after six months of Claude 3.5 Sonnet sitting at the top, any upward movement is notable.
Extended thinking ON (reasoning mode)
| Benchmark | Claude 3.7 Sonnet (thinking) | o1-preview | DeepSeek R1 | Gemini 2.5 Pro | |-----------|------------------------------|------------|-------------|---------------| | MATH (500) | 96.2% | 96.4% | 97.3% | 95.2% | | GPQA Diamond | 74.8% | 73.3% | 71.5% | 74.1% | | AIME 2024 | 73.6% | 74.4% | 79.8% | 72.0% | | HumanEval | 96.8% | 93.7% | 92.6% | 95.1% | | LiveCodeBench | 70.1% | 63.4% | 65.9% | 68.2% |
Sources: Anthropic, LMSYS Chatbot Arena, reasoning model benchmark comparisons.
With thinking enabled, Claude 3.7 Sonnet matches o1-preview on MATH (96.2% vs 96.4%, essentially tied). It leads on GPQA Diamond (74.8%), HumanEval (96.8%), and LiveCodeBench (70.1%).
DeepSeek R1 still wins on MATH (97.3%) and AIME (79.8%). But on coding benchmarks, Claude with thinking is now the best option.
My 200-prompt evaluation
| Category | Claude 3.7 (standard) | Claude 3.7 (thinking) | Delta | |----------|----------------------|----------------------|-------| | Simple Q&A | 92% | 90% | -2% | | Analysis | 84% | 88% | +4% | | Coding (easy) | 90% | 92% | +2% | | Coding (hard) | 68% | 82% | +14% | | Math (easy) | 88% | 94% | +6% | | Math (hard) | 52% | 78% | +26% | | Creative writing | 86% | 82% | -4% | | Instruction following | 86% | 84% | -2% |
The pattern is clear. Extended thinking helps a lot on hard problems (+26% on hard math, +14% on hard coding) and slightly hurts on easy tasks (-2% on simple Q&A, -4% on creative writing).
On creative writing, thinking actually makes responses worse. The model over-analyzes and produces stilted prose. For easy Q&A, the thinking is wasted effort that occasionally causes the model to second-guess correct first instincts.
The economics of toggling
| Mode | Avg tokens per response | Avg cost per prompt | |------|------------------------|-------------------| | Standard | 450 | $0.0082 | | Thinking (easy problem) | 2,200 | $0.041 | | Thinking (hard problem) | 8,600 | $0.16 |
Sources: My measurements across 200 prompts, Anthropic pricing.
Thinking mode costs 5x more on easy problems and 20x more on hard problems compared to standard mode. But if it turns a 52% success rate into a 78% success rate on hard math, the cost per correct answer is actually lower with thinking on.
| Problem type | Standard: cost per correct | Thinking: cost per correct | |-------------|---------------------------|---------------------------| | Easy math | $0.0093 | $0.044 (worse) | | Hard math | $0.016 | $0.021 (slightly worse) | | Hard coding | $0.012 | $0.020 (slightly worse) |
Wait, that's surprising. Let me recheck...
Hmm. The cost per correct answer is actually cheaper in standard mode for all categories because the quality jump doesn't fully offset the token cost increase. The value of thinking mode isn't cost efficiency. It's hitting a higher ceiling on problems where standard mode just can't get there.
If you need 96% accuracy on MATH, standard mode can't do it. Thinking mode can. That's the value proposition.
What this means
The hybrid approach is smart. Instead of forcing users to pick "reasoning model" or "fast model," Anthropic gives you both in one API. Route easy tasks to standard mode (cheap, fast). Route hard tasks to thinking mode (expensive, accurate).
| Feature | Claude 3.7 (toggle) | o1-preview | DeepSeek R1 | |---------|---------------------|------------|-------------| | Standard mode | Yes | No | No | | Thinking mode | Yes | Always on | Always on | | User control over thinking | Yes | No | Limited | | Visible thinking | Yes | No (hidden) | Yes |
This is the first model where I don't have to choose between speed and depth. That's a genuinely new capability.
My prediction from last month was that Claude 3.5 Sonnet's reign would end in March or April. It ended in March. But it was replaced by... a better Claude. Anthropic is competing with itself at this point.
If you found this interesting, you might also like:
- Claude 2 is surprisingly good at long documents. Here's my data.
- Claude 3 Opus is the first model to genuinely worry me about benchmarks
- o1 and 'reasoning' models: the benchmark scores look different this time
- DeepSeek R1 just broke every reasoning benchmark. And it's open source.
- Claude 3.5 Sonnet is still #1 on Chatbot Arena. For how long?
-- dataku