My 2022 prediction scorecard: how wrong was I?

Every January I write down 10 predictions about AI for the coming year. Then every December I score them.

This is the fun part. The part where I get to be publicly wrong.

Here are my 10 predictions from January 2022 and how they turned out.

The scorecard

| # | Prediction | Result | Score | |---|-----------|--------|-------| | 1 | GPT-4 will be announced in 2022 | Not announced (as of Dec 31) | Wrong | | 2 | Image generation will improve but stay niche | Quality: right. Niche: very wrong | Half | | 3 | At least one AI startup will hit $1B valuation based purely on generative AI | Stability AI hit $1B, Jasper hit $1.5B | Right | | 4 | OpenAI API prices will drop at least 20% | Prices unchanged all year | Wrong | | 5 | Open source LLMs will close the quality gap to within 90% of GPT-3 | GPT-J and BLOOM got close but not 90% on most benchmarks | Half | | 6 | Training costs for frontier models will exceed $20M | PaLM est. $10-15M, below $20M | Wrong | | 7 | Hugging Face will become the GitHub of ML | Downloads exploded, model count passed 100K | Right | | 8 | At least one major AI ethics controversy will involve training data | LAION dataset concerns, Stable Diffusion artist disputes | Right | | 9 | The "AI will take our jobs" discourse will peak and fade | It peaked, but it didn't fade. ChatGPT reignited it in December | Half | | 10 | No consumer AI product will achieve mainstream adoption in 2022 | ChatGPT: 1M users in 5 days | Right (wrong direction) |

Final score: 4 right, 3 half-right, 3 wrong.

Let me go through the interesting ones.

Prediction 1: GPT-4 (Wrong)

I was confident OpenAI would announce GPT-4 in 2022. They didn't. Instead they released InstructGPT, text-davinci-003, and ChatGPT. All based on GPT-3/3.5. The biggest product of the year was an iteration on a two-year-old model, not a new one.

This taught me something: model generations matter less than product packaging. ChatGPT had more impact than GPT-4 probably would have, because it made the technology accessible to non-technical people.

Prediction 4: API prices will drop (Wrong)

OpenAI's API prices didn't change at all in 2022. Davinci stayed at $0.02/1K tokens. No competitive pressure forced a drop, because the alternatives (Cohere, AI21) priced similarly, and open source models still require technical skill to deploy.

I think 2023 is when the pricing war starts. ChatGPT's eventual API and the growing open source options will create downward pressure. But in 2022, I was wrong.

Prediction 6: Training costs above $20M (Wrong)

I overestimated training costs. PaLM was the most expensive model trained in 2022, and estimates put it at $10-15M. The Chinchilla approach (smaller model, more data) is actually cheaper than the brute-force scaling I expected. Hardware improvements (A100s, TPU v4) also helped keep costs below my threshold.

Prediction 10: No mainstream consumer AI (Spectacularly Wrong)

This is the one that hurts. I wrote, in January 2022: "AI tools will continue to serve developers and businesses. No consumer-facing AI product will achieve mainstream recognition in 2022."

Then ChatGPT happened. 1 million users in 5 days. My mom heard about it. My neighbor asked me about it. It was on the evening news.

I was wrong by the largest possible margin. Not only did a consumer AI product achieve mainstream adoption, it set the all-time record for fastest product adoption in consumer technology history.

In my defense, on January 1, 2022, predicting ChatGPT would exist was not reasonable. The product was announced November 30, 2022, with essentially zero advance publicity. But that's the thing about predictions. You don't get points for the quality of your reasoning, only the outcome.

What the scorecard teaches me

I've done this exercise for three years now. My accuracy rate:

| Year | Right | Half | Wrong | Hit rate (right + half*0.5) | |------|-------|------|-------|---------------------------| | 2020 | 5 | 2 | 3 | 60% | | 2021 | 4 | 4 | 2 | 60% | | 2022 | 4 | 3 | 3 | 55% |

Consistent at 55-60%, which is better than a coin flip but not by much. I am a marginally better predictor than random chance. That's humbling, and it should be.

The pattern in my misses: I consistently overestimate technical progress (GPT-4, training costs) and underestimate product innovation (ChatGPT, Stable Diffusion going mainstream). I think about AI as a researcher. The market thinks about AI as a consumer.

I'll keep this bias in mind for my 2023 predictions.

2023 predictions (to be scored next December)

I'm writing these now so I can't chicken out. Ten predictions for 2023:

1. GPT-4 will launch in H1 2023
2. At least three major tech companies will integrate ChatGPT-like features
3. OpenAI will cut API prices by at least 30%
4. An open source model will match ChatGPT quality
5. AI-generated image detection will become a major product category
6. Total AI VC funding will exceed $40B in 2023
7. Midjourney will launch a standalone product outside Discord
8. At least one country will pass AI-specific legislation
9. The average person will be unable to tell AI text from human text
10. Someone will train a model with over 100 trillion tokens

I'll score these next December. The spreadsheet is already set up.

Happy New Year. May your predictions be better than mine.

---

If you found this interesting, you might also like:

• 5 charts that explain why GPU prices went insane in 2021
• AI research papers published in 2021: a mid-year count
• My 2021 AI data roundup: the 10 numbers that mattered most
• Every model released in 2022 so far, in one table
• 2022 in AI data: the year everything accelerated

-- dataku