Anthropic just raised $580M. Let's talk about the AI safety funding numbers.
I compiled every dollar raised by AI safety organizations in 2022. The total is $1.9 billion. But 87% went to just two companies. The distribution is incredibly top-heavy.
Anthropic closed a $580 million Series B last week. That's a lot of money for a company that's been around for about 18 months.
But the Anthropic raise is just one data point in a trend that's been building all year. I decided to compile every dollar that went to AI safety-focused organizations in 2022. The total is bigger than I expected, and the distribution is more concentrated than I expected.
The numbers
| Organization | Amount raised (2022) | Round/Type | Notable investors | |-------------|---------------------|-----------|-------------------| | Anthropic | $580M | Series B | Spark Capital, Google | | OpenAI | $1.0B (est. additional) | Various | Microsoft | | DeepMind | N/A (Google subsidiary) | Internal | Google (parent) | | Redwood Research | $20M | Grant/Seed | Open Philanthropy | | ARC (Alignment Research Center) | $6M | Grants | Open Philanthropy, individual donors | | MIRI | $5.3M | Donations | Individual donors | | Conjecture | $3.8M | Seed | Various | | EleutherAI | $2.5M | Grants | Various foundations | | Other safety orgs (combined) | ~$15M | Various | Various | | Total (excluding DeepMind) | ~$1.63B | | |
I'm estimating OpenAI's 2022 additional funding at $1B based on Crunchbase data and reporting from multiple sources. The exact number is hard to pin down because of their complex corporate structure.
DeepMind is excluded from the total because as a Google subsidiary, their "funding" is Google's internal budget allocation, which isn't publicly comparable.
The concentration problem
This is the number that stuck with me: Anthropic ($580M) and OpenAI ($1B) together account for approximately $1.58 billion of the $1.63 billion total. That's 96.9%.
If you include OpenAI's pre-2022 Microsoft investment of $1B, the concentration is even starker. But sticking to 2022 flows:
| Category | Amount | Share | |----------|--------|-------| | Top 2 (Anthropic + OpenAI) | $1.58B | 96.9% | | All other safety orgs combined | $52.6M | 3.1% |
96.9% of AI safety funding went to two companies that are also building frontier AI models. The non-profit research organizations (MIRI, ARC, Redwood) collectively raised about $31M, which is less than 2% of the total.
There's a tension here that the data makes visible. Most of the money flowing into "AI safety" is going to companies that are simultaneously pushing the frontier of AI capability. They argue (reasonably) that you need to be at the frontier to study frontier risks. But the result is that pure safety research, the kind done by organizations that only study risks without building competing products, gets a tiny slice.
Historical context
I went back and compiled rough annual totals for AI safety funding since 2019:
| Year | Total AI safety funding (est.) | Largest single amount | |------|-------------------------------|----------------------| | 2019 | ~$150M | Microsoft's $1B pledge to OpenAI (announced, multi-year) | | 2020 | ~$200M | OpenAI compute purchases | | 2021 | ~$350M | Anthropic Series A ($124M) | | 2022 | ~$1.63B | OpenAI additional funding (~$1B) |
The 2022 total is roughly 4.7x the 2021 total. That's a single-year jump that dwarfs the previous growth rate.
Where the safety research money actually goes
For the non-commercial safety organizations, I tried to estimate how the money is allocated:
| Organization | Primary focus | Staff size (est.) | Cost per researcher (est.) | |-------------|--------------|-------------------|--------------------------| | MIRI | Mathematical alignment | 15-20 | ~$265K-$353K | | ARC | Alignment evaluation | 8-12 | ~$500K-$750K | | Redwood Research | Applied alignment | 20-30 | ~$667K-$1M | | Conjecture | Alignment theory | 10-15 | ~$253K-$380K |
These are rough estimates based on published team sizes and funding amounts. The per-researcher costs look high, but they include compute costs (which can be significant), overhead, and the premium salaries needed to compete with Google and OpenAI for talent.
A senior alignment researcher who could earn $400K at Google or $500K at OpenAI needs a competitive offer. The talent market for people who understand both ML and safety is incredibly tight.
The question I keep asking
Is $52.6 million enough for non-commercial AI safety research? For comparison:
| Research area | Annual funding (US) | |--------------|-------------------| | AI safety (non-commercial, 2022) | $52.6M | | Cancer research (NIH, 2022) | $7.2B | | Cybersecurity research (NIST, 2022) | $1.2B | | Nuclear safety (NRC, 2022) | $880M |
I'm not saying AI safety deserves the same funding as cancer research. I'm saying the scale matters. If frontier AI models pose risks anywhere close to what the safety community claims, $52.6 million split across a dozen small organizations seems thin.
Anthropic's $580M and OpenAI's funding are partly directed at safety research within those companies. But those companies also have strong incentives to ship products. The research priorities of a company that needs revenue are different from the priorities of a non-profit focused purely on understanding risks.
What I think the data shows
Three things.
First, AI safety as a funding category is growing explosively. 4.7x year-over-year is venture-capital-style growth.
Second, the growth is almost entirely going to commercial entities. Pure safety research is growing too, but at a much slower rate and from a much smaller base.
Third, the talent competition between commercial AI labs and safety organizations is getting more lopsided as the commercial labs raise more money. Every dollar Anthropic raises increases the salary it can offer, which pulls researchers away from non-commercial alternatives.
Whether this concentration is good or bad depends on whether you believe commercial AI labs can effectively self-regulate their safety research. The data doesn't answer that question. But it does make the question impossible to ignore.
If you found this interesting, you might also like:
- I counted every AI startup that raised money in Q1 2021. The numbers are strange.
- The GPT-3 API waitlist is 6 months long. Here's what the early data looks like.
- Open source AI is having a moment. Here are the download numbers.
- Hugging Face just hit 10,000 models. Here's what the model zoo looks like.
- My 2021 AI data roundup: the 10 numbers that mattered most
-- dataku