Transcript of Why smarter AI models could drive up compute prices 10x

11:17 9 min read 0 views Published 4 months ago

Copy link
Episode summary

In this video recording of a post he wrote, Dwarkesh Patel examines how the compute situation for frontier AI labs could evolve over the next few years. He argues that Anthropic's revenue has grown roughly tenfold each year while compute has only tripled, so the gap must close through higher lab margins, higher compute prices, or a larger share of compute spent on inference.

He walks through evidence such as rising spot prices and inference margins, and explains why he thinks compute prices may stay high. Listeners will hear his reasoning about economies of scale, the limits of Moore's law and chip fabs, and why he is wary of comparisons to past scarcity bets.

In this episode Dwarkesh Patel

This transcript of the episode on rising AI compute prices was made with HappyScribe technology, the same kind that powers Teams AI note taker.

Dwarkesh Patel00:00:00

Today I want to talk about what the compute situation for the labs will look like over the next few years. For the last 3 consecutive years, Anthropic's revenue has 10x to year-over-year, and it's likely to do so again this year. So they ended last year with $9 billion in revenue; I think they'll probably end this year with somewhere between $100 billion to $150 billion in revenue. Now, for this trend to continue, Anthropic would need to make $1 trillion in revenue by the end of next year. Of course, there's no deep reason why this has to be true; it's a very wild conclusion, and it's ultimately a question of AI capabilities: does AI get that useful by the end of next year?

But suppose the trend does continue, well, I want to think through what happens in that world. Now, the other big trend in AI is that lab compute only 3x year-over-year. For a lab to keep 10xing revenue year-over-year while compute only 3x, one of the following 3 things needs to happen, or some combination of the 3, needs to happen: (1) lab margins have to increase, (2) the price of compute has to increase, or (3) the percentage of compute that labs spend on inference rather than training has to increase.

My understanding is that basically all 3 of these things are already happening. With regards to the margins, Anthropic's inference margins reportedly went from 40% in the middle of last year to upwards of 80% now. With regards to compute, the spot prices for compute are more than 40% higher than they were in the February trough that we had earlier this year. And with regards to the share of compute that goes to training versus inference, in 2024, according to Epoch, OpenAI was spending just a quarter of its compute on inference, and that number is likely closer to 50%, if not higher now.

Now, labs would prefer not to do this final thing of increasing the share of compute they spend on inference. The way that labs see the world, the whole point of inference revenue is to help convince investors to give you more money in order to train the next bigger, better model. And if you're spending most of your compute on inference, then you're basically declaring that AI progress has stalled, and you're just now in the business of being a cloud provider. Now, this is a less compelling business than building AGI, and so the labs do not want to be in this business, nor do they think they are in this world.

They think that within a year they'll have built models that make the current ones look extremely shitty. But they need to invest a lot of their compute, the majority of their compute, into doing the training and experiments that are necessary to build the next model. So that leaves only 2 options for how you can get out of this gap between the fact that lab compute only increases 3x year-over-year, but revenue increases 10x. Either the lab's margins have to increase so that they get this surplus, or the price of compute has to increase so that everybody in the stack below the lab gets the surplus.

It's not clear to me which world we end up in. Do we end up in a world where we go from 80% margins for some of the top models to greater than 90% margins if the lab margin effect dominates? Well, that would require the leading model to be so far ahead of the competition, because the nature of margins, why they exist in a market economy, is that the thing you are serving is so much better than what somebody else could go get and replace you on the market. But it's just really wild for me to consider that the margins for something like intelligence will be greater than 90%, and they don't get competitive AI at that level.

So that leaves only one other possibility of this escape valve between these 2 trends, which is that the price of compute has to increase. As I mentioned, this is already starting to happen, and the effect is even stronger when you look at the tranche of compute that the frontier labs actually need to accumulate. Because they can't just go out and buy a spot instance. They need to make sure that they get enough scale to get really good efficiency and flexibility, and also that they have the kind of compute that lends itself to the security they need for their own weights and for their customers' information.

So I think a relevant case study here is to look at the compute that Google and Anthropic are renting from SpaceX. Google, for example, is paying $900 million a month for 110,000 GPUs that are a blend of GB200s and GB300s. The price that Google is paying here is 2x the spot price per hour for those GPUs. And that spot price itself is more than 40% higher than it would have been in February. I want to emphasize the key conclusion here: that as AI models get smarter, they'll be better able to monetize the same amount of compute.

If a true human-level software engineer could run on an H100 equivalent, then at today's prices for software engineers, that H100 should rent for over $250K a year. That's over 15x the current spot price for an H100. And this is not even accounting for the fact that your AI can work nights and weekends. Of course, you might expect that if we had 10 million extra software engineers suddenly appear in the economy, the marginal value of a software engineer would decrease, and thus the revenue that that H100 would be able to generate would not be 15x higher than it is right now.

But I actually don't know if this is true. If we apply this argument to people instead of AIs, then this would be the classic lump of labor fallacy. For example, economists generally believe that high-skilled immigration does not decrease wages in the long run because of how innovation and specialization increase the value of labor. Maybe this labor supply shock will be so big and so fast that we can't count on this general heuristic anymore. But if you believe what standard economics says, then the marginal value of labor, and thus the marginal value of compute, should stay astonishingly high.

So let's think about what changes in such a world. Well, one of the things that would happen is that as the top labs get better and better at monetizing compute, and the cost of compute increases, it becomes harder for anybody else to compete against them, because they have to bid for this resource against somebody who is basically able to make better use of it. Another thing that will happen, and I think this is actually the most interesting implication of this whole thought exercise, is that if you can train the best, most efficient model, then you'll be able to charge much higher margins than you can today. This is the Alkin-Allen effect in economics, and what it's basically saying is that if it costs $20 an hour to rent an H100, then it would be extremely stupid to use a weaker, less efficient model, because it's going to burn more tokens on your expensive compute to get the exact same result.

So labs will be able to charge a much larger premium if they can train a model that better economizes this scarce input. Basically, if you have a model that can get the same result by using less compute, then you've, in some sense, created more compute, and the value of compute is going to increase. Another thing that will happen is that a lot of current popular applications of AI will probably get priced out. The reason AI is relatively cheap right now is that AI just can't do a lot of things that top humans can do. But this, at some point, will no longer be the case.

Dwarkesh Patel00:06:34

And at that point, Google, or Anthropic, or OpenAI will be willing to pay more for the tokens to automate AI research than you or I will be willing to pay to make more AI slop talk. I'm a bit worried that this kind of analysis honestly pattern-matches a lot onto the ways that people in the past have been wrong about scarcity. I'm, for example, thinking of the famous Simon Irlick bet. Paul Irlick was this famous doomer about population growth, and he made this bet that a basket of commodities would increase in price rather than decrease in the decade preceding 1990.

And this is a very famous bet because it's supposed to illustrate how Irlick's Malthusian worldview was wrong, and how he did not anticipate the way in which market signals and human ingenuity can find better ways to economize scarce inputs. I'm guessing that the analogy to this bet is probably wrong. Other analysis has shown that if that bet had been made in a different decade, Irlick might well have won. But more generally, I think the supply of compute is much less elastic and much less capable of absorbing large demand shocks, and much less capable of being accommodated by using different substitutes than the extraction of different metals.

To illustrate why I think this 3x in compute capacity year-over-year is hard to budge, or potentially even sustain, is that I don't see how any of the 3 elements that constitute that 3x can be much accelerated. So 1.4x of that is coming from Moore's law. Far from increasing it, I think it'll be a miracle if we can just keep it going for a few more years. 1.2x is coming from building new fabs. This process is ultimately going to be a bottleneck up to 2030, and potentially even beyond, by just building new ASML EUV machines.

Dillon, when he was on the podcast a few months ago, talked about this in great detail. And 1.8x comes from the fact that AI is absorbing a lot of wafer allocation that was previously going to smartphones and PCs. This is probably going to hit a wall by the end of next year, when at the leading edge N3 nodes at TSMC, AI will have gone from 60% to 86%. At some point, you have just absorbed all leading edge wafer capacity for AI, and you can't keep increasing this number. So I don't know how we get even to continue to do 3x compute scaling year-over-year for the next few years, much less go beyond that.

Advertisement00:08:56
Dwarkesh Patel00:10:32

By the way, the fact that Anthropic revenue has been 10x year-over-year, whereas their compute has only been 3x year-over-year, I think illustrates how strong the economies of scale are in the model business. And logically, this makes sense. When you train a model, you just have to spend this one-time cost to learn all these different skills that then get to be shared across all your users. This is very unlike human labor, where each instance has to be retrained from scratch. I wish we didn't live in a world with such strong economies of scale for intelligence, because I'm worried about power concentration.

But it seems we do. Okay, this was a narration of a blog post that I also released on my website at dwarkesh.com. Check it out for other posts or to be notified when I release a post in the future. Otherwise, I'll see you for the next full episode.

Episode description

This is a video recording of a post I wrote last week. If you want to read the original you can check it out here.

Thanks to Mercury for sponsoring this video. Mercury’s built-in AI, Command, helps me close my books and saves me a bunch of time. At the end of each month, Command categorizes my transactions and provides its rationale for every choice: I just review, fix anything that’s off, and approve... and then Mercury syncs everything to QuickBooks. Get started at mercury.com/command

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.dwarkesh.com