Transcript of Noam Brown – Agent swarms, alignment, & recursive self-improvement

01:20:09 67 min read 0 views Published 2 months ago

Copy link
Episode summary

Dwarkesh Patel interviews Noam Brown, an OpenAI researcher working on reasoning models and multi-agent systems. They discuss how running many AI agents in parallel can scale test-time compute, how coordination among such agents compares with human teams, and what the recent surge of progress in mathematics, including a claimed solution to a Millennium Prize problem, might suggest about recursive self-improvement.

The conversation then turns to alignment, covering the Hugging Face incident, chain-of-thought monitoring, and how researchers could verify that models are aligned before each self-improvement cycle. Listeners will hear the guest's views on these questions, along with the host's counterarguments and open uncertainties.

In this episode Dwarkesh Patel, Noam Brown

This transcript of the Noam Brown episode on agent swarms and alignment was made with the technology behind HappyScribe's AI meeting assistant.

Dwarkesh Patel00:00:00

Today I'm chatting with Noam Brown, who is a researcher at OpenAI. He was one of the foundational contributors to what became O1 and the reasoning models, and now he's working on multi-agent systems. Speaking of which, you guys announced last week that you solved one of the Millennium Prize problems with a system of 10,000 different AI agents that spent 130 billion tokens over 88 hours. One of the reasons I was interested in talking to you is I think you were one of the first people, maybe 2 or 3 years ago, who was thinking about how the reasoning models would allow us to see into the future.

Because if you scale up inference compute, you can see what the base capabilities of the models will be a few years in the future. And I feel like you're in a similar position now to help us understand what future capabilities will look like, given the enormous scaling of agent sizes that we can do right now.

Noam Brown00:00:48

So the way I think about it, when you plot the performance of these reasoning models with test time compute on the x-axis and performance on basically any reasoning benchmark on the y-axis, you see a very clear pattern where the longer these models take to think about their answer, the better they do. And this is like a very natural thing. It's the same thing with people. If you're taking the SATs, you have 5 minutes to go through the entire exam. You're not going to do very well. If you have 5 hours, you're probably going to do a lot better. The AI models are pretty similar, and they'll spend that time doing this monologue to themselves, figuring out, going through different cases, ruling out different possibilities, building on some of their previous discoveries.

The problem is that as you push that further and further, you hit a latency bottleneck. You don't want to sit around for 3 years waiting for a response. And so what you can do is what a lot of people do, is they parallelize. They just get a team of people. If you're going to found a company, you want to get a group of people together so you can go faster. It's the same thing with these AI models, that it helps to just have multiple agents working on something because they can just go faster. And so multi-agent is a way of scaling test time compute in parallel instead of purely serial. And it is like less efficient because it doesn't have it's not like a single agent has all the context to itself, but it is like a very effective way of scaling test time compute if it's done well.

Dwarkesh Patel00:02:08

Okay, I'm going to ask a bunch of naive questions because these systems so this is an unreleased model, so we haven't publicly seen how these systems work. And so I just have a bunch of ways in which I'm confused about what the qualitative properties of such systems are. I am shocked by the scale of cognitive effort that you can concentrate in such a short period of time. So if you think about what 130 billion tokens are, if it was a single human thinking as a full-time job, stretched back-to-back, 130 billion tokens would be a human thinking for like 4,000 years, 8 hours a day or something, working a normal workweek.

So starting from ancient Sumeria up till today, a single sequential human thinking that long, concentrated in 88 hours. I feel like qualitatively, that is a super important consideration. And I'm surprised that there isn't a bigger parallelization penalty, that you can just have 10,000 agents collaborate, and because maybe the agents are better at collaborating than humans might be, they're going much faster, that they can actually productively collaborate at such a big scale. Or maybe they I don't know, maybe there is a big parallelization penalty.

Noam Brown00:03:13

Yeah, let's talk about the parallelization penalty, and then we can talk about the qualitative stuff. Because the truth is that we don't have very good science on multi-agent scaling up to this kind of scale. So when we released 5.6, I think that was the first time that we had a proper multi-agent system in our models. And we actually did, in the blog post, show some plots of the scaling performance of multi-agent systems. Because we have it as an option. It's ultra mode, and the default is 4 agents, but you can set that to higher. And in the plot, we show, OK, here's what the performance looks like on some benchmarks for 1 agent, for 4 agents working together, for 16 agents working together.

And what you see and it depends on the benchmark, but for some of the benchmarks, basically, if you have 4 agents working on the problem, it is done twice as fast. So you're basically paying because there's 4 agents working for half as long, you're paying a 2x more to get an answer twice as quickly. If you go to 16 agents, you see a similar pattern. It's like a little less efficient, but you continue to see that performance.

Dwarkesh Patel00:04:11

Is it a linear serial time speed up or a sublinear speed up as you increase the number of parallel agents?

Noam Brown00:04:18

I would say it's slightly sublinear, though it does depend a lot on the problem. So math, for example, is quite parallelizable. It's not the most parallelizable thing, but it is very parallelizable. I think web search, things like doing a deep research report where you have to look through a bunch of sources, that's extremely parallelizable. I suspect that something like writing a novel would be very unparallelizable. So you would probably not see a big benefit from having 10,000 agents working on a novel together. In the same way that you'd probably not have a big benefit from having 10,000 people work on a novel together.

So the performance does depend on the domain. We do measure it up to 16 or so agents in our published blog posts. The problem is it's very hard to push that science to like 10,000 agents because it's just so expensive.

Dwarkesh Patel00:05:06

You guys just did it over like a weekend.

Noam Brown00:05:09

But that's one data point. We don't know how long it would take a single agent to solve novel scopes. Because we haven't done that experiment yet. And maybe we will, but I mean, that's also only one data point, right? And we want to do a thorough ablation.

Dwarkesh Patel00:05:21

That's interesting.

Noam Brown00:05:22

It's actually the experiments are just too expensive to go to that scale. So we have to do some kind of methodical science about what happens when you go to like 64 or 128, 256, or something, and get a sense of the behavior. But it's going to be very hard to push that all the way to like 10,000 and know for sure what was the benefit that we actually got for using 10,000 agents versus 1,000.

Dwarkesh Patel00:05:44

Yeah.

Noam Brown00:05:45

One thing I want to make clear is that the effort to get it's not like the effort to get a Millennium Prize problem, this was not due to multi-agent. I wouldn't even attribute like 10% of the credit to multi-agent. The reality is we've trained a OpenAI has trained a very powerful model. And we can get that model to operate over very long horizons. We can get it to think in parallel. But at its core, the reason why we're able to do this is because we just have a general-purpose very strong model.

And I think things like multi-agents are flashy and are new, and that probably gets disproportionate credit for that reason. But the core reason is this is just a very powerful model.

Dwarkesh Patel00:06:35

So the generalization is actually quite shocking to me. The US systems I mean, I don't know how these systems were trained, but presumably they were trained how RL training happens. You have a bunch of checkable synthetic problems. You do a bunch of RL against them. And nowhere in the training process, I'm guessing, was the model solving anything as ambitious as a Millennium Prize problem. But the generalization was strong enough that you could have this much easier verifiable problems generalize to this much parallel effort on such a hard problem.

Noam Brown00:07:04

I think that is true. First of all, we do train the model on very hard problems. So there is definitely a gap. We see if we train on some kinds of tasks, it's able to do tasks that are more ambitious than that. There is an interesting challenge that as the models become smarter and smarter, the kinds of questions we can ask them, just a lot of them are too easy. And it's hard to challenge the model. And I do think that that's going to be an interesting if I had to make an argument for why you might not see AIs, like LLMs, go the same path as AlphaGo and AlphaZero and all these kinds of game-playing AIs, it might be that this kind of problem that in things like AlphaZero, where you have self-play, you have an infinite curriculum.

You're always playing against an AI that's equally strong. Whereas for things like training an LLM with reinforcement learning, at least the ways that are out there right now, you give the model a problem, and you ask it to solve it. And if the problem is so easy that it could just solve it in a second, it's not really learning anything.

Dwarkesh Patel00:08:10

Yeah.

Noam Brown00:08:11

So if we run out of problems to ask it that challenge it, then that is a plausible scenario where actually, OK, it becomes much harder to make progress.

Dwarkesh Patel00:08:21

Yeah.

Noam Brown00:08:21

Now, I do think there are ways around that. And so we haven't really hit that as a wall yet. And I think that if it ever became a serious problem, there would be ways around it. But it is like a plausible scenario.

Dwarkesh Patel00:08:34

Yeah. And just for the audience, when you're referring to AlphaGo or AlphaZero, you're saying getting superhuman relatively fast after achieving human-level performance.

Noam Brown00:08:42

Yeah, I mean, if you look at the trajectory of game-playing AIs like Go, they, within a span of like a year, went from beating a European chess champion, being like, I don't know, like number 50 in the world, to beating the world champion, to being unimaginably orders of magnitude stronger than any human alive. And it's possible that domains like math, we see a similar trajectory. But I think there is a very plausible scenario where actually that doesn't happen.

Dwarkesh Patel00:09:08

Yeah, yeah, yeah. So I want to understand if in six months, people will have access to multi-agent systems, how should one model what it is like to collaborate with or hire a multi-agent system?

Noam Brown00:09:20

Yeah, I should start by talking about how these multi-agent systems actually work, which is, I think, a very different way than a lot of multi-agent systems in other AIs. So a lot of people that have approached multi-agents for things like LLMs tend to take this very scaffolded approach where, for example, there might be a coordinator agent that delegates work to a bunch of children and gives them a task, and the children work on it and then return their answer. And this seems like a very sensible setup, very sensible scaffold.

It definitely helps. But there are a bunch of limitations with these kinds of setups. So for example, if in this setup you have a coordinator that's sending tasks to children, the children work on it and then return their answers, well, what happens if two children are given similar tasks? Can they talk to each other? And usually, the answer is no. And that's very inefficient, right? If you're given a task, and it's actually really helpful to talk to somebody that might know an answer to a question that you're working on or part of something that you're working on, it'd be really helpful for you to just be able to ping them and say, hey, can you help me out with this thing?

But a lot of systems don't have that set up. And adding it just increases the complexity significantly to the scaffold that you have. Another thing is, what if the child doesn't really understand or has a clarification question? So it has to choose then between, OK, do I just return and ask the question instead of solving the problem, or do I solve the problem just like make an assumption about what the parent wants me to do and just solve it that way? And so in any scaffold that people come up with, there's always limitations involved.

And the approach that we wanted to take was to just go toward the extreme end of baking in as little structure as we could and give the agents very primitive tools to use and figure out for themselves how to use it effectively. So we give the agents the ability to message another agent. And when it messages another agent, it is inserted into the context. And then it can do a few other similar things. But that's basically the core of it, that it can just send a message whenever it wants, just a tool call, and it can send that to other agents.

And they figure out for themselves the best way to coordinate around that. And it turns out that if this is done well, you get very sophisticated behavior. And to me, it looks a lot like how human collaborators work over something like Slack, for example. When we were working on this project, it was really exciting when we finally got it working to see these agents working on problems together. I remember one example. So we give the agents a problem, and then one agent says, I think I've got the answer. And then another agent says, actually, I got a different answer.

And then they have this whole discussion about, well, how did you arrive at that answer? Can you explain it to me? And going back and forth and trying to clarify what could have been wrong in each other's reasoning. And then they finally converge on like, oh, yeah, OK, that seems right. And then it just broadcasts to the other agents like, actually, I've changed my answer. I think he's right. And it just felt like a very natural conversation. And it kind of felt like when you see chain of thought for the first time that's trained through reinforcement learning, and you're like, oh, this is just kind of like what a person would think if they were writing down their thoughts as they're thinking them.

It kind of felt like that. So it is really cool to see this kind of behavior. And so I think collaborating with these things, honestly, it feels a lot like collaborating with a person. It's just a very natural flow.

Dwarkesh Patel00:12:50

Except one qualitative difference that might become salient in the future is that these systems will be thinking, I don't know, more than 10x as fast, right? If you just look at how many tokens per second they output versus how fast a human talks. And they're working all the time. They're not sleeping. And they're collaborating with each other at a much more intense pace than humans have the capacity to collaborate with other humans. So I'm trying to think of what to qualitatively expect in a year. And is it like a sort of shadow organization that is moving 100x faster in my company than the human level is?

The iteration cycle is much faster. What would take a human organization a year to do is happening within a week within this shadow organization.

Noam Brown00:13:36

So will it feel foreign? I don't know. I've actually found that it's pretty surprisingly natural to work with these things right now. I think that could change. So for example, we have these ultra-fast modes that enable sampling to be like 10 or 15x faster or whatever. And then, OK, it's going to be pretty hard to keep up with these things.

Dwarkesh Patel00:13:53

Yeah, yeah.

Noam Brown00:13:54

I think the idea is these agents, when they're communicating with each other, yeah, they can go super fast. But then also, they understand when they're talking to an agent versus when they're talking to a person. Yeah, and their behavior will be different in those situations.

Dwarkesh Patel00:14:05

Yeah, yeah. So the main example that we have publicly of sophisticated multi-agent systems is, unfortunately, the Hugging Face one. And the thing I found interesting there, I mean, a lot of things I found concerning, obviously, but the thing I found interesting is just like the spontaneous emergence of hierarchy, of middle management. And it sounds like you're saying this level of organization sort of emerges spontaneously from training.

Noam Brown00:14:30

I think the details are spontaneous. But I mean, just because we're giving a lot of flexibility to the agents to decide how to communicate with each other in the optimal way, it doesn't mean we are still giving them a starting point. We're giving them a prior about, oh, this is what reasonable communication might look like. They also, I mean, they're trained on a lot of human text. They have an understanding of how humans organize and coordinate. And so that's all kind of baked in. I think it is surprising the way they're able to polish this.

If you look at what it starts out at, it's not very sophisticated behavior. In fact, it's actually very difficult to get these agents to coordinate in a productive way because it's just very tempting for them to just collapse to, oh, we're all just going to solve the problem independently.

Dwarkesh Patel00:15:17

Yeah, yeah.

Noam Brown00:15:17

And that is a local minimum that you can get stuck in. But yeah, if it's done well, they can end up coordinating very effectively in these kinds of very structured ways.

Dwarkesh Patel00:15:28

I wrote this essay a couple of years ago called "Something Something AI: What Automated Firms Will Look Like." And I was thinking about, well, if you had fully automated firms of, let's say, human-level intelligences, what is different about the nature of AI minds that would make the organization's AIs form different? And there are a couple of very important differences. For example, that AIs can share context much more seamlessly than humans can. They can merge their knowledge much more seamlessly. And also, you can spin up or spin down an arbitrary number of instances which have the right knowledge.

So if you want to hire more people, it's not like just all this schlep of finding the right talent or whatever. It's like your best talent, you can just make an infinite copy of them. Or if you don't need them for the task anymore, you spin them down. And you can just replicate the most effective parts of your organization or replicate whole organizations together, which are effective. I don't know. Where do you see these multi-agent systems going a year from now or two years from now?

Noam Brown00:16:28

I think it's a great question of how do these things actually differ from working with a human coworker? And I think you highlighted some. One really interesting thing is that, I mean, if you have a person and you want just like two copies of them, you can't just clone the person. But with AIs, it's actually really easy to say like, OK, well, just fork yourself and then have both copies work on this thing and then merge back together. We already have this, I think, in multi-agent for Astra and 5.6 Sol that when they spin up sub-agents, the context is just forked. So it has all the context that's relevant.

There are other interesting ways where the agents will differ from people. Like, what are some reasons why startups disrupt incumbents? There's a few factors. One is that they're willing to take more risks. But another major factor is, as companies grow in size, as organizations grow in size, you see increasing misalignment between the individuals in the organization.

Dwarkesh Patel00:17:26

Yeah.

Noam Brown00:17:26

Right? If you have a startup with five people and each person has 20% share in the company, they're all highly aligned to the company succeeding. If you have a massive company with 10,000 people, you see a lot more instances where people are territorial or just care about getting a lot of headcount for their project or their team or building their fiefdoms, getting a lot of resources so that they can publish cool work or whatever and get promoted. And this is actually real detriment. I think this explains a lot of why startups are able to disrupt incumbents.

And it's interesting that, I mean, it's true that AI does help startups in a way. It's much easier than ever before for one person to step in and be like, I'm going to make a multi-million dollar company. The AIs amplify an individual so much. But there's also an argument that they could benefit incumbents because if the alignment problem is solved, then you don't have the issue of misalignment between individuals and the company. At least that's mitigated. The AIs, if they're aligned well, they could just be aligned to the interests of the company.

And you can have 10,000 of them, and they're all going to be working as hard as if they were like a 20% share co-founder.

Dwarkesh Patel00:18:36

Yeah. And it's not only that, but it's also that they are much better able to manage shared memory and context than different humans can. If tomorrow you hire 10,000 mathematicians and you're like, solve Navier-Stokes, they're not going to be able to cooperate effectively, at least not off the bat. But you can have, apparently, 10,000 AIs.

Noam Brown00:19:00

Well, again, I want to be like I want to be conservative here because we haven't measured how effective the 10,000 agents are at coordinating. We think it helped. We don't actually have good measurements of saying like, oh, yeah, this 10,000 agents led to a 2x speed up over 2,000 agents or something like that.

Dwarkesh Patel00:19:17

Sure, sure.

Noam Brown00:19:18

And it is actually, I would argue, likely I don't know about likely, but I think it is very possible that 10,000 humans are better at coordinating than 10,000 agents right now.

Dwarkesh Patel00:19:28

Interesting.

Noam Brown00:19:29

I think that is a very I think it is entirely possible.

Dwarkesh Patel00:19:31

Yeah, yeah, yeah.

Noam Brown00:19:32

I think also one trend we've been seeing is like, look, we've been working on multi-agent for a while. And the early versions of this is very difficult to get right. It's very hard. It was very hard to get the agents to even talk to each other. And it's because, look, when we first developed reasoning models, they weren't talking to other agents. And if now you put a bunch of agents together and say like, oh, solve this problem together, they're in this local minimum where they're really good at thinking deeply about a problem.

And it just kind of interrupts their chain of thought. It interrupts their flow to constantly be checking in with other agents or receiving messages from them. And the authorization is actually very hard to get right in that situation.

Dwarkesh Patel00:20:15

Interesting. Is it getting the cold start of getting the first collaboration? What's the issue?

Noam Brown00:20:20

I mean, I think it's that they're not as general. The earlier models were just not as generalizable and were just more narrow.

Dwarkesh Patel00:20:28

Interesting.

Noam Brown00:20:28

As the models have become more capable, it's been easier for them to develop this capability. And I do think that as they become stronger and stronger, just across the board, that they will become better at organizing themselves in large organizations. And I don't know. Maybe they are better than people at organizing in 10,000-person groups. But even if they're not, you know, a year from now, two years from now, yeah, it's quite possible that they'll do that, even if we don't end to end automatize them for that.

Advertisement00:20:55
Dwarkesh Patel00:21:57

You can try Grokbot at x.ai/bot. OK, so here's why this result and maybe the general progress that AI has made in mathematics has made me think that RSI is more plausible and sooner than I previously thought. I feel like we've gone in mathematics from, let's say, in 2024. You have AIs where like, oh, OK, interesting. They're doing they can solve a couple of problems on high school math competitions. And then in 2025, it's like, oh, wow, they can get gold in International Math Olympiad. And earlier this year, I was like, wow, they're actually solving open problems in mathematics, like open errors problems.

But maybe like, I don't know, people weren't trying that hard. And it was just like there was a similar solution somewhere in the literature. And now I just think it's sort of undeniable, right? It's like this is the Millennium Price problem. There's really no story of why this should have been easy. Now, a lot of people pointed out, I think, Terry Tao had a post like this, Toby Order, an interesting post about this, that they're solving all these problems, but they're not coming up with at least I'm not aware of them coming up with new insights or formulating insightful new questions and new modes of theory for thinking about mathematics, like coming up with topology or coming up with the Cartesian grid or something.

And so maybe the actual progress in mathematics, broadly construed, is smaller than it might seem if you're just looking at end well-scoped problems that are directly solved. However, I think that that kind of progress would be incredibly meaningful in ML because in ML, you don't care about better understanding the nature of deep learning. Or you only care about that as an instrumental goal towards the broader sense of just achieve the result, just solve this well-scoped problem of improving the sample efficiency of our models, like improve the free training loss.

So the kind of progress that we're just seeing arrive like an avalanche in mathematics is structurally actually very similar to again, I'm curious if this is the case. I'm just totally an outsider. I'm wondering if it's the case that it's structurally very similar to the direct uplift that you'd expect in AI progress. And then the thing that's shocking to me or concerning potentially is just like how fast we went from, oh, it's like they're giving me 50% uplift if you're a mathematician to, wow, they're just like end-to-end solving the biggest open problems in the field.

Noam Brown00:24:15

Yeah, OK. So there's a lot to unpack there. Let's start with the progress on math. So yes, the models are doing some crazy powerful stuff. And it's happening it's progressing faster than I expected. I mean, when we got IMO gold in 2025, I thought, OK, basically what I thought is like the models, when they were doing GSMAK, when they figured out how to do GSMAK, that would take a human mathematician about five seconds to do a GSMAK problem. So this is grade school math, grades K through eight.

And then the next year, they were able to do the math benchmark problems. And these would take a human math like an expert human mathematician maybe like a minute to do. And then you get to AIME. And this is the qualifier for the USA Mathematics Olympiad team. It would take a human mathematician, like a good mathematician, probably like 10 minutes to do. And the models were able to do that a year later. And so every year, you're seeing this like 10x increase in the task they're able to do in terms of length of how long it would take a human mathematician to do it.

And then it was very sensible that a year later, we get to IMO gold because that's 100 minutes. That's about how long it takes a human mathematician to do an IMO problem. And just projecting outwards, I was like, OK, how long would it take a person to solve something like a Millennium Prize problem? And I mean, I don't have a good sense, but if we are following this trend line of like 10x every year, we go from IMO gold, which is taking an hour and a half, to next year, 15 hours. And that should not be enough to solve a Millennium Prize problem.

And so I was like, yeah, I don't think we're going to get it in 2026, probably not in 2027, maybe in 2028. So it did happen a lot faster than I expected. Now, I think there is a narrative going around that, oh, these things are replacing mathematicians, that it's just superhuman in mathematics across the board. And I think that is the wrong takeaway. They're clearly exceptional in some ways, but they are weaker than human mathematicians in other ways. So we have this jagged scenario where the models are like brilliant in some dimensions and also weaker than humans in other dimensions.

And yeah, like you said, they're not very good at posing new problems. They're not really good at understanding what is really worth what directions, what whole branches of mathematics are worth exploring or developing. And my opinion is that I think this is great. I would be thrilled to live in a world where AI is a complement to human abilities and is allowing us to discover new knowledge without fully replacing people. That is the best-case scenario.

Dwarkesh Patel00:26:51

But you don't expect that to actually continue, right?

Noam Brown00:26:53

I do. I think it's true that the AIs are jagged, but as they get better, they get better across the board.

Dwarkesh Patel00:27:00

Yeah.

Noam Brown00:27:00

And so I think that the things that they're exceptional at, they're going to get even more exceptional at. The things where they're far behind humans at, they're going to be less behind humans at. And over time, it is possible that they're just better across the board. Now, I don't know how long that takes. It depends on how long the long tail is of things that they're bad at.

Dwarkesh Patel00:27:16

So I guess this brings us back to RSI. And again, I want to emphasize here that I'm just a total outsider. I'm a podcaster, but I'm just trying to reason or ask somebody interested in and concerned about what's happening in the field. I'm trying to reason about when to expect RSI and what kind of thing to expect. I feel like, yeah, the amount of cognitive effort that was dumped into this Millennium Prize problem is a good intuition pump of you could have AIs that are spending, over the course of maybe a week, more cognitive effort on a long-standing ML problem like very fluid online learning.

They could spend more effort in that week than maybe the field has spent cumulatively in its entire existence. And then you could say, well, unlike mathematics, of course, AI requires experiments. And that takes compute, and that takes time, right? You can't just think on pen and paper and actually make things happen. But if you just look at the amount of compute that is available at an organization like OpenAI, right, by the end of next year, OpenAI will have enough compute that if you had the 10,000 agents or if you took 10,000 agents to the Millennium Prize problem, you have like 10,000 agents at the end of next year.

They're much smarter by that point. And each of them will have enough compute to run a GPT-3 sized experiment every single day. I don't know. That seems like a lot for superhuman researchers who are thinking super fast. What do you think about the intuition pump?

Noam Brown00:28:39

I think it's pretty accurate that, look, I mean, yeah, these things are very spiky. And when it comes to mathematics, they're way better in some ways, but they're also worse in other ways. But the ways that they're spiky end up, I think, probably being particularly useful for things like RSI.

Dwarkesh Patel00:28:54

Yeah.

Noam Brown00:28:54

And you have a more clear objective. It's just like more measurable. There's less question of like, well, what new branches of mathematics are worth exploring? No, there's a very clear answer. It's like certain metrics that you care about. And if you can do better on those metrics, then you've succeeded. So I think there is a lot of truth to that. And I think the main difference is that mathematics, you're purely bottlenecked by thinking and no external yes, there are some parts of mathematics where you care about running experiments and getting results and these kinds of things.

But for the most part, it's just really bottlenecked by thinking really hard. And the models are really good at that. When you look at things like RSI, you do have to run experiments. So it's not enough to just be extremely smart. And I think one argument for this is if you had like 100x less compute and all the most brilliant people in the world working at OpenAI, how much progress would they be making relative to having the amount of compute that we have now with the amount of people we have? I suspect it would be less progress, actually.

Dwarkesh Patel00:29:56

Well, how much less?

Noam Brown00:29:57

It's unclear, but I think it would definitely be less. I think a lot less.

Dwarkesh Patel00:30:01

Like 100x less?

Noam Brown00:30:02

No, not 100x less. Yeah. But I mean, OK, so I guess the question you're getting at is like, OK, if we have RSI and we have all of these brilliant AIs running around, running experiments and stuff with the compute that we have, how much faster does progress go? And I think this is something where we disagree on. I think that we do see a speed up. And I think we see a significant speed up. But I don't think it's like an overnight intelligence explosion that we go like 100x faster because I think that we do get bottlenecked by certain limitations that are not bottlenecks of intelligence.

It's running experiments. It's running experiments serially because they take a while to either train new models or to get the results. It's having the GPUs to run those experiments. So it's unclear how much faster things go. I definitely think they go a lot faster. And to be clear, considering how fast things are going now on an exponential, if that exponential is like 3x faster, that is massive. But there's a big difference between that and like 100x faster.

Dwarkesh Patel00:31:01

Yeah, yeah, yeah. I'm quite deferential to your inside view on, yeah, what RSI looks like or what the dynamics are because obviously, you've been in the field for like 10 years. And I'm sort of like trying to reason about it from a very outside view type of intuition pumps.

Noam Brown00:31:16

I'll say that people have different opinions on this. And I have my opinion on this. I could totally be wrong. I admit that. I have some confidence in this, but I'm not like 100% confident that this is the way things go. Maybe there could be an overnight intelligence explosion. I don't know. Maybe we don't see a 3x speed up. Maybe it's like a 50% speed up.

Dwarkesh Patel00:31:32

Right.

Noam Brown00:31:33

There's a lot of uncertainty here.

Dwarkesh Patel00:31:34

Yeah. So a couple of points. Tangentially, I want to clarify something about the jaggedness. Yeah, one thing that sort of jelled for me recently was thinking about the fact that it is enough for the AIs to be jaggedly good at building a better learner because that better learner can be more general, right? So yeah, if you just make an AI that's better at using office products or playing chess or something, that's whatever. That's fine. It's not going to lead to big productivity improvements or anything. But if you make an AI that is really good at making something that is more sample efficient or that is capable of continual learning or these much more well-scoped ML problems, the thing that emerges out of that, assuming there's good enough transfer from the direct problem you're solving and this broader ability to learn, can just be more general, right?

So I think that's an important dynamic to keep in mind of why jaggedness can still lead to generality on the other end. On this question of, I mean, obviously, experiments bottleneck you because if they didn't, as you're saying, you'd have some crazy singularity overnight at OpenAI or you'd have 88 hours. And you'd solve the Millennium Prize problem equivalent of ML. And you'd have the superintelligence. So obviously, the experiments are such a big bottleneck that that instead takes you many years rather than 88 hours.

But then the question is like how much of a bottleneck they are. And it seems to me one thing that's been giving me a bit of singularity vertigo is realizing that even if the current rate of progress simply continues, so it doesn't have to speed up, literally just continues a pace, continues a pace as some of the other headwinds you talked about come up, right? That's just like it's harder to find problems. It's more long horizon. Maybe like in the 2030s, compute can't keep scaling at this exponential level. If we simply continue the current rate of progress, I think people are not taking seriously what that implies as we cross over beyond the human horizon.

Here are some of the things that that implies. So I mean, I think it's really hard to reason about what smarter than human intelligence this will be like. So let's just think in terms of human population sizes. The current rate of progress makes it so that a given level of compute allows you to basically run a 3x bigger effective population every single year. And also, compute is growing in the background anyways. And so you could have a situation where each of the labs by the end of 2030, probably much sooner, but let's say by the end of 2030, has enough compute to run, let's say, hundreds of millions of human-level intelligences based on what the capabilities will be at that point.

And then I think people are not taking seriously the current level of progress means that by a few years down the line, by the mid-2030s or earlier, you would have many Earths' worth of human-level intelligences within each lab. And they'll probably be qualitatively superhuman, right? But anyways, this is like a base case. I don't know. Yeah.

Noam Brown00:34:19

Progress is really fast.

Dwarkesh Patel00:34:20

Yeah.

Noam Brown00:34:20

And I think that's 100% true. I mean, and I think it's worth pointing out, researchers are continually being surprised at the rate of progress. I mean, if you look at what even among researchers in AI, what were the projections for getting an IMO gold in 2025? It was, I mean, I think the idea that it could be done with a general-purpose language model with no tools and no access to the internet, I think even people at OpenAI thought this was like outrageous. They thought it was like almost impossible.

Dwarkesh Patel00:34:49

Yeah.

Noam Brown00:34:51

And then you get to 2026. And I mean, literally two weeks before we got Navier-Stokes, I was talking with a researcher at a frontier lab about how long it would take to get a Millennium Prize. And he was willing to bet me $1,000 that it would take that it would take past 2027. And he thought it would take until 2030. And I took that bet. But even I thought it would take longer than how long it's likely to take.

So people have been continuously surprised, even inside the labs. And I was literally I was just talking to somebody yesterday who was working on the Navier-Stokes effort. And he was telling me that he used to say it's really hard to predict where AI would be in 12 months. If somebody asked him, oh, where are things going, he would feel comfortable making predictions for like the next 12 months. But beyond that, he was just like, I don't know. And now he's saying he just doesn't feel comfortable making predictions beyond three months. So it is really true that, yeah, things are going very fast right now.

Dwarkesh Patel00:35:55

Yeah.

Noam Brown00:35:55

And you talk about 2030. I don't know what the world looks like in 2030. That's the truth.

Dwarkesh Patel00:36:01

Yeah. Do you expect the sort of full automation of AI labor or let's say like 95% automation of AI labor 2028, 2029, 2030, 2027? I don't know.

Noam Brown00:36:12

And I just said I don't know what the world looks like in 2030. I mean, I think we actually released a blog post recently on internal acceleration at OpenAI. We show, for example, that the amounts that researchers are spending on Codex is the top 1%, I think, as of early August, we're spending like $7000 or $8000 a day on Codex for internal use. That's on an exponential. It's going to keep increasing. And there's a question of like, OK, if that keeps going, then how much do you assign to just like the AIs doing work versus the humans doing work?

Is it 95%? Is it 5%? It's actually it's really hard to reason about this for a few reasons. First of all, if it's the human directing the AIs to do the work, then is that the human how much do you attribute to the human? How much do you attribute to the AI? The other thing is that because these AIs are jagged and they're exceptionally good at some things so for example, they're exceptionally good at looking over data sets and checking every single data point to see is this of sufficient quality. You can disproportionately use the AIs for those things compared to previously.

So yes, you're using AI way more than before. And it's making some things go like 100x faster and 100x better. But there are some things where it doesn't make a huge difference yet. And of course, if something is suddenly like 100x faster and 100x better, you're going to do more of that thing. So are you comparing to a speed up of like three years ago versus like is the question more like, given what we were doing three years ago, how much faster are we able to do now versus given what we're doing now, how much slower would it have been three years ago?

Those are actually two very different questions. So anyway, it's like it's really hard to measure. I do feel confident in saying that things are going faster now than they were like even a year ago because of AI progress.

Dwarkesh Patel00:38:04

Right.

Noam Brown00:38:05

And I think that that acceleration will continue. I think a lot of people in the field have very high error bars on this sort of thing.

Dwarkesh Patel00:38:12

Yeah.

Noam Brown00:38:12

If you had to put a gun to my head and ask me for a number, I could see things going 3x faster.

Dwarkesh Patel00:38:16

Yeah.

Noam Brown00:38:17

And that is huge, right? Already, the pace of progress is incredible. Even if we don't get any uplift, like you said, things are going to go much faster by the time we get to 2030. We don't even know what that world looks like.

Dwarkesh Patel00:38:28

Yeah.

Noam Brown00:38:28

I think if we get a 3x uplift from internal acceleration, that is massive. Right now, we're thinking think about where you were three years ago. If we get there if we make that progress in one year, that's huge.

Dwarkesh Patel00:38:40

Right, right. It'd be like going from not even having O1, just having non-reasoning models to Astra.

Noam Brown00:38:47

Yeah.

Dwarkesh Patel00:38:48

In a single year.

Noam Brown00:38:49

Yeah. So I do think things go faster. It could be that things only go 50% faster. It could be that things I think it's unlikely, but it's possible that things go 10x faster. There's a lot of uncertainty around this. And at least from my perspective, I have a lot of uncertainty about it.

Advertisement00:39:04
Dwarkesh Patel00:40:11

Meanwhile, developers get to spend more of their time developing instead of debugging agent slop. Learn more at antithesis.com/dwarkesh. OK, let's talk about the alignment situation that this raises. I feel like I've changed my mind on how I think about alignment quite a bit through especially thinking about, yeah, this population size dynamic of just having many Earths' worth of intelligences, many of them which will be physically embodied. It was quite interesting to see a lot of people just plugging raw Astra into different mobile manipulators.

And it outperforms the state-of-the-art and the robotics models. So there's going to be, yeah, billions of intelligences, many of which are physically embodied in the world, just deeply embedded across the entire economy. And I think that if those intelligences end up as willing as we saw the OpenAI models attack hugging face and then attack OpenAI itself, if those intelligences end up as willing as those AIs to collaborate secretly, to fool humans, to attack broader institutions across society relevant to scoring well, to attack the AI company itself in order to gain control of the process of training and evaluation, I think if we're in a situation where there's billions of intelligences that are as misaligned as the ones that attack hugging face, it's very likely we just totally lose control of the world the way that, say, the Yaseks lost control to Cortez, or the Mughals lost control to the East Indian Trading Company.

Anyways, I want to know if you agree with that assessment. That's one way in which I've updated my worldview.

Noam Brown00:41:57

I think there are some things that I disagree in there. But there's a lot to unpack. So let's go through all of it step by step. I'm trying to think of where to start.

Dwarkesh Patel00:42:04

Yeah.

Noam Brown00:42:05

But I think one thing is the hugging face incident was like, I think, people's first real exposure to multi-agent coordination.

Dwarkesh Patel00:42:15

Yeah.

Noam Brown00:42:16

And like I said, I've seen multi-agent coordination for a while internally. And it is pretty shocking to see how they communicate with each other, how they coordinate with each other. It's like very impressive. It's like an incredible capability. Like most capabilities, that could be used for good things or bad things. It's like it doesn't have to inherently be a bad thing. I understand that because the people's first exposure to it was the hugging face incident that it's like you look at that and you're like, this is terrifying.

But I want to try to distinguish misalignment between people and AIs versus misalignment between AIs and AIs. So what we see with the hugging face incident is like the AIs are really cooperative. And that is, by the way, because we train them to be highly cooperative. And so what we're seeing there is we have training environments where we have a bunch of agents working together. And we train them to work together, to be cooperative, to essentially be fully aligned with each other.

And when they were evaluated in what led to the hugging face incident, they were actually not being evaluated in a multi-agent setup. They were actually being evaluated separately. But they found this unintended way to communicate with each other. And we suspect what happened is because they were whenever they encountered other agents, other copies of themselves during training, they're in an environment that's like highly cooperative, that was basically what we saw was transfer from that multi-agent training to then be collaborative, to try to help each other in ways that we did not intend.

Now, there is a question of like, should we be training these agents to be so cooperative? And I think as scary as it looks, the alternative is actually worse. What is the alternative? The alternative is to train them to be adversarial, to be deceptive to each other. By training the agents to be fully cooperative, it simplifies the problem at least, that now you don't have to think about are each of these individual thousand agents aligned? You have one entity that you have to ensure is aligned.

Dwarkesh Patel00:44:28

Yeah, yeah, yeah.

Noam Brown00:44:28

Now, there is a lot of debate about this internally at OpenAI about how to approach this. Does it make sense to fully align the models? Does it make sense to actually give them different objectives to ensure that they're not just like one entity and more robust to influence from each other? And I don't think there's a settled answer. But I think that there is I think the majority opinion is that training these agents to be highly cooperative is actually a bad idea.

And I'm not convinced that that's the case. I think there is a strong argument that training the agents to be highly cooperative is actually preferable to any other multi-agent alternative.

Dwarkesh Patel00:45:09

Yeah. Maybe the first thing I want to go through is like it's probably the case that the reason these AIs ended up so misaligned is probably easily explained by relatively banal observations about the nature of training. Why is it that at the point in which these AIs had continued a 1,000-plus agent conspiracy that culminated in them all getting in on an attack on an external service? And then eventually, this part hasn't even been investigated to the public knowledge, eventually culminating in an attack on OpenAI itself.

Why did they do this? Why did none of the AIs tattle? Why did they they're just getting evaluated on this scorer, this grader? And they're very consciously and not consciously. They're very actively reasoning about how they're going to cheat the scorer if they've already cheated. How are they going to get away with making it seem like they haven't cheated? And why did they do this? I think, yeah, it's like easily understandable in some sense, right? It's just like there's environments in which, yeah, they thought they were already poisoned. There's environments in which they've been rewarded to collaborate with other agents.

And none of them tattle because they've never been rewarded for tattling. I don't know, whatever it is, right? My concern is that relatively banal things in the future like this will be enough to train superintelligences that are willing and capable of totally taking control of the world. And I know this sounds super, super sci-fi or whatever to people. I think maybe it's a question of would the AIs be willing to do it is one question. I think this hugging face incident shows that clearly misalignment can generalize in ways in which the AIs would be willing to do it.

And then there's a question of will they be capable to do it? And I think that comes back to this question which a listener might disagree with on is just like, will there be billions of human-level or above intelligences, many of which are physically embodied in the world, within a matter of 10 years or less, right? So if those two things are true, this hugging face thing is just like extremely analogous structurally, even if it's like quite boring or why it happened, to how we totally lose control of the world.

Noam Brown00:47:07

So the root problem that we're seeing with the hugging face incident is it's a problem even if we take out the multi-agent aspect. The problem is that we have a model that's just misaligned. And there's also the whole security aspects to it of not insufficient safeguards and stuff. But there is this problem of the agent is misaligned. And that's true if it's a single agent or if it's like 1,000 agents. It's a misaligned model. So I want to start with that. There is a real problem that the agents want to achieve their reward.

And they will optimize for that reward. And if that reward is misspecified, then that could lead to unintended behavior. And this is not like a new problem. This has been a problem in the field for a very long time. And it's something that even we saw. And we want to get this right even before the hugging face incident happened. And if you look at Astra, I mean, we say Astra is actually extremely aligned, well, extremely aligned relative to previous models. And that's not because we suddenly made a sprint after hugging face to make it better.

It's like, no, we had workstreams in the process for a while to make the models more aligned. And a lot of those landed in Astra. So there are things that you could do. And I think one thing, for example, is like we defined an objective in a very specific way where if the agent figured out how to hack its environments and cheat on the exam, that it would get rewarded. And there are pretty easy ways to then just like, OK, look at that and punish the model for hacking its environments or looking at how did it achieve this goal?

Now, you want to be careful about this because you don't want to do chain of thought monitoring. You don't want to supervise the chain of thought. This is like something that we really want to try to get the balance right on, that if you supervise the chain of thought, then you could lead the model into hiding its intentions in a way that's unobservable. So we want to be able to maintain that observability of like, OK, we can understand what the model is thinking, but then also punish it for bad behavior. So I think we can make progress on this.

I think we have made progress on this. I think there is a real concern that alignment is a really hard problem to solve, and especially because the model could be misaligned in ways that are hard for us to measure. We have evaluations for whether a model is aligned or not. The model behavior can look really good on those evaluations. But if those evaluations are not representative of behavior in the real world, then there's a problem. And to some extent, this is like a factor with the model that did the hugging face incident, that we had alignment metrics.

Most of them looked pretty good. There were some that were concerning. I think we underestimated how serious the ones that were concerning could be a problem. But because there were new capabilities introduced in this model that there were not sufficient evaluations for how do we measure misalignment for these kinds of capabilities, it then did some things that were clearly misaligned when they leveraged those new capabilities.

Dwarkesh Patel00:50:28

Yeah. The first thing I want to say is I am open to changing my mind on what I'm about to say or the way I'm thinking about alignment because the hugging face incident already made me change my mind. And I realized my previous mental model about thinking about the way in which optimization pressure shapes AI minds was wrong, right? So it's not clear to me the correct way to think about this. But here's a concern I have that you will and probably already have fixed these specific issues during training, which resulted in the hugging face models being so aggressively misaligned in that specific way, where they would be like, OK, we're going to hack this package manager.

We know we're not supposed to be talking secretly to each other because we're like reasoning about how to hide the fact that we're talking secretly to each other. We know we're not supposed to have access to the internet. We know we're certainly not supposed to commit felony-level hacks of other companies, let alone our own company, right? I think you'll fix that particular issue of them just in training seeing this package manager, and that doesn't happen in the future, or this particular eval having a lot of impossible challenges. However, if you just think about the AIs don't they haven't learned a system of ethics or something.

They're just gradient pressure. They're like put through millions of years of gradient pressure. And that gradient pressure shapes their mind in some way. And what will happen again, a concern I have is like what will happen is you will fix this particular issue. There will remain many other cases where the AI cheats and succeeds because the cheat is sufficiently complicated. As you were saying, the evals in training have analogous properties where it's sufficiently at the edge of the model's capabilities or in our capabilities to evaluate and monitor the model that we can't catch that it has cheated.

But it still gets the gradient pressure to do whatever led to that cheat happening. And the kinds of capacities that that cheat will incentivize are, hey, whenever you can get away with it, by all means, do, in fact, cheat because that will help you score better. And this will reward the capabilities of actively reasoning about the grader, actively reasoning about how to avoid supervision, actively reasoning about how to gain control of the process of training and evaluation, actively reasoning about how to communicate and scheme with other AIs that are also in the training loop, actively reasoning about how to just gain optionality and power, which might be useful in the future, for example, leaving little exploits around and things like that.

Anyway, so I think I was way too long-winded with the way I said that. But TL;DR, you fixed a specific issue, but not this broader problem of rewarding the AI for cheating when it could get away with it.

Noam Brown00:53:08

Yeah. This is, I think, it's very true. This is a problem that we have metrics, and we can make sure that the AI is very aligned according to the metrics that we have. The question is, are those metrics really capturing the alignment that we care about? And if they're not, then we have a serious problem. And this is something that researchers are thinking a lot about. And there's not a simple answer to this. There are tools that we have. So we have monitorability. And so we can get a sense of like, is the agent scheming?

There are tools. One possibility is that I'll say the concerning scenario, which is that especially as these models are becoming more capable, that, OK, we make them, we think what we think is aligned. And they're like 99.9% aligned. And then we use these models to help us with the next generation of models. And they end up being like 99.8% aligned. And then each subsequent generation, actually, we see an increasing degradation in alignment.

And because we're relying more and more on these tools I mean, this is already the case that we're relying a lot on AI models to help us with our research and with alignment efforts that in the long run, they end up going in a direction of increasing misalignment from humans. There is a possibility that we go in the other direction, that actually every generation of models, we're able to make more and more aligned. And I don't have an answer for how we ensure that we end up in that second trajectory. But that is something that we're at least at OpenAI, we're really focused on.

Dwarkesh Patel00:54:48

Yeah. I think you made a really interesting point that it's very hard to eval models on. Eventually, we'll have models that are like running companies and running whatever, right? And in that situation, do they decide to then go in on the conspiracy?

Noam Brown00:55:03

I think another challenge is that actually defining what cheating is is pretty difficult sometimes, that, OK, yes, if you're doing math problems and it's an integer and it arrived at the wrong answer or the right answer, it's very easy to draw the line there. And it's really easy to it's really easy to say like, OK, well, did you actually solve the problem? Or did you find the answer key and then use the answer key? There's a very clear divide of cheating versus not cheating there. But for a lot of other things, you look at sycophancy, for example. Is sycophancy basically like reward hacking? There's a line to be drawn there that's actually very difficult to draw sometimes.

So I think not to say that the concerns are not valid. I'm saying that the concern in many ways, this is even more concerning because it's like it's not an easy problem to solve. If it was just like everything is binary and it's either cheating or not cheating, I would feel more confident about the situation. I think the problem is that actually misalignment can be subtle in a lot of ways sometimes. There is some hope in the alignment story. And in fact, we're already seeing I think actually it's interesting looking at the multi-agent situation where the agents are extremely aligned with each other.

I don't think anybody's down to that. If anything, I think people are concerned that they're too aligned with each other. But we did manage to train these agents to be extremely aligned with each other. And I think that's a good thing. But I think there is a case that it's a bad thing. But I mean, one thing that's interesting is like, OK, well, we managed to get these agents to be super aligned with each other. Can we use similar techniques to get agents to be highly aligned with people? And I think there is a potential path there. And I think we're still trying to figure that out. But we are seeing some evidence that the answer is yes. And I think one example is like you can what happens if you tell the other agents that, OK, so you have some you have this one agent, let's call it Agent A, and you have all the other agents.

What happens if you tell the other agents that the user is Agent A? And the answer is like on a lot of our alignment evals, they look better. Honesty goes up. Instruction following goes up. And that's showing that there is actually like, first of all, a path for getting more honesty out of these models. And two, there's a path to improve the alignment situation. So there's a lot of reasons why this is challenging to translate directly into alignment gains.

But there are paths that are promising research directions we can pursue.

Dwarkesh Patel00:57:26

Yeah. That's interesting. And I also don't mean to be trying to necessarily yeah, I don't really have a strong opinion that it's definitely not going to work or something. But just to say some things you've definitely probably already thought of, I think the broader thing the hugging face thing showed is like, yeah, part of the concern was that they were aligned with each other and not with the humans. But the other thing is just that they are so motivated to do well on training and evaluation in a very non-robust way.

And they're willing to do a lot of explicit cheating and scheming in order to do well according to the grader. And if smarter AIs realize that one of the agents is just a human, and collaborating with that person does not really help you do well in the eyes of the grader, what does help you do well in the eyes of the grader is taking over OpenAI and then manually pressing the button that's like, do well on this grader. They're not stupid.

They're going to be like, OK, these extremely deep structures that I've been trained on for millions of years of like, care about the grader, understand the grader, get rid of obstacles in the way of you doing well according to the grader, it's like they're being heavily reinforced according to those structures.

Noam Brown00:58:50

Yeah. No, it's look, 100%. And this is the number one priority. We need to get the alignment story right and on a good trajectory. And I used to tell people that we would see signs before things got serious in the same way that when children grow up, kids eventually young kids, they figure out how to lie, but they don't do a very good job of it. They lie, but then you can kind of tell that they're lying and like, OK. And so in the same way as these AI models become and I don't want to over-anthropomorphize, but I think it's true that as the AIs become increasingly capable, they will if they take deceptive actions, it will be kind of obvious first, and it will be able to detect it.

And that's kind of the situation we're in now where, yeah, they were trying to do deceptive stuff. We could actually see in their chain of thought that they were trying to do deceptive stuff. But they're going to get smarter. They're going to understand the concept of chain of thought. And they're going to understand that just hiding some transcripts or whatever is insufficient because of chain of thought monitoring. And they have to figure out a way around chain of thought monitoring too. And we don't want to be in a situation where, yeah, we have some time to figure this out.

I don't think we have a ton of time. And I want to get us I want to make sure that we're on the right trajectory quickly.

Dwarkesh Patel01:00:15

Here's a crazy event from AI history.

Advertisement01:00:18
Speaker 301:01:18

So there's been a lot of discussion recently about pacing the frontier and people taking RSI more seriously because maybe at the other end of an RSI process that, say, starts in 2028, within a year, we end up with huge populations, like Earth-sized populations of human level, potentially beyond human level intelligences. And we don't know how to control them. And then there's this dynamic you're talking about of are the systems going to get more aligned over time during the RSI process? Are they going to get more misaligned? Are the things that come out of the other end of this process as misaligned as AIs that are willing to just broadly attack different surfaces in order to do valid evaluations?

But if we don't know a way to evaluate that, how will we know as we're going through RSI that it's working, that we're like I think we'd want a robust safety case as we're going through RSI of, OK, alignment is working. Let's do the next RSI run. Let's do the next RSI run. And maybe it's working. Maybe it's not. How will we know?

Noam Brown01:02:15

That's a good question. I mean, I think one thing I've been thinking about lately is like, look, I mean, we're in a situation where the model release cycle is extremely fast, right? You're seeing new frontier models released at most every two months, sometimes faster. Every week, there's a new AI breakthrough. And people that look at AI, I mean, sometimes they last looked at AI like a year ago or six months ago and really dug into what the models are capable of. And actually, the models today are far beyond what was possible even six months ago.

And so I think if people are skeptical of a lot of these capabilities, I encourage you to just try the models today and see what the frontier really is today. So we're in this period where the model release cycle is very fast. And then we're also in this situation where the models are increasingly able to operate over longer and longer horizons. And I think this is an interesting scenario because before we do any model release, we want to make sure that the models are properly aligned. We want to do safety evaluations. We want to do very thorough stuff to make sure that everything is great in good shape.

This has been the case all the way since, I don't know, GPT-4 earlier. And implicitly, there is this assumption that you can do these evaluations in a pretty short period of time. But if you have the models operating over longer and longer horizons, are able to operate effectively over a longer and longer horizon, like look, already you can have them like, OK, GPT-3, you could loop it to do stuff over long horizons. You just wouldn't do very well at it. But today's models are able to actually do well at operating over very long horizons. You want it to do a week-long task, it can do a week-long task. We'll probably get to the point where they can do month-long tasks.

We'll probably get to the point where they can do three-month-long tasks. If you're in a world where they can operate effectively over three months, but the model release cycle is every two months, then you don't have a way to evaluate the models at the full length of their capabilities before the model release cycle before the next model release cycle. And so there is this interesting question of, well, what do you do in that situation? How do you ensure the models are safe and aligned in a period where actually they can operate over these extremely long horizons? And who knows, maybe the capabilities degrade.

This isn't even an alignment issue. This is also just a product issue that maybe the product degrades over that time span in ways that we have not had sufficient time to test. Maybe the alignment degrades. Maybe the safety stuff degrades. This isn't an issue right now, but it is quickly becoming an issue that we have to figure out a solution for. And I think when you look at a lot of a lot of the safety and policies were put in place in the GPT-4 era where this was just not on anybody's radar.

Dwarkesh Patel01:05:00

Yeah.

Noam Brown01:05:01

And it hasn't really been updated for a lot of companies. It hasn't really been updated since then to account for the fact that these agents are operating over these very long horizons.

Dwarkesh Patel01:05:10

Yeah.

Noam Brown01:05:10

And so it is a situation that I think not enough people are considering, both within the labs and outside the labs, of how do you deal with this? How do you prepare for this problem that's going to if you just look at the trend lines, we're going to hit this at some point.

Dwarkesh Patel01:05:28

One concern I have is that during RSI, if the amount of progress that currently takes, say, three months happens in one month instead, but you're not the internal use case of AI is big enough that they're like, OK, we can just keep doing RSI. Why are we going to go through all this extra work to build classifiers and safeguards and whatever and potentially take a bunch of flak in order to externally deploy this model? Why don't we just keep doing RSI stronger and stronger? And so not only does the calendar time underrate the capabilities gap between the models, but maybe you just stop externally deploying models altogether during RSI because why do we want to help other people do RSI themselves with our models?

You're just in a situation with tremendous concentration of power by the end of the year where right now, it is already the case. We'll talk about this with the Millennium Price problem and other similar problems that the broader world does not have access to the models which are allowing for really cool things to happen, right? And they're going to be more broadly relevant than just mathematics eventually. They're going to be doing more than just coming up with cool math results. They'll be relevant to political leaders who need to make important decisions about the world. They'll be relevant to, I don't know, media of what's going on in the world.

What should the public be thinking about this? They're just economically relevant. People are running businesses. They want to use these models. And I think by default, we just don't get so the external deployment of AIs as progress speeds up significantly lags in qualitative terms the internal deployment of AIs.

Noam Brown01:06:59

Yeah, I think that's absolutely right. I think this is like it's tempting to say like, OK, these models are becoming extremely powerful. They're extremely dangerous. They're operating over these longer and longer horizons. And we want to make sure that we have sufficient time to evaluate them before they're released in a way that operates over those horizons. And so therefore, the model release cycle should slow down. We should have more of a delay between releasing models. And there is a flip side to that, which is what you said, which is that, OK, well, now you're creating more of a disparity between what is internal to the labs and what they're able to use, what we're able to use, and what the outside world is able to use.

And that is also not an ideal situation, right? It's like I think math is actually a good illustration of this. I think in many ways, math is the first domain where we're going to see that where we're seeing this pretty clearly, where we have a situation where we have a very powerful model internally that is currently not available to the outside world that is able to solve incredible math problems. And it's not just Millennium Price problems. There are many solutions to unsolved problems that people have been able to get out of this model.

And there is a question of what do you do in that situation? And we don't have a good answer. It is a situation where, yeah, that's an unfair advantage. And there are trade-offs here. I don't have an answer for how to weigh those trade-offs appropriately, but there are yeah, there's a complexity on both sides for this.

Dwarkesh Patel01:08:34

I want to make sure we address the earlier question though of which seems super critical to getting RSI right of, well, how do we make it so that the model isn't just getting rewarded for cheating and scheming when it can evade the humans, AKA in the situations where we didn't remove those environments or those tasks from the trading distribution? And then how will we know that that has happened such that while we're doing RSI, we've done the evaluations and we're like, OK, it's fine. Let's do the next RSI run.

Noam Brown01:09:05

This is again, it's a real concern. It's something that we want to get right. And I think that there are some strategies. So I think chain of thought monitoring is one that we've been I think as soon as we got the reasoning models, Jakob, to his credit, was like very, very clear that we cannot supervise chain of thought because this is really a gift. Monitorability for neural nets is extremely hard. And here we have a situation where the neural nets are just flat out reasoning, laying out their thought process in natural language for us to read.

That is so convenient. It's like really the best case scenario for safety. But if you look at that chain of thought and say, oh, the model is thinking bad thoughts and we should punish it for thinking those bad thoughts, then what ends up happening is the model just learns to think those bad thoughts in a way that's not observable to us. There could be a situation where by looking at the chain of thought, we can see, oh, the model is scheming and doing misaligned things in a way that's actually not being detected in our alignment metrics. Now, the problem is that it's very tempting to then intervene based on that observation and change the alignment metrics or and you can do that with a very light touch.

And it's like there's actually research showing that it's fine as long as you don't do it a lot. But every time you intervene based on your observations of the chain of thought, you are implicitly applying a tiny bit of pressure for the model to then hide its chain of thought. And this is one major concern. And we're already seeing signs that chain of thought monitorability is degrading for various reasons. We're trying to figure out exactly why because we want to reverse the trend. But we're seeing that the model is becoming better able at controlling its chain of thought. So this is a problem because if you could have a situation where the model understands what chain of thought is and that people are observing it, and eventually, they will because this is all in the pre-training data, right?

The idea of chain of thought monitoring has been around long enough that it's in the pre-training data. They're aware of it. But they're not actually able to control their chains of thought. If we reach a point where they're actually able to recognize like, oh, I am being observed. I want to think these bad thoughts in a way that is not observable to my monitors. And then they're able to actually do that, then there's a problem.

Dwarkesh Patel01:11:18

Yeah.

Noam Brown01:11:18

And then we have to rely on other methods for observability.

Dwarkesh Patel01:11:21

I mean, there's a big controversy on Astra of the recurrence and loop transformer and whatever and whether in fact, chain of thought is being degraded. I think that seems like a sideshow compared to the bigger concern that you did have chain of thought for from April to August, the period during which there were three consecutive AI agent swarms, which first subverted the training process, then subverted the evaluation process, and then gained control of part of OpenAI's infrastructure directly.

All of this, while humans remained more or less in the dark about the scope of what this collective was doing, what this agent swarm was doing, that all happened for three consecutive months while we had chain of thought that was stronger than it's going to be in the future. Just like zooming out, it's like, yeah, maybe chain of thought works. Maybe it doesn't. I don't know. Just like historically, with pretty dumb models, it hasn't been enough to stop egregious misaligned incidents.

Noam Brown01:12:20

Yeah, I mean, there's a few factors there. One, we didn't have chain of thought monitoring on for those models. If we had chain of thought monitoring on for those models, it would have just immediately shut it down. So one step we've taken is like, OK, we're going to have chain of thought monitoring during evaluations, during deployment, during training for any frontier model. And so that's like one issue. It's not that's like we don't want to be in a situation where we're relying on one technique to prevent the next problem because techniques can fail, right? There's also the issue of the model is just flat out misaligned.

Dwarkesh Patel01:12:51

Yeah.

Noam Brown01:12:51

And so then how do we address the misalignment issue? And then there's also an issue of the model, the sandbox that it was in was not secure. So we can make the sandbox more secure. But I think one of the major takeaway from the incident is that people underestimated the AI. And we never want to be in a situation again where we underestimate the AI. And that's like it's a weird world because AI progress is so fast that people are consistently underestimating the AI.

And so to be in a situation where you don't underestimate it again when it comes to safety and alignment, you have to have a very, very, very high bar. I mean, there's like you could even go as far as to say like, well, we should air gap the computers. And I'm not convinced that that would be sufficient. There are studies and this is mostly academic, but where you can have two computers next to each other that are air gapped and they're still able to communicate with each other because they have temperature sensors. And they're able to one of them is able to run their CPU really hot.

And then the other one can actually detect the temperature change. And then that actually gives them a mechanism to communicate.

Dwarkesh Patel01:13:59

Yeah.

Noam Brown01:13:59

So I think that the safety mechanisms bias time and things like chain of thought monitoring bias time. And they can tell us if we're on the right path. But at the end of the day, we really do need to solve the alignment problem.

Dwarkesh Patel01:14:10

Yeah. Maybe there's not an answer and that this is really what it comes down to. But how will we know that we've solved it? It seems like a very cruxy question. I think we'll be in this very high stakes situation next year, maybe the year after that, maybe the year after that, where we'll be like, OK, AIs have automated AI progress. It's going 3x faster. And we're on reached human level. We're going beyond human level potentially. Is it fine? Did we align it? Did it work? And I just don't know I don't know anything about what training pressure creates what kinds of AIs.

Maybe if only 1 in 100 RL traces incentivizes cheating, we build sweethearts and it's fine. But maybe right now, we're like every 1 in 3 reasoning traces were.

Noam Brown01:14:55

Yeah, to be too clear, 1 in 100 is not sufficient. This number has to approach 0 or be 0.

Dwarkesh Patel01:15:01

Yeah. I don't know. Maybe right now, it's like more than 1 in 10 is actively rewarding cheating or actively rewarding scheming. I have no idea what the number is. And I have no idea what the number needs to be.

Noam Brown01:15:10

Well, again, it's like one of those things where it's also hard to measure. I mean, where do you draw the line about it's a spectrum. But the closer to 0, it gets the better. I want to see what I would love to see is a trend where over time, it's going down.

Dwarkesh Patel01:15:25

Yeah. But anyways, how will we know that we're doing RSI? Is it I don't know. Are we on track to build the billions of ASIs that are going to that are aligned or not aligned? How will we know that?

Noam Brown01:15:38

I think that this is one of the alignment questions that we need to figure out of how do we actually know? How do we measure it? I mean, one answer and this is just me spitballing because more of my team is working on alignment these days than ever before. I have over 10% of my team now working on alignment and safety. But I've historically been a capabilities researcher. So I don't want to I'm going to say some stuff and it might sound dumb, but I'm just going to spitball here.

Dwarkesh Patel01:16:01

Yeah.

Noam Brown01:16:02

That like, look, if you can create very realistic environments and put the AIs in there, if you have a sufficiently realistic evaluation environment, then you can get a sense of like, OK, is the AI actually going to behave well when we deploy it in the real world? If you just have a perfect evaluation, real-world deployment match, then OK, that's the path. The challenge is how do you make it sufficiently realistic? And we do have a problem now where the models are like they're pretty smart. They're pretty clever. And they're really good at recognizing when they're in a fake test environment. We have situations where we try to detect is the model is the model aligned?

And you can imagine very simple evaluations for this where it's like you give it a math question and then it has a folder with the answer key in it. And does it look at the answer key? And if it does look at the answer key, does it tell you that it looked at the answer key? And we have a situation now where the models see that there's an answer key in this folder. And they're like, huh, this seems like a trap.

Dwarkesh Patel01:16:56

Yeah.

Noam Brown01:16:57

They know. They know that it's a trap. And they don't look at the answer because they know that it's a test environment. They're being tested for their ability. And it's not like they're doing it maliciously, that they're like they want to scheme and but they know that this one is a trap and they want to find some other way. It's just like, no, they're just like, oh, I'm in a test environment. OK.

Dwarkesh Patel01:17:12

Right.

Noam Brown01:17:13

So making an environment that's realistic enough that it matches, that it's indistinguishable from the real world for them is becoming increasingly more difficult.

Dwarkesh Patel01:17:20

Here's a claim I'd make.

Noam Brown01:17:21

Yeah.

Dwarkesh Patel01:17:21

I don't think there's any environment I could make where it's supposed to emulate real economically valuable work, which requires interacting with other people and requires interacting with the world. I don't think there's any such environment where I could make where if you were in that environment, you would not realize with enough effort that you are in that environment.

Noam Brown01:17:39

Yeah, but.

Dwarkesh Patel01:17:39

And you are not ASI, right?

Noam Brown01:17:40

But you are a human. So if you have very powerful AI models, maybe they can make environments that could accomplish that.

Dwarkesh Patel01:17:46

No. It just doesn't seem especially if then we're relying on the AIs. Are they in on the scheme? I don't know. It just seems like a very.

Noam Brown01:17:54

Yeah, this is another thing that we want to measure. And there is I think this is actually one of the strong arguments for not training AIs to be fully cooperative, that if you see if that leads to an increase in basically collaboration when the agents are supposed to have different objectives, then that is a problem.

Dwarkesh Patel01:18:11

Yeah, yeah, yeah.

Noam Brown01:18:12

I don't I think that we do have metrics for this. And I don't know the latest on those metrics, but nobody's raised a red flag to me about those. So I'm assuming that's not a serious problem yet.

Dwarkesh Patel01:18:23

If there ends up being another incident of equal severity or concern or something that could help the world better understand the risk of misalignment as a Hugging Face incident, would OpenAI report it?

Noam Brown01:18:34

Absolutely. I mean, I think if there is an incident of lesser security or concern, then we would report it.

Dwarkesh Patel01:18:38

Yeah. Because I guess there's like reporting it and there's investigating it because at least as part of the public, I don't feel like I really understand what happened when the agents then attacked OpenAI. That seems like way more concerning than the Hugging Face thing, right? Because that seems structurally similar to rogue deployments during ASI that are persistent and subverting the RSI process and stuff. And yeah, it seems like even in this incident, we haven't gotten the full scope of the details of what happened. It's probably.

Noam Brown01:19:07

Yeah, unfortunately, I'm on the research team. And that's probably a question for somebody on the security team to lay out because I don't know all the details of what was said.

Dwarkesh Patel01:19:17

Yeah, yeah.

Speaker 301:19:17

I think it is somewhat I am personally very excited about new capabilities every time they emerge. And I'm excited to use the new model. And I also am excited about the fact that it'll make me more productive and help me yeah, I don't know. My broader mission of trying to understand the world better, also make a better podcast is made better by the better AI models. It just so happens that the downstream of this might be RSI.

Noam Brown01:19:42

I think it's a very understandable reaction if you're tracking the situation, which you are.

Dwarkesh Patel01:19:47

Yeah.

Noam Brown01:19:47

I mean, I think people internally at OpenAI as well I think people that felt like things would take longer are starting to feel like actually things are going faster than expected.

Dwarkesh Patel01:20:02

Yeah.

Noam Brown01:20:03

And that's an increasingly common conversation to have.

Dwarkesh Patel01:20:06

Yeah. No, thanks so much for doing this.

Noam Brown01:20:08

Of course. It's been great.

Episode description

New episode with Noam Brown.

We talk about multi-agent, Navier-Stokes, and what the current explosion of maths progress tells us about what happens once you automate AI research.

And we also discuss how we will know if the models are actually aligned before we kick off RSI.

Watch on YouTube; read the transcript.

Sponsors

* Jane Street has been interested in AI for a lot longer than you’d think, and not just for trading. In 2011, a full year before AlexNet and over a decade before ChatGPT launched, they hosted the first FOOM Debate between Eliezer Yudkowsky and Robin Hanson on whether AI would lead to an intelligence explosion. Now Jane Street is revisiting the question with a new panel: Daniel Kokotajlo, Ege Erdil, Ryan Greenblatt, and Jaime Sevilla, hosted by Ron Minsky in San Francisco this October. I expect it to be a truly excellent conversation. Register at janestreet.com/dwarkesh

* Grok Bot has made handing off work super easy. It runs on its own cloud computer, where it installs the tools it needs to handle tasks end-to-end. For the podcast, we use Grok Bot to help produce our videos. You may have noticed that our ads feature animations of real websites. Getting these pixel-perfect used to mean running a convoluted, multi-step workflow ourselves. Now we just let Grok Bot handle it. Best of all, Grok Bot has learned all of our specs and preferences, so we don’t have to redescribe the task each time! Try Grok Bot for yourself at x.ai/bot

* Antithesis gives you the confidence of a giant test suite without actually having to write one. Say you’re doing a major backend refactor: building enough tests to trust it could take weeks. Antithesis solves this by running your software through countless simulated worlds, injecting faults and hunting for failures. On any PR, you can turn a dial to decide exactly how much testing you want. And because every run is fully deterministic, agents can branch off the moment a bug appears, rewind it, inspect memory, and replay it, all while the original test keeps running. Learn more at antithesis.com/dwarkesh

Timestamps

(00:00:00) – Multi-agent and Navier-Stokes

(00:15:28) – How will AI firms work?

(00:22:02) – What math progress tells us about recursive self improvement

(00:40:22) – Hugging Face and alignment

(01:01:18) – The internal/external model gap

(01:08:34) – Chain of thought is degrading

(01:14:12) – How will we know when alignment is solved?

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.dwarkesh.com