Transcript of Eric Jang – Building AlphaGo from scratch

02:37:29 147 min read 0 views Published 6 months ago

Copy link
Episode summary

This episode walks through building an AlphaGo-style Go program from scratch, using modern AI coding tools. Eric Jang explains the basic rules of Go, including the Tromp-Taylor scoring used by computer programs, and then the search machinery behind AlphaGo: Monte Carlo tree search, the UCB and PUCT action-selection rules, and how a value network and a policy network make the search tractable.

The conversation then turns to self-play and why training on search-improved move labels can be more stable than reinforcement learning from win or loss signals. Jang and host Dwarkesh Patel compare this with reinforcement learning for language models, arguing about why tree search suits Go but is harder to apply to reasoning.

The episode closes with observations on what current AI tools do well and poorly when automating research, such as hyperparameter tuning and running experiments versus choosing the next question to investigate.

In this episode Dwarkesh Patel, Eric Jang

This episode's transcript for the AlphaGo discussion with Eric Jang was made with HappyScribe's technology, the same technology behind the Microsoft Teams note taker.

Dwarkesh Patel00:00:00

Today I'm here with Eric Jang, who was most recently Vice President of AI at 1X Technologies, before that Senior Research Scientist at what is now Google DeepMind Robotics. And you've been on sabbatical for the last few months. One of the things you've been doing is rebuilding and improving and hacking on AlphaGo, and so we're— today what we're going to do is you're going to explain building AlphaGo from scratch and what it tells us about the future of AI research and development. But, uh, before we get to that, why is AlphaGo interesting? Why is this— why is this the project you decided to do on sabbatical rather than just hanging out at the beach?

Eric Jang00:00:33

Sure, yeah. Um, I like making things, and AlphaGo and Go AI is one of those things that really got me into the field. When I saw the kind of early breakthroughs on AlphaGo in 2014, 2015, 2016, and so forth, it was just profound to see, you know, how smart AI systems could become, and the— the kind of computational complexity class that they could tackle with deep learning. This is a problem that has, you know, long been understood to be kind of intractable for search, and yet it was solved through— through deep learning.

And so, so that was quite mysterious to me, and I've always wanted to understand that phenomena a little bit better. My training is often in deep neural nets for robotics, where it's, uh, the— the decisions made by the neural networks are a bit more intuitive. But AlphaGo is a sort of problem where the— the decisions are actually the result of a very, very deep search. And it's always been very mysterious to me how, like, a 10-layer network can sort of amortize the simulation of something so, so, uh, so deep in the— in the game tree.

Dwarkesh Patel00:01:35

Yeah, interesting.

Eric Jang00:01:35

So if you plot out how much compute it took to build various iterations of strong Go bots over the years, you can see that in 2020 there was an open-source project called KataGo, um, by David Wu from Jane Street, who— who basically achieved a 40x reduction in compute needed to train a really strong Go bot Tableau Raza. I'm not certain if it's stronger than AlphaGo Zero or AlphaZero MuZero, but it's very, very strong, and this is what most Go practitioners today train against when they're— when they're playing in AI. And thanks to LLM coding, what took a whole team of research scientists at DeepMind and, you know, millions of dollars of research and compute can now be done for, you know, a few thousand dollars of rented compute.

Dwarkesh Patel00:02:14

By the way, if you're listening to this on an audio platform, this is a Blackboard lecture, so I highly recommend switching over to a video platform like YouTube if you can to look at the math and the graphs and the Go board. Okay, I guess we should first discuss how Go works.

Eric Jang00:02:28

Great.

Dwarkesh Patel00:02:29

So, yeah, how does the game work?

Eric Jang00:02:30

Um, so the game of Go is a very simple one that can be implemented, uh, quickly and easily in a computer. The— the objective of the game is basically to put down black and white stones and try to occupy as much territory in the game as possible. So I might start by putting down a black stone. Black always goes first. Go ahead. And so the way you capture an opponent's stones is that for every intersection, if you can surround all four of its neighbors with, um, with your stones, then, um, then this one is sort of cut off from oxygen, if you will, and then it, uh, it is a dead— dead stone.

So, so then now I control these four stones as well as this empty intersection here. So there's, like, slight variations between Chinese, Japanese, and what is called Trump-Taylor rules. Trump-Taylor rules are designed to be completely unambiguous for Go, so this is what all Go AIs train against and resolve against. So in typical Go, like, where the humans play, you're actually not allowed to put this white stone down here. It would be instant suicide. In Trump-Taylor, it's actually fine. You put it down and then it immediately resolves to death, so the outcome is sort of the same. Let's go ahead and start over and play a few stones, and then I'll explain some more.

So I'll just start there.

Dwarkesh Patel00:03:44

All right, I'm, like, basically playing randomly here, but I'm trying to get around your stones and see if I can clear them.

Eric Jang00:03:50

Sounds good.

Dwarkesh Patel00:03:50

Yeah.

Eric Jang00:03:52

Hmm. So this move, um, basically exposes one empty neighbor for your white stone, and it's very akin to a check in chess, where if you don't respond immediately by putting one here, then I can immediately capture this.

Dwarkesh Patel00:04:09

Ah, I see, okay. Because it is— it is sort of the diagonals that determine whether you're bounded in.

Eric Jang00:04:13

The cross-section, not the diagonals. So, so this one is surrounded on three sides.

Dwarkesh Patel00:04:17

Yep.

Eric Jang00:04:18

And so, um, you're at threat of losing that stone if you don't play one immediately there. Now you can see that I'm starting to pressure you because by putting a stone, uh, here, now you are forced to, um, put one here.

Dwarkesh Patel00:04:31

Otherwise you would have this two-block to yourself.

Eric Jang00:04:34

Yes. And then if you think— think through, like, what happens if you were to respond here, you can probably, you know, search into the future and deduce what I'll do in response once you— once you do that.

Dwarkesh Patel00:04:43

You have a lot of confidence in my abilities, but I'm guessing you'd put the black here.

Eric Jang00:04:47

That's right. And then I would capture all three of these stones.

Dwarkesh Patel00:04:49

So I should just assume that this is gone? This little block is gone?

Eric Jang00:04:53

Yes. So in Go, it's actually okay to let an opponent capture, um, some stones if, for example, it allows you to position, uh, to capture more stones in somewhere else on the board.

Dwarkesh Patel00:05:02

Yeah.

Eric Jang00:05:02

And this is what makes Go a very beautiful game is that, um, you can kind of, uh, lose the battle but win the war, right?

Dwarkesh Patel00:05:08

Interesting.

Eric Jang00:05:08

And as the board size increases, the complexity of these kind of, like, micro versus macro dynamics gets— gets more interesting.

Dwarkesh Patel00:05:15

Presumably you'd put one here.

Eric Jang00:05:16

Mm-hmm. And so now I would capture this entire group.

Dwarkesh Patel00:05:20

Okay.

Eric Jang00:05:20

And this would be mine. Okay, there's one more, um, case that I want to demonstrate, which, uh, actually I had a bug in my code recently, which is the following situation. So let's consider a formation like this, right? And then, you know, we have other pieces on the board in play or whatever. Um, and so, um, let's talk a little bit about how the game ends, right? Um, in this territory, who controls these— these areas? Is it white or is it black?

Dwarkesh Patel00:05:49

White.

Eric Jang00:05:50

It's actually black because I have actually surrounded this whole area.

Dwarkesh Patel00:05:53

Yeah.

Eric Jang00:05:53

And it's, um, very— assuming I have, like, other black stones here, it's actually very hard for you to break this out of the control of these stones.

Dwarkesh Patel00:06:02

Yeah. So when the final score is tallied, would— would these ones also count as being in, um?

Eric Jang00:06:06

Yeah, great question. So, so, um, this is where different rule sets have different ways of scoring, and so we should talk a little bit about how, like, uh, you resolve scores between humans and how you resolve scores between computer Go.

Dwarkesh Patel00:06:17

Mm-hmm.

Eric Jang00:06:18

Because there's actually some ambiguity in how humans evaluate this. So most humans would look at this board configuration and conclude that, like, black has kind of totally surrounded white, and so white has no chance of life.

Dwarkesh Patel00:06:28

Yeah.

Eric Jang00:06:28

We could play out more here, but then at the end I would capture everything.

Dwarkesh Patel00:06:32

Yeah.

Eric Jang00:06:32

Um, however, if you have a way of breaking this formation and connecting white to something outside of it, then it can flip, right?

Dwarkesh Patel00:06:38

Yeah, yeah.

Eric Jang00:06:39

And so this is where it's, you know, a little bit hard for a computer to decide these kind of things, right? So how do humans do it, right? Like, it's worth thinking a little bit about how humans resolve this because this will actually map later to how we think about the deep neural network. Um, humans basically say, uh, "I think the game is done," and then you— you have to also say, "I think the game is done," and then we'll say, like, "I think these are— these are my stones," and then you have to agree. If you don't agree, then we keep playing.

Dwarkesh Patel00:07:03

Yeah.

Eric Jang00:07:04

So, um, essentially, once two humans, their so-called value function, um, agree on a consensus, then the— then the Chinese rules result that.

Dwarkesh Patel00:07:13

Yeah, interesting.

Eric Jang00:07:13

So in Trump-Taylor scoring, um, it's perfectly unambiguous, so it can be decided, you know, algorithmically by a computer. So if— if let's say you— you have this at the end— endgame, the way you score this is that you first count how many stones you control, and that's unambiguous.

Dwarkesh Patel00:07:29

Yeah.

Eric Jang00:07:29

Then you count how many empty intersections that are not touched by your opponent's stones. So these intersections would not count for either player because both of— all of these intersections are connected to both white stones and black stones, right?

Dwarkesh Patel00:07:42

Mm-hmm.

Eric Jang00:07:43

If, um, this were like this.

Dwarkesh Patel00:07:46

Yeah.

Eric Jang00:07:46

Then white would get three points. Now this is, uh, a little odd because a human would know that white is actually losing these points.

Dwarkesh Patel00:07:53

Yeah.

Eric Jang00:07:54

But, um, Trump-Taylor scoring would consider white to have all of these points as well as these points.

Dwarkesh Patel00:07:59

Got it, okay.

Eric Jang00:07:59

Right. So, so that is a very big difference in, um, how computer Go scores things and how humans score things.

Dwarkesh Patel00:08:06

How does the game end?

Eric Jang00:08:07

The game ends when either a player chooses to resign.

Dwarkesh Patel00:08:10

Mm-hmm.

Eric Jang00:08:10

Or both players pass consecutively.

Dwarkesh Patel00:08:13

Cool.

Eric Jang00:08:13

Yep. So that's the rules.

Dwarkesh Patel00:08:14

Nice. All right. Now help me crack this with AI.

Eric Jang00:08:17

Great, okay.

Dwarkesh Patel00:08:18

Let's understand how, um, AlphaGo actually works and how somebody in the audience might be able to implement it.

Eric Jang00:08:25

Great, yeah. Let's start with, um, kind of an intuition about the underlying, um, you know, search process used to make moves, and we'll layer on, uh, ideas from deep learning to make it much more efficient and tractable. So Go is a game where there's just two players. We're going to draw a person here, and we're going to draw an AI here. And, um, let's say this person is playing black, so they go first. So we're going to draw— they go here.

And then now the AI, um, is going to make a move based on what it sees here. So there's a question of, like, how you encode these inputs into the AI. Maybe you could use ones and zeros, but you want to represent, um, you know, black, white, and empty. So, so you would need at least three different values here, right? So maybe you could use zero ones and twos or something. So, so the AI might see something like, you know, zero zero zero zero, uh, one.

Great. So, so this is the input to the AI, uh, on its turn.

Dwarkesh Patel00:09:37

Yeah.

Eric Jang00:09:38

So, so the AI can choose, let's just pick three possible random moves that it can go, and I just drew these at random. And so which— which move is best here, right? Well, we don't know until the game ends. Um, there's no— Go does not have any kind of local reward of which move here is good. And this is what makes Go a very difficult game is that you don't actually know who won until you really get to the end of the game. So how deep is this tree, right? Well, in a 19 by 19, um, Go board, there are, uh, you know, roughly to the order of 361 moves on any given, uh, move. And of course, as it fills up, you have less moves.

Um, and, and the— the number of steps in the game can be somewhere from 250 to 300 moves. And maybe experts might, uh, decide to end the game, uh, well before that, but, uh, you know, under Trump-Taylor scoring, you actually have to play things all the way to the end. So this could be, like, 300 moves or something, right? So, like, 300, um, like, depth of the tree.

Dwarkesh Patel00:10:31

Yeah.

Eric Jang00:10:31

So if you keep on expanding possible moves here, so in— in this move, the AI is going, and then, you know, here the human would go, and then, you know, there's— there's some. And so forth. You can find that, like, essentially what you end up with is an enormous explosion in the possible game outcomes originating from just this one state. So this is something to the order of, like, you know, 361 to 300, power of 300, which is far more than the number of atoms in the universe, right?

Like, it's— it's just, uh, it's just— and of course, actually, there are redundancies and symmetries, so it's not actually 300, but— but that's sort of the— if you were to do a naive tree where there were no merging of children, then actually you end up with a tree about this big.

Dwarkesh Patel00:11:21

What do you mean by merging of children?

Eric Jang00:11:22

Right. Let me, uh, use this board here. So if we start here, and then you play here, and then, um, I play here, and then you play here, that is equivalent to I start here, you play here, I play here.

Dwarkesh Patel00:11:37

Yeah.

Eric Jang00:11:38

And then you play here, right?

Dwarkesh Patel00:11:39

Yeah.

Eric Jang00:11:39

So, so both of them arrived at the same spot but through different paths. So this child node can be thought about as a shared ancestor.

Dwarkesh Patel00:11:46

Right, right.

Eric Jang00:11:46

Yeah.

Dwarkesh Patel00:11:46

And I guess it's not 361— it starts at 361, but it decreases by one each time.

Eric Jang00:11:50

Yes. And the branching factor decreases by one each time.

Dwarkesh Patel00:11:52

Yeah.

Eric Jang00:11:52

Yes, yes. But in— in any case, this is a very, very, very large tree.

Dwarkesh Patel00:11:55

Yeah.

Eric Jang00:11:55

And, uh, this is also why, you know, computer scientists for many years thought that Go was not a tractable problem this century.

Dwarkesh Patel00:12:02

Yeah.

Eric Jang00:12:02

Because the amount of compute you would need to exhaustively search every possible possibility, um, is just too large.

Dwarkesh Patel00:12:09

Yeah.

Eric Jang00:12:09

Um, if you could, Go is actually a deterministic game. So, um, on any given state, you can actually compute what the, uh, best possible strategy you can, uh, you can make is in order to win the game. You can search all the possible futures where you win and then just make sure you always stay in— in that, you know, set of futures.

Dwarkesh Patel00:12:25

Mm-hmm.

Eric Jang00:12:26

Um, so AlphaGo's kind of core conceptual breakthrough was using neural nets to make this search problem tractable. So before we get into, you know, how neural networks are involved, let's talk a little bit about how we can, you know, assuming we had a powerful enough computer, um, search this, uh, this tree to find the best move, right? So in the beginning, um, you're not going to build out the whole tree, uh, because storing that tree would be very expensive. Instead, you might do something like interactively figure out which, um, which leaves of this tree are worthy of exploring and expanding into the future to see, you know, what else is there.

So, um, there are some early algorithms in, uh, bandit literature like, you know, UCB, uh, 1, which is not exactly appropriate for a, you know, sequential game like Go, but very much inspired the action selection, um, algorithm used in, uh, in AlphaGo. So, so UCB 1 looks like on— on every move, we're going to take the best action, uh, or, you know, the argmax over A that maximizes, um, um, you know, the Q of A, and I'll explain what Q of A is in a moment, plus some sort of exploration bonus.

So on every node, we're going to track a few quantities. So, so let's, you know, consider each of these a node. This is— this is the root node, um, where you're making decisions from. And these are the children of the root node. And, um, we're going to say each node is basically a data structure that is, um, it stores a visit count of this, um, this node, this— this child node.

Dwarkesh Patel00:14:16

Is how often the parent visited this node.

Eric Jang00:14:17

Yes. And we'll call this an action. So, so one thing that is easy to trip on is, like, if you come from, uh, you know, robotics or, um, other kinds of reinforcement learning is, like, where are the actions, right? I'm only talking about nodes.

Dwarkesh Patel00:14:28

Mm-hmm.

Eric Jang00:14:29

Um, nodes here represent states, and because this is a perfectly deterministic game with no randomness, you can actually just infer the action based on the child.

Dwarkesh Patel00:14:37

Oh, yeah.

Eric Jang00:14:38

So, so if I go here, that implies an action.

Dwarkesh Patel00:14:40

Yep.

Eric Jang00:14:40

And this is the state that we resolve, right?

Dwarkesh Patel00:14:42

Makes sense.

Eric Jang00:14:42

So, so the LLMs, if you ask to, uh, you know, vibe code a, uh, MCTS implementation, it'll most likely design the right data structure here. But, um, you know, it's up— it's sort of a chef's choice. You can actually rewrite the tree structure however you like.

Dwarkesh Patel00:14:56

Yeah.

Eric Jang00:14:56

This was what, um, Claude 4.6 wrote for me when I— when I asked it, and it was a very reasonable choice. So, um, so then, you know, Q represents the, um, mean action value of this action. And I'll use a subscript A to denote that this kind of corresponds to taking a specific action to— to get here, right, from— from the— from the root node. Um, so, so, like, uh, if— if we have root, basically taking A gets us to this— this node here.

Dwarkesh Patel00:15:27

Yeah.

Eric Jang00:15:28

Um, and then we're going to also store the probability of taking this action.

Dwarkesh Patel00:15:35

Again, from the parent.

Eric Jang00:15:36

From the parent, yes.

Dwarkesh Patel00:15:37

Yeah.

Eric Jang00:15:37

Like, like, what are the odds that we sample this one?

Dwarkesh Patel00:15:39

Yeah.

Eric Jang00:15:39

And— and this will become relevant later, you know, like, uh, we've talked about a deterministic tree for now, so I'll— I'll bring probabilities into this later. And then finally, we have a sort of, uh, dictionary of children, which is just like, uh, you know, more of these nodes in a— in a sort of classic linked list style reference tree. So, um, this is the basic data structure to implement a tree. And, um, in AlphaGo, um, they use a slightly different action selection criteria called, um, PUCT, and it's short for predicted upper confidence with trees.

And, uh, this is basically, um, when you— when you select which— which child to take, you do argmax A of Q of SA plus, uh, constant so the equation forms are actually pretty similar.

Um, these are both scoring criteria, right? Like, you want to argmax this quantity, and you want to argmax this quantity to determine which action to take. So let's break down the intuition of, like, how you select actions here. This is the mean action value, so how good is a given child on average? Um, and— and— and if you actually, you know, knew the whole tree, then, uh, this is all you need, right, to select the best action. You don't really need to do more than that. But if you're interactively building this tree as you're, um, figuring out what the Q values should be, then what you have to do is occasionally try some other actions, you know, as a sort of explore versus exploit trade-off.

So, um, in both UCB and, uh, PUCT, there is this term here that basically rewards, um, taking actions that you haven't taken before. So as we mentioned before, each node stores the visit count of taking that specific action, right? So everything is initialized to zero. And so for a given action, let's just say, like, call it N, like action A, um, initially it's zero. And— and so as N is increasing, if— if let's say we've already made 10, um, 10, uh, action selections from that root node, um, but we haven't picked A yet, then this term actually starts to become quite large for A.

Dwarkesh Patel00:17:55

Yeah.

Eric Jang00:17:55

Right. And conversely, if we have chosen A 10 times out of 10, then now this term is quite small.

Dwarkesh Patel00:18:01

Yeah.

Eric Jang00:18:01

It— it diminishes very quickly. And the same thing is actually true here.

Dwarkesh Patel00:18:05

I'm just going to make sure I'm understanding it. Maybe I can, uh, put it in my own words.

Eric Jang00:18:11

Mm-hmm.

Dwarkesh Patel00:18:12

Let's just focus on UCB. What we're saying here, you can think of it conceptually as two different things, the Q and then this exploration term.

Eric Jang00:18:20

Mm-hmm.

Dwarkesh Patel00:18:21

Let's just be clear about what Q is. Q is basically saying, hey, once we do these rollouts, so you're actually running all these simulations, you go down the tree, and then you figure out, okay, if I end up at the terminal value of this tree, do I win this game or not? And then you do this, you average whether I win this game or not across all the, you know, the leafs of this tree starting from this node. That average, you put in Q.

Eric Jang00:18:45

Correct.

Dwarkesh Patel00:18:45

And so you're saying the Q is basically representing, will I win this game or not? Uh, what— what is the probability that I'll win this game starting at this node? That's your sort of, um, that is your sort of exploit. That is like saying, I've run these simulations, I think this is a good move or not. And then this other term is saying, um, have I explored this branch enough yet relative to the other actions I could be exploring? Or I have already explored. Uh, if I haven't explored this branch yet, you know, maybe I think it has a low score, but I just haven't explored that many bran leafs of this, uh, down this, uh, leaves down this, uh, down this node in this tree.

So I should maybe, like, try this even though the Q, the sort of exploit is telling me that this is not that valuable. And because L and of N grow slower than N, uh, basically as over time, you will move from the argmax being dominated by this exploration term, which is the second term here, to the argmax being dominated by the Q term, which is like, okay, I've done enough simulations, I'm quite confident that, like, this is the branch to go down.

Eric Jang00:19:45

Yes, that's right. So, um, the motivation for UCB was to come up with an algorithm where if you don't know the payoff of the arms, uh, the— the different actions you can select to begin with, this, uh, strategy basically with given some exploration term here.

Dwarkesh Patel00:20:01

Yeah.

Eric Jang00:20:01

Bounds your regret.

Dwarkesh Patel00:20:02

Yeah.

Eric Jang00:20:02

Uh, in terms of how wrong you can possibly be.

Dwarkesh Patel00:20:04

Yeah.

Eric Jang00:20:04

Um, I don't know the proof. I don't also know if this one is proved to have a logarithmically or— or like, uh, you know, square root bounded regret or anything, but I think the algorithm was just derived to look something like this. And you can tell that these terms are a little— they grow a little bit differently, and this is actually just to account for the fact that Go has many more actions.

Dwarkesh Patel00:20:20

Yeah.

Eric Jang00:20:20

In every given move compared to your standard bandit problem.

Dwarkesh Patel00:20:23

Yep.

Eric Jang00:20:23

Um, so, uh, one small clarification to make is that you talked a little bit about simulations on probabilities and so forth. Um, we should remember that Go fundamentally is a deterministic game. So the notion of pro— like, where does the notion of probability come from here, right?

Dwarkesh Patel00:20:37

Mm-hmm.

Eric Jang00:20:37

Um, uh, if you had a very powerful computer, there is no probabilities. You just— you can just compute the true average of what the— the mean action value is. So where does the probability come in? Well, it turns out that, um, uh, as in, you know, computer Go before, uh, AlphaGo, we've always done some sort of Monte Carlo method where we have some— we— we take the, um, expected Q value averaged over a randomly selected tree. Um, and that randomly selected tree is where probabilities come in.

Dwarkesh Patel00:21:10

Mm-hmm.

Eric Jang00:21:10

So the interpretation of Q is, um, what is the expected action value under the, um, under the random distribution induced by some random search process?

Dwarkesh Patel00:21:21

Makes sense.

Eric Jang00:21:22

And so where does the random search process come in? That's where, uh, you know, P of action comes in.

Dwarkesh Patel00:21:27

Yeah.

Eric Jang00:21:27

So if we assume a very naive algorithm where you have a uniform probability of taking any valid action, then this would just be one over, you know, the— the number of valid moves.

Dwarkesh Patel00:21:36

Yeah.

Eric Jang00:21:36

In, uh, in the— in the, uh, setup. And you would be kind of taking this average over this very diffuse tree, right? And— and this is, uh, this is a valid, um, interval you can take, but it's very slow because you're going to consider a lot of trees that have very low value.

Dwarkesh Patel00:21:50

Yeah.

Eric Jang00:21:50

And, uh, it's essentially almost like a important sampling problem where you want to— there's only a few actions and tree and— and sort of, uh, paths that can contribute, you know, high value, and almost everything else is low value. So— so that's a sort of a tricky, um, problem here. Okay. Um, so this is the action selection criteria for how you decide which moves to move down. Now, as you move down, um, in— in tree search, you will eventually run into a node where, um, it's quite clear you've won or lost, right?

At the— at the very, very end of the game when— when there are no valid moves to play left under— under Trump Taylor scoring, you can decide whether you, like, you know, won or lost, right? So you— you either win or you lost. And so this is basically, um, you know, the— the final return of the whole game.

Dwarkesh Patel00:22:39

Mm-hmm.

Eric Jang00:22:40

Right. Um, and so the— the question here is, like, we— we can assign a, um, a value U to a— a terminal leaf node of the tree, but how do we assign the values for nodes prior to that, the parents? And it turns out, um, you know, what you simply do is you just take the, um, your mean action value is essentially your average. So let's suppose these were leaf nodes. Um, uh, sorry, these were all leaf nodes. The— the mean action value of this node, you know, this action here is just the average of whether you won or lost at the leaf nodes.

Dwarkesh Patel00:23:17

Yep.

Eric Jang00:23:18

And, uh, correspondingly, you can kind of walk up the chain and say, like, well, the mean action value of this node, let's call this like QB, and this is action B, is just the average of a weighted average of these ones here.

Dwarkesh Patel00:23:30

Yeah.

Eric Jang00:23:31

Right. And— and the weighted average is, um, it could be dependent on if you have a different sampling distribution or not, but the— the basic intuition is that you want to resolve the game where you have a deterministic win or lose, and then you can kind of go backwards. Uh, this is called the backup step and assign values to these, uh, these— these— these nodes or actions, um, uh, corresponding to the averaged over— over the final terminal leaf. Yeah. Okay. So, um, if you were to do this without neural networks, it would still be intractable.

Um, you would— you would have a trouble finding, you know, which, um, actions to sample. A lot of the actions will contribute very low value, especially if you're, like, you know, trying to fight your way out of a losing position, and only a few actions give you high value. So the search in practice is still very, very expensive. Um, but— but the— the idea is that, like, if you can— because Go follows a tree structure, you can actually, you know, inform a very good estimate of the value of this node based on the, uh, values of, uh, uh, you know, downstream, assuming they're all correct and assuming you've searched deep enough.

Dwarkesh Patel00:24:34

Mm-hmm. Your explanation earlier about the, um, these sorts of states where it's obvious to a human who's going to win, but it's not obvious to.

Eric Jang00:24:41

Right.

Dwarkesh Patel00:24:41

Or, like, you deterministically you still had to play it out.

Eric Jang00:24:44

Mm-hmm.

Dwarkesh Patel00:24:44

Actually drove home the intuition of why the value function both is trainable and two, why it's necessary in order to actually be able to learn this game effectively. And maybe it's worth defining value in the first place, but.

Eric Jang00:24:58

Sounds good.

Dwarkesh Patel00:24:59

Yeah.

Eric Jang00:24:59

Yeah. So we— we talked about, uh, you know, this U value being, you know, your final resolution of whether you won or lost, and this is a terminal leaf node condition. Um, now, humans don't play all the way to the sort of edges of the— the tree, the leaves of the tree.

Dwarkesh Patel00:25:12

Yeah.

Eric Jang00:25:12

Right? They kind of stop, you know, some dozens of moves before, maybe— maybe even 100 moves before in— in sort of high-level play. So how do they know, right? Like, you can think about humans as implicitly having a neural network called a value function that basically, um, you know, takes in, uh, a board state and then it kind of evaluates, um, you know, he win. And so the human glances at the board and they know, like, I'm probably going to lose, right?

Dwarkesh Patel00:25:41

Yeah.

Eric Jang00:25:41

And— and they're essentially running a neural network that looks at a board, and implicitly they are amortizing a huge number of possible game playouts and— and taking that average and then deciding whether the board is winnable or not, and then whether they should concede or— or, you know, keep playing or not. And, uh, this is remarkable. If you think about, like, the, um, the beauty of something like this, it's like a— a neural network in a— in a human can somehow do all of this simulation at a glance and then just know, like, within a few seconds, without actually playing every single game logically, based on just kind of, like, crystallized knowledge and experience that, like, they can do this.

And so this gives us a hint that, like, in games like Go, um, there are ways to basically radically speed up the search process. And this is one of the fundamental intuitions behind why AlphaGo works, is that you can train a value function to look at a board and quickly resolve the game without playing out all of these trees into the, you know, into a very deep search depth.

Dwarkesh Patel00:26:41

Yep. Makes sense. I— I also for the audience, um, I sort of found, uh, for previous episodes when I was prepping and it seemed somewhat relevant to understand how AlphaGo works, I would find it very, very confusing. And, but it's the kind of thing where once you understand the problem in this way and then you'll build the next few pieces, it is actually much more understandable and it will make a lot of sense. And it's okay to be confused right now, uh, but it's— it's probably simpler to understand by the end of this lecture than you anticipate. So.

Eric Jang00:27:10

Mm-hmm.

Dwarkesh Patel00:27:11

I'll just make that note for the audience.

Eric Jang00:27:12

Uh, yeah, the important intuition at a high level, just to, you know, step back about where we're going with all this is that, um, classically for games like Go, you could build a tree, but we don't have computers powerful enough for that.

Dwarkesh Patel00:27:24

Yeah.

Eric Jang00:27:25

And, um, estimating the value of every action that you could possibly take is also hard because you don't know until the end of the game.

Dwarkesh Patel00:27:33

Yeah.

Eric Jang00:27:34

You could take averages, uh, by playing them to the end, but that's also hard because you don't know which actions to take.

Dwarkesh Patel00:27:39

Yeah.

Eric Jang00:27:40

To sample these averages. So conceptually, there's kind of two problems. There's the breadth of the tree and then there's the depth of the tree.

Dwarkesh Patel00:27:47

Yeah.

Eric Jang00:27:47

And AlphaGo gives us a way to basically, uh, shrink both of those to be very tractable.

Dwarkesh Patel00:27:52

Yeah.

Eric Jang00:27:52

That's— that's essentially the kind of core idea behind it. Okay. So we, uh, we take this idea that, like, you know, humans can glance at a board and instantly predict whether we win, and maybe that gives us the opportunity to really truncate the how deep we— we search.

Dwarkesh Patel00:28:05

Yeah.

Eric Jang00:28:06

And then, you know, we also know that humans can look at a board and, um, and, uh, decide, you know, um, what— what boards, you know, like, intuitively at a glance, what moves might be good on a good board, right?

Dwarkesh Patel00:28:22

Yeah.

Eric Jang00:28:22

So— so these are kind of two things that we can use deep neural networks for to accelerate this search process. Um, let's go back. Before we talk about neural nets, let's just go back to how this playout works. So we've only talked about making one move, right? So— so the AI looks at this encoded Go board. It has a tree. Um, it searches for, you know, deeply into the tree to find out which of its actions might be the best. And then it takes that action. And then now, you know, it goes back to the human. So maybe now the human sees, uh, a Go board that looks like, you know, like this.

And, um, and then they, um, they make their move. So maybe they put, um, they put their stone here. And then now we, um, we go back to the AI, which now looks at a new encoded board. So I've used two to denote the AI's playing as white and one to denote the human playing as black and zero as empty.

And then now on the AI's turn, it does the MCTS tree search all over again from scratch, right? So— so it throws away this old tree that it searched last round, and now there's a new root node and it begins to search anew. And then so and so forth. So MCTS is basically a— you can think about it like a search algorithm that is, um, deciding what moves to play best, aided by neural networks. Um, and— and it's— it's done on every— every move. Okay. Great. So let's talk about the neural network part of this.

Dwarkesh Patel00:29:57

Mm-hmm. And while you were racing, another sort of thing that was important for me to understand was this MCTS data structure with nodes and childrens of nodes and whatever. Um, this is done per move and reinstantiated once a move is made. So a human makes a move, then the AI looks at this and is trying to basically run a bunch of simulations, uh, to figure out, okay, what should make move should I make next? And those simulations just— a simulation is basically like exploring one more node in this MCTS tree.

And at the end, um, once all these— once all this, you know, you run 1,000 simulations, that informs then this, um, I guess as you will explain, this probability of what move to make next. That's what you store. You— you sort of choose the best move given those probabilities. You discard all of that. Then the next player makes a move and you restart this process at the beginning of every move.

Eric Jang00:30:52

Correct. One small addendum. You don't discard all of that. You keep one thing behind that we'll use later.

Dwarkesh Patel00:30:58

Yeah.

Eric Jang00:30:59

Yeah.

Dwarkesh Patel00:30:59

Just like I did for Reiner, I wanted to make flashcards for this episode so that people could retain these concepts. And ideally, an LLM could generate some candidates for me to then refine. But to actually get high-quality suggestions, I needed to design a whole pipeline where the AI could take and ingest screenshots of the blackboard at the right timestamps and then make SVG diagrams in case visuals were helpful, and then run their writing and drawing through a critic and then revise the card in response to this feedback. It's very hard to accomplish this just by stacking LLM calls. This sort of step-by-step recipe works much better if you have a durable agent that's been engaging with the task across all the previous stages.

Advertisement00:31:35
Eric Jang00:32:05

Okay. So now we have a basic intuition of how moves are made with search.

Dwarkesh Patel00:32:09

Yeah.

Eric Jang00:32:09

We're going to talk about how neural networks can speed this up, uh, by providing an analog to, like, the human intuition.

Dwarkesh Patel00:32:14

Mm-hmm.

Eric Jang00:32:15

So there's two networks. There is the value network, which takes in a state and it predicts, you know, am I going to win or lose? It's a binary classification problem. Then we're going to have a policy network, which, uh, induces a distribution over good actions to take.

Dwarkesh Patel00:32:36

Mm-hmm.

Eric Jang00:32:37

So, um, I'm going to draw a one-dimensional flattened move distribution, but this is really like, you know, a— a square kind of grid, right? So, um, so maybe like it thinks actions are like these are the kind of probability distribution over good actions.

Dwarkesh Patel00:32:53

Mm-hmm.

Eric Jang00:32:54

And both of these are, uh, categorical classification problems, right? So you can train this like any classifier in, uh, with deep learning, um, uh, you know, cross-entropy, loss, that kind of stuff. So the, um, the specific architecture does not actually matter too much. I— I tried a few different architectures. Transformers work, ResNets work.

Dwarkesh Patel00:33:12

Mm-hmm.

Eric Jang00:33:13

For small data regimes, um, my experience is that ResNets still kind of outperform Transformers and, um, and kind of give you more bang for the bug at lower budgets, but this may not be true.

Dwarkesh Patel00:33:22

Oh, wait. Why is that?

Eric Jang00:33:23

Um, they— they provide the inductive bias of, like, local convolutions.

Dwarkesh Patel00:33:26

Yeah.

Eric Jang00:33:26

And generally, Transformers start to outperform, uh, residual convolutional networks when you want more global context.

Dwarkesh Patel00:33:33

I see. Okay.

Eric Jang00:33:34

So— so one, um, interesting finding from the CutterGo paper was that they found it actually quite useful to pool together global features together, um, and aggregate global features like, uh, throughout the network, um, to kind of give the network a global sense of how to, like, connect value from one side of the board to another side of the board.

Dwarkesh Patel00:33:52

What does it mean to aggregate global features?

Eric Jang00:33:54

Yeah. So if you have a, um, Go a— a very large 19x19 Go board.

Dwarkesh Patel00:34:00

Yeah.

Eric Jang00:34:00

And you— you know, you've got some— some sort of battles going on here and you've got some battles going on here. Um, when you pass this through a convolutional neural network.

Dwarkesh Patel00:34:09

Yep.

Eric Jang00:34:09

The receptive fields of the convolutional network are going to be good at computing local things and making that invariant, but, um, they won't be able to kind of connect these two features easily, right? They need to sort of be pooled together and attend to each other somehow. So the argument about, you know, why Transformers are good for computer vision tasks, like with, uh, you know, vision Transformers and so forth, is that because they have a sort of global attention across the whole thing.

Dwarkesh Patel00:34:37

Yeah.

Eric Jang00:34:37

They can more easily draw these kind of predictions.

Dwarkesh Patel00:34:39

Right.

Eric Jang00:34:39

But you do need more data there so that you can kind of, uh, learn through data the— the sort of invariant local— local features.

Dwarkesh Patel00:34:47

Got it.

Eric Jang00:34:48

Um, I've tried very hard to make Transformers work for this problem because I was kind of curious if Transformers would present some sort of breakthrough in Go and just remove a lot of those tricks. But try as I might, I actually haven't figured out a way to make Transformers better than ResNets for— for now.

Dwarkesh Patel00:35:01

So one, uh, sorry, one more tangential question. It makes sense why Transformers with their, like, global pooling of information would be better if you need to consider information that is not just spatially, um, uh, or yeah, CNNs give you a sort of bias that the things that are next to you are especially irrelevant.

Eric Jang00:35:21

And then they're sort of aggregated up.

Dwarkesh Patel00:35:22

Yeah, exactly.

Eric Jang00:35:23

So, yes.

Dwarkesh Patel00:35:23

But suppose okay, so for games where it— it isn't that relevant what is happening locally, you just kind of have to consider the whole thing.

Eric Jang00:35:30

Mm-hmm.

Dwarkesh Patel00:35:30

You're saying Transformers would work better. How about games where so the particular talking about the spatial dimension, how about the temporal dimension where right now we're only considering the previous move because it is a deterministic full information game.

Eric Jang00:35:42

Yeah.

Dwarkesh Patel00:35:42

Where, um, uh, but what if it was something like poker or diplomacy where really a bluff they made a while back is sort of relevant to understanding now and isolating to decide to make your next move, and so you need to consider all those previous states? Would that then change the consideration of what inductive bias is most relevant and what architecture is most relevant?

Eric Jang00:36:02

Right. Great question. So Go is a perfect information game.

Dwarkesh Patel00:36:06

Yeah.

Eric Jang00:36:06

And in perfect information games, um, there does exist a Nash equilibrium strategy for which you can do no worse than any other strategy.

Dwarkesh Patel00:36:15

Mm-hmm.

Eric Jang00:36:16

So, um, if you know that your opponent has a particular bias, like they— they love to play aggressively, you can actually, in principle, counter that specific strategy better than a Nash equilibrium policy. But, um, to counter any given strategy, um, there does exist a single, um, Nash equilibrium that can be decided solely using the current state.

Dwarkesh Patel00:36:36

Mm-hmm.

Eric Jang00:36:36

So, um, that— that is a design choice that most Go agents, uh, AlphaGo chose to do, which in hindsight turned out to work very well because the, uh, Nash equilibrium seems to be superhuman.

Dwarkesh Patel00:36:47

Yeah.

Eric Jang00:36:47

Like, uh, like no human strategy seems to be able to beat it. Now, there are variations of this where you would actually need to consider temporal history. So and— and this is a very exciting research area that I— I would encourage people to kind of fork my repo and try these things out, which is, um, if you were to play, let's say, 2v2 Go, then you actually need to model your partner's, uh, behavior and you like you may not have information on how they play, so you need to aggregate some information on, like, how they play so that you can respond accordingly.

Dwarkesh Patel00:37:12

Yeah.

Eric Jang00:37:13

Right? Like these are, uh, situations where it's no longer a perfect information game.

Dwarkesh Patel00:37:18

Yeah.

Eric Jang00:37:18

And then in those cases in— in games of imperfect information or partial observability, then you do need some context to build a model.

Dwarkesh Patel00:37:24

Yeah.

Eric Jang00:37:24

Yeah. And— and I think that's a— a place where things get very, very exciting in terms of, like, self-play or, you know, diplomacy style.

Dwarkesh Patel00:37:31

Yeah, interesting.

Eric Jang00:37:32

Okay. So, uh, returning back to the neural network, the architecture again is not super important. You can get it to work with Transformers. You can get it to work with ResNets. I found that for low-budget experiments, uh, ResNets work a little better.

Dwarkesh Patel00:37:43

Mm-hmm.

Eric Jang00:37:44

Um, you can also use kind of a Karpathy-style auto-research hyperparameter tuning to make— make your architecture pretty good. And so— so you don't have to worry too much about that. You just need to sort of set up the problem.

Dwarkesh Patel00:37:53

Mm-hmm.

Eric Jang00:37:54

So that you have a— a sort of target optimization.

Dwarkesh Patel00:37:56

Yeah.

Eric Jang00:37:56

Okay. So we're going to pick just a somewhat arbitrary architecture that worked for— for, you know, what I did, but again, this part is not super important. Um, you have your encoded board state and, um, we're going to just choose to, let's say, do three, uh, three like, you know, similar to an RGB, we're going to have three kind of channels. Uh, one channel to encode black, one channel to encode white, and then, um, and then, uh, one channel maybe to encode like, um, uh, empties or, um, maybe like a masked region if you want to train on multiple board sizes.

I'm actually not going to talk about multiple board sizes for now. That's a little bit too complicated. So we'll just say like, you know, we've got this two or three channel, uh, RGB-like image, and then we go into a, you know, a ResNet. And then we have two branching heads. Um, one head predicts the— the value function, and this is like a— a single logit. So this is like R1. And then we have the policy, which is, you know, R361.

So, um, this is the architecture and, uh, we're going to basically train this to predict the outcomes of games given the board state. And we're also going to train this to predict what are good moves.

Dwarkesh Patel00:39:18

Yeah.

Eric Jang00:39:18

Right. So the OG AlphaGo paper, uh, or called AlphaGo Lee, um, initialized this network with a supervised learning dataset of expert human play. Um, later they removed this restriction by having the model teach itself how to play well. But I find it actually from a matter of, like, implementation for your audience, super, super nice to always kind of initialize your— your experiments to something that's easy and then, like, you know, get the problem working before, you know, trying to bite off the whole thing and learn a tabular resume.

Dwarkesh Patel00:39:47

Mm-hmm.

Eric Jang00:39:47

You— you generally want to kind of initialize just as in deep learning, initi— initialization is everything, right? Um, you always want to initialize your research project to something as close to success as possible, um, especially if you're, you know, doing something new that you haven't done before. Like always pick something that works and then get it to do something better rather than start from something that doesn't work at all and then, you know, um, try to make it work. So, um, under that philosophy, it's a great idea to start from something that, like, you know, has a good initialization. So we're going to take human expert plays, um, and train this model to predict, um, you know, good actions, right?

So we're going to take all of the winning games, all— all the moves in which a— a human won and, um, sorry, an expert won and then predict those actions. And then, uh, regardless of board state, like, you know, whether you won or lost, you're going to predict the outcome.

Dwarkesh Patel00:40:34

Yeah.

Eric Jang00:40:34

So you might be wondering like, okay, well, some of the early boards, you know, where basically only one stone has been put down, how could you possibly know whether who— who the winner of this game is, right? Well, if you have, uh, you know, hundreds of thousands of games, then in on average, you'll probably see that boards that start like this have a sort of half of the games that branch off from this will win and half of the games that branch off from this will lose. So that'll actually be fine. When you train this model to predict those, the logit will sort of converge to, uh, you know, 0.5. Um, and— and so— so for these— for these things, it's— it's sort of expected that once you train the model, a— a starting board state will look like 0.5.

And then as you progress towards the end of the game, it'll actually look something like, you know, if this is 0.5, the— the win probability will sort of either go like this or it'll— it'll go like this, right? And, um, and— and this is sort of your move number.

Dwarkesh Patel00:41:25

Yep.

Eric Jang00:41:27

And so as you, you know, get hundreds of steps into the game, it becomes much more clear like who's more likely to win or who's more— more likely to lose under your expert data distribution.

Dwarkesh Patel00:41:37

I— I didn't understand the significance of why the this way of thinking about value is especially relevant to the expert data.

Eric Jang00:41:43

It is not relevant to the expert data. It's true for any data that you train it on.

Dwarkesh Patel00:41:46

Okay.

Eric Jang00:41:47

Yeah. So if you were to learn a tabular resume, you would also expect this to fall out.

Dwarkesh Patel00:41:51

Yeah.

Eric Jang00:41:51

Yeah. So, um, if you just do this, like, so imagine, you know, you're vibe coding AlphaGo and you, um, you— you gather some expert datasets from like CutterGo online, um, or you, you know, you have a dataset of human players and you train this model. Actually, it turns out this model is already a pretty good Go player. It'll most likely beat most human players, right? So— so like if you just take this policy recommendation and take the argmax over, you know, it's, uh, if this is the, you know, probabilities, if you take the argmax and you just take this action as your Go play, um, it'll be a very, very fast Go player that doesn't think in terms of, like, reasoning steps.

It just kind of shoots from the hip and it'll be a very strong Go player, which is already quite miraculous if you think about like, you know, 10 neural network layers, maybe under like 3 million parameters can already do something that impressive.

Dwarkesh Patel00:42:43

Yeah.

Eric Jang00:42:43

Um, um, and so you can start this way and it's important when implementing this to kind of just verify that this is probably true. It's good to verify that your Go rules are implemented correctly, that like, you know, you— you can run these simulations relatively quickly, uh, and— and just as almost like a— a sort of a, um, a checkpoint that like you want to make sure that you can actually do this basic step before you try to layer on more complex, uh, things like search.

Dwarkesh Patel00:43:08

Yeah.

Eric Jang00:43:08

Um, so but yeah, we can do a lot better than taking the raw neural network and playing the moves. And, uh, this is how we can apply it to Monte Carlo tree search. So let's, uh, apply the neural network to, um, to improve Monte Carlo tree search. So we start with our root node. And we now have a four-step, uh, iterative process to do MCTS. So this tripped me up when I was first reading the paper and trying to understand it, but, um, uh, essentially what we're going to do is we're going to choose a number of simulations.

So like, you know, num simulations. And this number varies. This can be, you know, somewhere between 200 to, uh, 2048. I believe in, um, in the AlphaGo Lee match, they used tens of thousands of simulations per move because they really wanted to boost the strength of the model as much as possible.

Dwarkesh Patel00:44:04

Yeah.

Eric Jang00:44:05

Um, but in training, you don't actually need too many. And CutterGo, I think, uses something on this order as well.

Dwarkesh Patel00:44:10

Do you know if they used, uh, if you watched a documentary, they had a laptop out during the game?

Eric Jang00:44:14

Yeah.

Dwarkesh Patel00:44:14

They didn't use a laptop itself. It was like on some.

Eric Jang00:44:16

It was on some TPU pod, I think.

Dwarkesh Patel00:44:18

Oh.

Eric Jang00:44:18

Yeah. Um, but now.

Dwarkesh Patel00:44:20

I don't think it's kind of unfair.

Eric Jang00:44:21

Well, uh.

Dwarkesh Patel00:44:22

Like Lee is not using like 1e22 flops to do a move, you know?

Eric Jang00:44:27

Fair enough. Um, interestingly enough, modern Go bots don't need that much compute at test time.

Dwarkesh Patel00:44:33

Yeah, interesting.

Eric Jang00:44:33

And what we'll actually find out, uh, as we talk about how the, um, MCTS policy improvement works is that over time, the raw network actually takes all of the burden of that big TPU pod and just push— pushes it into the network. And— and you can do actually all of that work with one, you know, neural network forecast.

Dwarkesh Patel00:44:51

Yeah.

Eric Jang00:44:51

Um, but— but the TPU pod will always add the extra oomph on top. And so that's what they wanted for the match.

Dwarkesh Patel00:44:56

Yeah.

Eric Jang00:44:57

So— so we're going to pick this kind of like num simulations thing. And, uh, for every simulation, we're going to basically do several things simultaneously. We're going to see which— which moves are the best in the current tree. We're going to add extra leaves to the tree if we get to a point where we need to add a leaf. And we're going to update the action values for— for the tree. So that's— that's what every— every simulation involves, these kind of like four-step process. So— so the four-step process is basically selection, um, expansion, evaluation, and backup.

So— so at the beginning of our, um, Monte Carlo tree search, our tree is very, uh, basic. It only has the— the root node or our current board that our, um, AI wants to play at. And so we're going to basically select the best action for this. So when this root node is created, we're also— we also know that we can evaluate this under our neural network and get the quantities, um, you know, v theta as well as our, um, probability over, uh, actions.

And I'm going to say root. So for all of the actions here, we can create a bunch of children, right? So— so this one has, um, well, in— in this case, I'm drawing a 3x3 board with one— one board missing. So basically there are, um, you know, eight possible children, uh, associated with this root node. So like.

And each of these has an associated probability of taking that action, right? So— so there's P8, P1, P2, etc. Okay. So at the beginning of our Monte Carlo tree search, we have our root node and we can initialize it with some children, right? Because we know it's, uh, the— the policy network evaluated on the root node gives us on a 3x3 board with one existing, uh, stone placed, eight possible children that this, uh, AI could take. Um, so with each of the children, their policy network also gives us the probability of selecting that child.

So the, um, first step is to do the selection of the tree. And again, this is a very shallow tree. All we have so far is a tree of depth one, essentially, right? So our— our first move is to select by maximizing or argmaxing the Pucked criteria, which is basically, you know, Q, um, QSA plus, um, you know, CPucked times P of A divided by N over 1 plus NA.

So for each of these, we're going to, uh, you know, NA is zero for— for all the actions initially. N is zero. And, um, and so we're going to basically just, you know, pick according to this. Um, initially, uh, what is going to be the, you know, chosen action here is most likely going to be biased towards, um, you know, the highest likelihood action here, right? Because these are sort of uniform for everybody.

Dwarkesh Patel00:48:17

Yeah.

Eric Jang00:48:18

So, um, let's suppose P1 was the highest probability node. So you— you— you selected this one here. Now you got to this node and you realize that it's not a leaf node, right? There are more ga— it's not a terminal game, so you cannot resolve.

Dwarkesh Patel00:48:31

Yeah.

Eric Jang00:48:32

The— the final resolution. So the next step that you do is, um, expansion. So you, um, you will then run this node, this board state through the policy network. Note that this is the AI's move, right? Like A— A— AI is making this move. And so when we expand this tree, we're now thinking about what the human might do or any opponent might do, right?

Dwarkesh Patel00:48:53

Yeah.

Eric Jang00:48:53

So— so this is like, you know, your— your op— your opponent. Um, the tree expansion process actually is completely, uh, so— so— so when we evaluate the, um, the node here, we're going to now evaluate the— the node from the perspective of this player.

Dwarkesh Patel00:49:09

Yep.

Eric Jang00:49:11

Um, so then this one has possible actions that we could take. And, uh, we— we expand basically the— the— the leaf nodes here. So for each of these nodes, um, that we could, you know, arrive at, we're going to now check how good those nodes are, right? So— so maybe, um, from here, like the human could play here, the human could play here, or the human could play here. And we're going to, um, store essentially the v theta for each of these things. So v theta of, you know, node one or like node one, um, prime v theta node one prime.

Um, and— and— and so we're— we're basically using our neural network to make an intuitive guess of how good is this, um, board from the perspective of this player.

Dwarkesh Patel00:50:07

Yeah.

Eric Jang00:50:07

And, uh, fortunately, because the, uh, it's a zero-sum game, it's easy to deduce that, you know, the value for this player at this— this step is just one minus the value for, you know, from this perspective. So it's easy to flip the search process, uh, depending on which player you're at. Um, and so— so this is the expansion step. You've taken a— a non-leaf node and expanded it and evaluated the value. And this is essentially a quick guess as to like if I were to play to the end, am I going to win or not, right? So you can almost think about the v theta as a shortcut for, uh, searching to the end of the tree for— for any given simulation.

Um, and then we're go— and this is— this is essentially the evaluation step. We're— we're evaluating the quality of each of these boards. In original AlphaGo Lee, they actually did something, uh, kind of interesting, which is that they took this value and they averaged it with the value of a real Go playout. So they actually played a real game from here all the way to the end. So— so like I'm just going to draw this squiggly line to indicate some path. And, uh, they kind of like played this all the way to res Trump Taylor resolution of a full board.

And so this is like a zero or one, right? And so they took this value and they just averaged it with— with this one here. So the— the— the formula they did was like, uh, you know, alpha times v theta of— of like, you know, some— some node, um, plus, uh, sort of like 1 minus alpha of a— of a true randomly sampled playout. And you might be wondering like, okay, well, how do they play this out, right?

Like it would be very, very costly to do another search on— on this playout, like almost like a tree within a tree.

Dwarkesh Patel00:51:55

Yeah.

Eric Jang00:51:56

So they don't do this. Instead, they just, uh, take the policy network and play it against itself. So they just take this as both players and they just play it all the way to the end. And— and, um, this is something that helps ground the, um, the estimates here in— in reality because you can get a single sample estimate of like whether you win or not. You can think about in the end game where the board is almost resolved that this one actually becomes quite useful because the random— the— the play according to the policy will most likely decide a pretty reasonable guess of the game. And so you're not, you know, facing a problem where this one kind of becomes untethered from— from reality.

It turns out this is totally unnecessary. So in all subsequent papers after AlphaGo Lee, they just got rid of this.

Dwarkesh Patel00:52:36

Yeah.

Eric Jang00:52:36

And so in my implementation, I also did the same and it speeds things up a lot because you don't have to roll these games out on every single simulation.

Dwarkesh Patel00:52:42

Yeah. Okay. So, uh, again, just to, uh, reinforce my own understanding and just to re-explain it for the audience, by the way, in case it's not obvious, the P there in the select, that is the probability coming from the network in this case.

Eric Jang00:52:56

Correct. The policy network here.

Dwarkesh Patel00:52:58

Yeah. Okay. So fundamentally, a simulation, just think of it as like rolling out one more node in the search process.

Eric Jang00:53:09

Um, almost. So a simulation is easy to think about when the whole tree already exists, right? You just walk down the tree, um, using the Pucked selection criteria and you— you, uh, and then— and then you keep going.

Dwarkesh Patel00:53:23

Yeah.

Eric Jang00:53:24

Now, uh, in AlphaGo, the— the data structure is such that we begin with a tree that has no like basically only depth one, which is its only children.

Dwarkesh Patel00:53:33

Yeah.

Eric Jang00:53:33

And you want to iteratively build out the tree.

Dwarkesh Patel00:53:36

Yeah.

Eric Jang00:53:36

As you're also selecting actions down the tree. So that's the kind of core thing here is that because Go is such a combinatorially complex game, you cannot afford to build the tree in advance and then search it. You must search while building the tree.

Dwarkesh Patel00:53:48

Right. Okay.

Eric Jang00:53:49

So, um, let me just finish up with actually the last step, which is the backup, right? So once you've scored these things, you basically take the mean, the— the value, the— the Q value assigned to the node here for taking this action is now just the average across your evaluated values here.

Dwarkesh Patel00:54:07

Yeah.

Eric Jang00:54:07

It's, uh, you take a running mean over— over, uh, all of the, um, the— the simulations that you've taken and they average the values of the children nodes.

Dwarkesh Patel00:54:16

Yeah.

Eric Jang00:54:17

So— so that's what is known as the backup step. And once you evaluate this, you can actually kind of recursively go back. So if you know the, you know, the action value of this node, you can then take the average on its parent and so on and so forth. So— so you have this kind of four-step process where you are choosing the best action that you know of so far. Then you may run into a node where you, uh, you— you haven't been to before, so you need to grow the tree a bit. And then you run it through the network to guess whether you're going to win or not. And then you walk all the way back up to the— to the root node to update your values on what the best moves are.

So as you do this iteratively, this selection criteria will cause you to visit the— because you're always selecting according to this criteria, you're always going to be selecting the best action you think at any given branch, right? So— so the final, um, visit counts of like how often you chose these things will reflect your correct policy distribution as induced through this search process. Um, and so— so the visit count that we store in the node earlier actually becomes the sort of vote for like which way we should finally select an action here.

Dwarkesh Patel00:55:23

Yep.

Eric Jang00:55:23

So, um, you know, as a sort of test of understanding, it's worth thinking a little bit about whether we could make this even simpler, right? Like could we actually maybe even get rid of this one and still make the thing work? Um, so recall that, you know, when you do an expansion and then an evaluation at, let's say, this node, you— you are checking the sort of win probability of each of the child nodes, right? And so if this one is, you know, like one and these are zero, um, you do kind of know something about which action might be better to take.

And so why would you need still need this, right? Like why not just, uh, normalize this one into some distribution and call that your— your policy distribution? Um, this is fine. You can do this and, um, this probably does work, but in practice, having a single forward pass that gives you a pretty good guess is, um, is how the, uh, the breadth is— is, uh, is pruned out. Um, the— there is a sort of duality here. Like it would be weird if, let's say, the policy recommended an action that disagreed with the value, right?

If— if, let's say, the policy said this was very high probability, but this one said it was a, you know, low value, then there's actually something kind of fundamentally wrong between, um, between your policy head and your value head. So— so they are linked and, uh, you probably could get rid of this if you came up with a different way to recover this from just the value evaluations.

Dwarkesh Patel00:56:44

Right. But— but just to make sure I understand, the reason you don't do that is so that you don't have to do 360 independent, uh, forward passes to like, okay, here's the value of everything and let's argmax over it, right? Instead, you can just do one forward pass and get like the probabilities of all of them.

Eric Jang00:56:59

Um, you can usually batch these somewhat efficiently.

Dwarkesh Patel00:57:03

Mm.

Eric Jang00:57:03

Um, so it probably is not a huge computational burden in practice.

Dwarkesh Patel00:57:07

Mm.

Eric Jang00:57:08

But yes, you would have to pass 361 board like up to 361 boards into a single mini batch update.

Dwarkesh Patel00:57:14

Yeah.

Eric Jang00:57:14

To evaluate all the values here, then normalize them. Um, now there's actually a more, uh, important reason why we still do this, which is how Monte Carlo tree search is used to feed back on itself.

Dwarkesh Patel00:57:25

Mm.

Eric Jang00:57:25

Um, and— and sort of recursively improve its own predictions and search capabilities.

Dwarkesh Patel00:57:29

Right.

Eric Jang00:57:30

And that's where this— this one, having this as an explicit entity you're modeling rather than an implicit normalization over your value is— is a— is a good idea.

Dwarkesh Patel00:57:38

Makes sense. Okay.

Eric Jang00:57:39

Okay. So, um, so we— we talked about the simulations and basically, you know, what you end up with as you roll out the number of simulations is a tree that kind of looks like I'm— I'm— I'm drawing a very low dimensional version of this. Of course, it's in— in— in the real game, it's— it's much more high dimensional, but like you— you'll end up with basically a tree structure that like has, um, a— a lot of leaves that kind of terminate and are not visited again because their value is deemed to be too low. But then, you know, along one path, there will be a set of actions with very, very high visit counts that kind of gravitate towards that one set of decisions as you increase, uh, end here.

So— so this is kind of like the— the mental picture of what the tree in Monte Carlo tree search looks like. And you should contrast this with like an exhaustive tree like in Tic Tac Toe where you could say like, you know, there's— there's nine actions and then eight and then seven and six. And so it's sort of like nine factorial, uh, sized, um, tree. Um, the Monte Carlo tree search in Go is very, very sparse, right? It only considers the paths that you've expanded children nodes on. Okay. So, um, now that we have the search algorithm that applies the value function as well as the policy function, um, we can now talk about how the Monte Carlo tree search algorithm can actually act as a improvement operator on top of these guys here.

Dwarkesh Patel00:59:06

20 years ago, Jane Street's data center fit in the corner of an office. Ron Minsky, who co-leads the tech group there, told me about how it all got started. One of our compute clusters we called The Hive. And I remember the first version of The Hive was literally like six Dell boxes stacked on top of each other at the end of the row. And the trading systems themselves we also had there because we actually wanted the ability to make sure we could turn the damn thing off. I mean, there were ups and downs. Like literally at some point, you know, one of the people who was cleaning the office unplugged one of the trading systems in the middle of the day as they were vacuuming.

Advertisement00:59:39
Eric Jang01:00:34

Okay. So, um, we now talk about the RL part of like how this thing gets stronger by playing itself, right? Um, let's say we play a game where you have the AI. So you make a move. AI— AI, um, will— will kind of compute the search and then this is the sort of visit count distribution. Um, let— let's say this is your policy— your policy initial policy recommendation at the— at the— at this node.

Dwarkesh Patel01:01:03

Mm-hmm.

Eric Jang01:01:04

And then after MCTS, it, uh, gets more confident about one of these actions, right? And— and so maybe the— the distribution looks a bit more picky like this based on the— the search. Now, of course, you can tune the search process so that it ends up more diffuse, but that's probably not a good idea. MCTS should get more confident about specific actions, um, than others, but it of course might place a lot of weight on, you know, other actions initially. And then as you increase the number of sims, it should converge to a very picky distribution. Um, so— so this is your new, uh, let's call this like pi— let— let's wrap this in like a MCTS operator of, you know, A given S, right?

So after applying MCTS process, your— your policy recommended distribution looks like this. It's— it's a bit more picky than— than the previous one. Um, and so then you take the argmax or maybe you just sample from this. Uh, it doesn't have to be the argmax. And then you— you make your move. And then, um, and then you throw away the tree and then you— you begin anew on the next move, right? So again, like you, um, you know, you compute a new distribution.

So initially maybe your guess looks like this and then you refine it through MCTS.

Dwarkesh Patel01:02:28

There should be one more X on the board, right?

Eric Jang01:02:30

I'm sorry. That's correct. Yes. Um, to something that looks like this, right? So— so on every move, you have your initial guess from your policy network. Um, and then the search process that combines your policy network and your value network arrives at a more confident action that you take.

Dwarkesh Patel01:02:50

Mm-hmm.

Eric Jang01:02:50

And— and then so and so forth. And then the game ends and one person wins and one person loses. So a, um, the way that, uh, the— the beauty of— of how AlphaGo trains itself is that it actually can take this final search process, the outcome of the search process, and tell the policy network, "Hey, like, you know, instead of having MCTS do all this, you know, legwork to arrive here, why don't you just predict that from the get-go, right? Like why don't you like, you know, not use this guess and just predict this to begin with?" And if you have this guess to begin with in your policy network, then MCTS has to do a lot less work to— to get things to work.

And so if we draw like a sort of test time scaling plot, um, so— so let's say like this is like number of simulations. Um, let's say, you know, at— at zero simulations, your— your sort of implicit win rate is like, um, is like, I don't know, here. And then— and then, um, without any simulation, if you just take this raw action, this is what your win rate is. And let's say as we increase the number of sims, um, maybe— maybe you kind of have a— a win rate that looks like this, right? So, um, when you search for, let's say, 1,000 simulation steps, that gets you to a, um, a policy here that gets you to here, which is great.

But if you were to distill this MCTS policy network back into your sort of, uh, shoot from the hip policy network, then you could actually, um, uh, you know, start here. Like if— if let's say this was, you know, zero, um, for, uh, by— by distillation, then if you spend another 1,000 sim steps, then you actually kind of get to here. It's almost like if you could just, you know, um, amortize the— the first 1,000 steps actually into the policy network.

Dwarkesh Patel01:04:38

Yeah.

Eric Jang01:04:38

Instead of the search process, then you could begin at a much better starting point and then get a much better result for— for your, uh, for the number of sims that you put.

Dwarkesh Patel01:04:46

The sigmoid type nature of test time scaling as the number of simulations increases, the— the increase in win rate is, uh, smaller. Is that true even for the distilled network? That is to say, is there some gain of like, okay, we start from the distilled, we get these early gains again, or is that just inherent to like the nature of.

Eric Jang01:05:02

Yeah.

Dwarkesh Patel01:05:02

MCTS?

Eric Jang01:05:03

To be honest, I actually don't know the test time scaling behavior of MCTS simulations. And I— I believe it might actually be quite sensitive to how strong this one is in practice.

Dwarkesh Patel01:05:12

Yeah.

Eric Jang01:05:12

I'm just drawing a monotonically increasing function that gets to one.

Dwarkesh Patel01:05:15

Okay. Cool.

Eric Jang01:05:15

So, yeah. So— so don't pay too much attention to the shape of the curve. Just know that it's monotonic with respect to sim. Um, okay. So, um, so— so the idea of MCTS is very brilliant, which is like we're gonna— we got something better by applying search.

Dwarkesh Patel01:05:29

Yeah.

Eric Jang01:05:30

And, um, we're going to now on our next iteration of updating this network, just train this to approximate the outcome of 1,000 steps of search.

Dwarkesh Patel01:05:39

Yeah.

Eric Jang01:05:39

And so instead of starting here, we get to now have a neural network start here and then— and then, you know, the— the play gets stronger once we then apply another 1,000 steps on top of it.

Dwarkesh Patel01:05:47

Yeah.

Eric Jang01:05:48

And you can keep going, right? So— so the training algorithm for AlphaGo is to basically take the games where you've applied the search on every move that the policy encountered, uh, whether you won or lost. And that's quite important. Um, and you're just gonna train the— the model to imitate the search process. So, um, there's an analogy to robotics actually, which is the dagger algorithm.

Dwarkesh Patel01:06:09

Mm.

Eric Jang01:06:10

Um, uh, first— first I'm gonna draw like a, uh, schematic of like, let's say, you know, the states, right? So S0, S1, S2, S3. So let's say, you know, we— we took a series of actions in an MDP to— to get, uh, a trajectory. And these actions may be suboptimal, right? Maybe we lost at the end of this game. So, um, there is a family of algorithms that basically take trajectories, uh, and relabel the actions to better trajectories.

So maybe a better action here would have been to take, you know, A0 prime. A better action here would have been to take A1 prime and then yet another one like A2 prime, A3 prime. So, um, what MCTS is doing is basically saying like you play this game where you eventually lost, but on every single action, I'm gonna give you a strictly better action that you should take instead. It does not guarantee that you are going to win, but it does guarantee that, you know, if you take these tuples as training data so that you retrain your, uh, your— your policy network to predict these ones instead of these ones, you're gonna do better.

And, uh, this is very related to dagger in, uh, robotics and in imitation learning where you want to collect a intervention here. And even if you're in a— in a not great state, for example, like a self-driving car that, you know, veers off the side of the road, there is still a valid action that kind of corrects you and brings you back.

Dwarkesh Patel01:07:42

Yeah. Okay. So, um, pretending question, but is there a guarantee that MCTS must be better than the policy? For example, you could imagine early on in training because MCTS is informed by the value network.

Eric Jang01:07:55

Yeah.

Dwarkesh Patel01:07:55

Early on in training, uh, when the value network hasn't been well trained on finished, uh, games, um, that like MCTS is worse than sort of randomly exercised policy. So is it just like a heuristic that MCTS is better than the policy, or is that like is there some guarantee?

Eric Jang01:08:10

Right. In— in practice, it is a heuristic. Um, and it does work in also in practice, but let me illustrate an example where MCTS can give you a worse distribution than your policy network.

Dwarkesh Patel01:08:20

Yeah.

Eric Jang01:08:20

So, um, and this can often happen if your self-play algorithm, um, has trained to a good point, but then somehow it's, uh, it collapses because it's, um, it's not trained on diverse data or something, right? So, um, let's say we have a board state where, um, the policy recommendations here are very good. So— so like, you know, pi of AS is like great, but, um, somehow because maybe we're playing on a lot of games where the— the bots just resign instead of playing all the way to the Trump Taylor resolution, they kind of forget, um, how to evaluate those kind of late stage plants, right?

Like in the— in the case that we showed with the corner play, maybe like 100% of our training data in our replay buffer has lost examples of how to evaluate the value function at those states. So you might end up in a scenario where your terminal value, um, is like very bad. And if the terminal values of the leaves are not good, then this will actually propagate all the way up and cause your, um, your— your puck selection criteria and your backups to be off.

Dwarkesh Patel01:09:24

Yeah.

Eric Jang01:09:24

And then you'll end up visiting a very, very different distribution than what your policy initially recommended.

Dwarkesh Patel01:09:29

Yeah.

Eric Jang01:09:29

Um, also if your number of sims is low, then you might also have a variance issue where you just don't explore enough, right?

Dwarkesh Patel01:09:35

Yeah.

Eric Jang01:09:35

Like— like a it's only guaranteed to converge when you kind of take n to infinity.

Dwarkesh Patel01:09:39

Yeah.

Eric Jang01:09:39

Um, so— so variance in, you know, your search process as well as inaccuracies in your evaluation can definitely screw with the quality of your policy network recommendation.

Dwarkesh Patel01:09:50

Yeah.

Eric Jang01:09:50

And so that's why it's not a guarantee to improve.

Dwarkesh Patel01:09:52

Yeah.

Eric Jang01:09:52

But and— and that is why I think I— I suspect why AlphaGo Lee had the, uh, playouts to the end in their training algorithm.

Dwarkesh Patel01:09:59

Yeah.

Eric Jang01:09:59

So that they could ground this thing in real playouts.

Dwarkesh Patel01:10:02

Yeah.

Eric Jang01:10:02

Um, in practice, what you could also do is just like for 10% of the games, you— you prevent the bots from resigning and you just say like resolve it to the end.

Dwarkesh Patel01:10:09

Mm-hmm.

Eric Jang01:10:09

So you get some training data in your replay buffer to really resolve those kind of like late stage playouts that normal human players would— would kind of, uh, not play to.

Dwarkesh Patel01:10:18

Yeah. Yeah.

Eric Jang01:10:19

So, um, so this is why MCTS kind of if you assume that the value functions are correct, uh, why it gives you a better policy is because assum and it's a very critical, you know, chain of assumptions. Assuming that this is accurate, then, uh, your search process should give you a better recommendation than your initial guess.

Dwarkesh Patel01:10:36

Right. Okay. So if you have a cold started policy, uh, if you have an AlphaZero type thing.

Eric Jang01:10:41

Yeah.

Dwarkesh Patel01:10:41

Really what's happening for the first few epochs is the policy is kind of useless and what you're really just doing is, "Hey, let's play full games and, uh, once we have played full games for the preceding moves, we'll have labeled who won, who didn't win." And the loss for AlphaZero has two components, which is like how good is the policy relative to MCTS and how good is the value prediction relative to who actually won the game from this move. And this is this sort of like you can think of this being applied to every single action or every single move.

Eric Jang01:11:14

Correct.

Dwarkesh Patel01:11:14

And really what's happening in the beginning of AlphaZero training is just like we're trying to get the value function to actually predict who will win the game if you're if you find yourself in this state and you're this player. Uh, and functionally that's all that's happening. And later on once that's well trained, now the policy is also improving.

Eric Jang01:11:31

Correct.

Dwarkesh Patel01:11:32

Okay.

Eric Jang01:11:32

One trick I did find to be pretty useful, and this is not a peer-reviewed claim, so just like take this with a grain of salt, is like I— I found it useful in my own implementation to do the do the following. Um, you wanna first make sure that this is good before you invest a lot of cycles doing MCTS.

Dwarkesh Patel01:11:46

Yeah.

Eric Jang01:11:47

Right? Like— like it— it doesn't really make a lot of sense to do search on garbage value predictions.

Dwarkesh Patel01:11:52

Yeah.

Eric Jang01:11:52

Um, so— so you wanna kind of start at a good place where this works. AlphaGo Lee does a very good thing where it just takes human games and then you— you like, uh, train on it and it just works, right? Totally works. Um, you could also take an open source GoBot, play it against itself, um, generate data, also works. Uh, so— so if you have some like offline data set that, um, that, uh, has realistic good play, you can easily learn the late stage, um, value functions pretty well. And that's the that's what you kind of need to start the search process.

Dwarkesh Patel01:12:21

Sorry, can you just read that sentence one more time?

Eric Jang01:12:23

Sure. So— so, um, it— it's quite easy to evaluate a late stage Go— Go game. Like when almost all the pieces are on the board.

Dwarkesh Patel01:12:29

Yeah.

Eric Jang01:12:30

Like it's almost like a decidable problem, right? Because there's lower and lower uncertainty as to like the depth of the tree.

Dwarkesh Patel01:12:35

Yeah.

Eric Jang01:12:35

So, um, most games played to the end by reasonable people, um, will be good training data to train a good value function at terminal parts of the tree.

Dwarkesh Patel01:12:43

Got it. Okay.

Eric Jang01:12:44

Then as you play more games, the, um, the— the search will back up good values into the— the sort of intermediate nodes of the tree.

Dwarkesh Patel01:12:51

Yeah.

Eric Jang01:12:51

And then like as you increase the amount of data, your— your value head gets a good intuition of like what is a healthy board state versus a not healthy board state.

Dwarkesh Patel01:13:00

Yeah.

Eric Jang01:13:00

That— that those are much more subtle to judge in the midgame than the beginning or the end. So— so the most difficult part to score is like not the beginning or the end 'cause the beginning is just like obviously 0.5 and then at the end it's like pretty obvious who's winning.

Dwarkesh Patel01:13:11

Yeah.

Eric Jang01:13:12

So— so the hard part that you wanna learn in the value function is like who is winning in the middle.

Dwarkesh Patel01:13:15

And so this is actually very analogous to TD learning.

Eric Jang01:13:18

Yes. And there's a— a beautiful connection to TD learning that we can, you know, talk about in a bit.

Dwarkesh Patel01:13:22

Okay.

Eric Jang01:13:22

Uh, as opposed to, you know, contrasting with Monte Carlo tree search. So, um, so you first wanna get good value functions and expert data can kind of give you a quick shortcut. I recommend for, you know, practitioners just do that first just to, you know, initialize to a good starting point. And then if you wanna do the AlphaZero thing or, uh, or KataGo kind of, uh, tabula rasa learning, um, then what you can try to do is on a small board play random games. Just take a random agent. And if you play like, uh, you know, 50,000 games, you'll actually learn a pretty good value function as well because on a 9x9 board, there's actually, um, you can see enough of the common patterns with random play.

And then if you train a model that kind of can train on both 9x9 and 19x9 data, um, and KataGo was a was a proposed one of these architectures, then, um, there's some pretty good transfer learning from the value, uh, value head evaluated at 9x9 to the 19x9.

Dwarkesh Patel01:14:12

Right. Because this— this unlike other games has much like a very much a sense of like there's not like a new kind of piece that is introduced when you.

Eric Jang01:14:19

Yeah.

Dwarkesh Patel01:14:19

Increase the size or something.

Eric Jang01:14:20

If— if we take it to its limit and consider like a very tiny like 4x4 Go board.

Dwarkesh Patel01:14:24

Yeah.

Eric Jang01:14:24

Like if you play 50,000 games, you're gonna have a lot of end states that look like human play, right?

Dwarkesh Patel01:14:28

Yeah.

Eric Jang01:14:28

Like it's just like tic tac toe at that point.

Dwarkesh Patel01:14:30

Yeah.

Eric Jang01:14:30

Um, so— so if you like broaden this a little bit to like 9x9 or 5x5 or 9x9, it's not unrealistic to imagine that like purely random play will actually generate pretty reasonable looking.

Dwarkesh Patel01:14:39

Yeah.

Eric Jang01:14:40

Boards. Um, and then so you can score those pretty easily. And so that is what gives you the bootstrapping to be able to then improve your policy.

Dwarkesh Patel01:14:46

Yeah.

Eric Jang01:14:46

With search. But it's very, very critical that MCTS has accurate value, uh, estimates.

Dwarkesh Patel01:14:51

Yeah.

Eric Jang01:14:52

And you need to ground the value. Ultimately, MCTS will fall apart if you don't have a grounding, uh, function for, um, for the value.

Dwarkesh Patel01:14:59

I— I'd be curious how much compute you save by training the value and policy on the same network that because they share the same representations, how much more efficient learning is? 'Cause that would be interesting if, um, they're basically kind of we've just talked about how they're kind of making similar predictions or they should be in line with each other.

Eric Jang01:15:16

Yeah.

Dwarkesh Patel01:15:16

And so I'd be curious if like actually, yeah, you just you're— you're like halving the amount of compute you had to do by keeping the same network.

Eric Jang01:15:22

Right. AlphaGo Lee, the original AlphaGo paper had two separate networks.

Dwarkesh Patel01:15:25

Yeah.

Eric Jang01:15:25

And then, um, in all subsequent papers, um, they— they merged them into two heads.

Dwarkesh Patel01:15:30

Yeah.

Eric Jang01:15:30

And presumably this saves compute. But answering that question in a very rigorous scientific way is actually, uh, it's— it's a simple question, but— but in practice actually takes like if you really wanna chase that question down to its— its limit, it's, uh, it takes quite a bit of work to, you know, really resolve that.

Dwarkesh Patel01:15:43

Yeah. Yeah.

Eric Jang01:15:44

Um, but intuitively, yes, they share a lot of representations. So— so and, you know, as we mentioned, there is— there is a sort of like your— your policy network and your value network when doing evaluation should kind of agree, right? So there— there really should be this sort of consistency between them.

Dwarkesh Patel01:15:57

Yeah. Tell me if this is the wrong way to think about it. I feel like when I learn how an LLM works and how simple RLVR is, at least as an algorithm, how simple it is, I'm sort of stunned by the kinds of things it can do, uh, that it can learn how to build very complicated code repositories and whatever simply from getting like a yes/no. And here I feel like the if you understand it more deeply of like just predicting MCTS and it actually seems AlphaGo seems less impressive in retrospect the more you understand it because you're like, "Oh, you're putting in a lot of bias by just saying how much you'd ex you're like telling it how it should titrate exploration as things go on.

You're building this very explicit, uh, tree search for it." And so I don't know if you show that intuition where it actually the more you understand it, the less impressive the accomplishment in 2017 seems.

Eric Jang01:16:51

I, uh, I personally disagree. I— I think they're profound for different reasons.

Dwarkesh Patel01:16:54

Mm-hmm.

Eric Jang01:16:55

And I don't understand the LLM RL like enough to like kind of comment on your podcast.

Dwarkesh Patel01:16:59

Yeah.

Eric Jang01:16:59

About it. Um, but, um, I think AlphaGo so— so, yeah, why is it a profound accomplishment? I think maybe it's worth stepping back a little bit.

Dwarkesh Patel01:17:07

Yeah.

Eric Jang01:17:07

And just like, uh, it is different than modern RL and we can talk a little bit about like some of the algorithmic choices there. But, um, I think the most profound thing here is that a, um, a 10-layer neural network pass.

Dwarkesh Patel01:17:20

Yeah.

Eric Jang01:17:20

So basically, uh, 10 steps of 10 steps of reasoning.

Dwarkesh Patel01:17:25

Yeah.

Eric Jang01:17:25

And of course, the reasoning is not just one trail of thought. It could be like the distributed representations and a lot of thoughts going on at the same time. But, uh, by construction, let's say a 10-layer neural network can only do 10 sequential steps of thinking.

Dwarkesh Patel01:17:37

Yeah.

Eric Jang01:17:38

Right? Um, 10 steps of neural network paralyzed distributed representation thinking is able to amortize and approximate to a very, very high fidelity a nearly intractable search problem.

Dwarkesh Patel01:17:51

Yeah.

Eric Jang01:17:52

So— so this was a breakthrough that I think most people don't even understand today, like fully comprehend like how profound that accomplishment is.

Dwarkesh Patel01:18:01

Mm-hmm.

Eric Jang01:18:01

And then this is what also girds like, um, AlphaFold, for example, right? Like, um, uh, where you have a very, very difficult physical simulation process that you would need to roll out so many micro-scale simulations and yet like 10 steps of a somewhat small neural network can somehow capture what feels like a, you know, NP-class, uh, problem.

Dwarkesh Patel01:18:23

Right.

Eric Jang01:18:23

Into a single, um, problem. And— and so I— I it— it actually makes me wonder if, you know, our understanding of problems like P equals NP or, you know, these very fundamental like computational hardness problems are incomplete, right? Like, like.

Dwarkesh Patel01:18:39

Yeah.

Eric Jang01:18:39

It's— it's not like, you know, obviously this is not a proof of like P equals NP or anything, but.

Dwarkesh Patel01:18:44

Yeah.

Eric Jang01:18:44

But there's something to it that like kind of is very disturbing where like what felt like a very hard problem can fall to a very, very simple macroscopic solution.

Dwarkesh Patel01:18:52

Yeah. That is a very interesting insight that a lot of problems which are proven to be NP-hard, like I don't know if Go is proven to be NP-hard, but.

Eric Jang01:19:00

It is.

Dwarkesh Patel01:19:01

Okay. Protein folding, et cetera, have been like neural networks can solve them 'cause they're NP-hard in the worst case, but we're not dealing with the worst. We're usually not concerned about the worst case. We're, you know, like these.

Eric Jang01:19:11

Yeah.

Dwarkesh Patel01:19:11

These problems have a lot of structure to them, usually.

Eric Jang01:19:12

Yeah. I— I think that the— the kind of question we should be asking ourselves is like we've been formulating, you know, solutions to NP-hard problems as in like kind of worst case complexity.

Dwarkesh Patel01:19:23

Mm-hmm.

Eric Jang01:19:24

And I wouldn't say, you know, this solves Go, right? It doesn't give us a exact solution of the optimum.

Dwarkesh Patel01:19:28

Yeah.

Eric Jang01:19:28

But in practice, like it is extremely useful.

Dwarkesh Patel01:19:31

Yeah.

Eric Jang01:19:32

And the same thing has been shown in like AlphaTensor, AlphaFold, where like yes, there is a very hard problem that in the worst case seems intractable.

Dwarkesh Patel01:19:40

Yeah.

Eric Jang01:19:40

And yet we're able to make like almost arbitrary amounts of progress. So— so here's a sort of like, uh, you know, in the limit, what is what— what— what might this look like, right? Well, um, if you want to simulate, you know, something very complex like weather or, um, predict the future, like, you know, do we live in a simulation or not, um, the computing resources you need to build a very complex simulation might be much smaller than you think based on, you know, our ability to amortize a lot of that computation into the forward pass of a single network.

Dwarkesh Patel01:20:09

Interesting.

Eric Jang01:20:10

So to me, yeah, AlphaGo was the first paper that kind of like really showed this like profound level of, you know, simulation being compressed into a small amount of computing.

Dwarkesh Patel01:20:19

Um, I— I feel totally not at all qualified on the computational complexity or the math to comment on this. But I wonder if, um, there's an important role of chaos here where if under what is the problem with weather and why does it take 10x the amount of resources to predict weather a day out, uh, and.

Eric Jang01:20:39

Right.

Dwarkesh Patel01:20:40

Continually so for every more day out is because it's a chaotic system. And so small perturbations can totally change the final estimate as— as time goes on. And, um, I guess it's interesting. Well, I guess you would expect that for Go and protein folding as well.

Eric Jang01:20:55

So here's an analogy to weather that— that might be relevant in Go. So, um, the problem of like, you know, here's our current board state.

Dwarkesh Patel01:21:01

Yeah.

Eric Jang01:21:05

Um, given what we know about both players, what is the board state in the future?

Dwarkesh Patel01:21:11

Yeah.

Eric Jang01:21:11

What is the exact board state in the future?

Dwarkesh Patel01:21:13

Yeah.

Eric Jang01:21:13

Right? This is, uh, this is extremely sensitive to initial conditions.

Dwarkesh Patel01:21:18

Yeah.

Eric Jang01:21:18

Like a single stone placed here can kind of disrupt the entire prediction.

Dwarkesh Patel01:21:21

Yeah.

Eric Jang01:21:21

Right? So this is hard. This is kind of intuitively the chaotic problem. Um, and yet somehow so this is— this is hard. Somehow we can predict who's gonna win.

Dwarkesh Patel01:21:33

Yeah.

Eric Jang01:21:34

Like, and this captures a lot of possibilities here.

Dwarkesh Patel01:21:37

Mm-hmm.

Eric Jang01:21:38

And so that there's this more macroscopic quantity that we really care about, which is the average or expectation or some sort of global.

Dwarkesh Patel01:21:44

Mm-hmm.

Eric Jang01:21:44

Macrostructure over a lot of like, you know, possible futures.

Dwarkesh Patel01:21:48

Yeah. It's interesting way to think about it.

Eric Jang01:21:49

And so— so in— in weather it could be the same thing, right? Like we don't exactly care like what the, you know, velocity of wind 6,000, uh, feet above a specific latitude longitude is. We kind of care like where's the hurricane or, you know, you know, things like that. And— and I would say like in chaos, you know, there's a classic like Lorenz attractor, which kind of looks like this, right? Um, yes, you— you don't if you start anywhere on the Lorenz attractor, you don't know where you're gonna end up, but you— you do know that the thing looks like this.

Dwarkesh Patel01:22:16

Yeah. Yeah.

Eric Jang01:22:17

Right? And— and so there's this kind of beauty of like sometimes we don't necessarily care about the micro-scale things.

Dwarkesh Patel01:22:22

Mm-hmm.

Eric Jang01:22:22

We actually care about the macroscopic structure.

Dwarkesh Patel01:22:24

That's interesting.

Eric Jang01:22:24

And— and these things can be predictable.

Dwarkesh Patel01:22:26

And contrast that, say, to something like a hash function, which is also incredibly, uh, dependent on initial conditions, but doesn't have a macro structure, or at least hopefully if like the algorithms work.

Eric Jang01:22:37

One would help. Yes, one would help.

Dwarkesh Patel01:22:37

Um, and so there's like no equivalent of a value function or like broadly how's the weather gonna be that is interesting there. It's really just about what is the move? What is a board gonna look like a hundred moves from now exactly?

Eric Jang01:22:50

Uh, yes, intuitively that seems correct. I— I, uh, and then again, this is also out of my— my area of expertise, but I, um, I find it interesting that like cryptography.

Dwarkesh Patel01:23:00

Cryptography.

Eric Jang01:23:01

Not to the tools of cryptography and, uh, you know, um, hashing have also not been able to prove that like, uh, you cannot come up with fast approximation. Like you cannot come up with fast approximations, right? Like if that if they were able to do that, then you could prove P is not equal to NP.

Dwarkesh Patel01:23:17

Yeah. Yeah. In fact, we know that there's structure in many cryptographic protocols, obviously like, uh, uh, RSA cryptography. There is structure and that structure is what quantum computers exploit.

Eric Jang01:23:26

I see.

Dwarkesh Patel01:23:26

To, uh, to break them, right?

Eric Jang01:23:27

I see.

Dwarkesh Patel01:23:28

Um, Reiner has a very interesting blog post, which we talked about in the episode where he, uh, talks about how if you look at, at a high level, what cryptographic protocols look like and what neural networks look like, it's extremely similar.

Eric Jang01:23:40

Mm-hmm.

Dwarkesh Patel01:23:40

Where you have sequential layers of jumbling information together. And it's because there's this conversion devolution in the algorithms where in cryptography you want the final state to be incredibly sensitive to initial conditions so that it can come out sort of looking jumbled based on if you change anything. And then neural networks, you similarly want everything to be dependent on all the information because you want to process all the information and consider how it relates to itself.

Eric Jang01:24:08

Yeah. You have the maximum power of a neural network at the edge of chaos. Um, I think there's some like research papers from, uh, Joshua Stroldik's team.

Dwarkesh Patel01:24:15

Yeah.

Eric Jang01:24:15

On, on, on this.

Dwarkesh Patel01:24:16

Yeah.

Eric Jang01:24:17

Yeah. Like— like there's something kind of quite fundamental about like chaos that is it's not just like hopeless noise. It's like there's something kind of useful, right, in— in, um, in chaotic systems.

Dwarkesh Patel01:24:27

Yeah.

Eric Jang01:24:27

At least at that boundary. Um, but yeah, this is just my like think about this as a philosophy. I don't— I don't actually know the math, uh, well enough to comment on it. Anyway, if we, um, go back to, um, we'll talk about LLM RL in a little bit 'cause there's some connections there, but let— let's just go back to like the MCTS. Like what is it doing? It is not, um, crucially it is not saying we're going to, uh, increase the probability of winning directly. It's not gonna say like we're gonna, um, upweight all actions that won and downweight all actions that didn't win.

Dwarkesh Patel01:24:56

Yeah.

Eric Jang01:24:56

Um, importantly, what it is doing is saying for every action we took, we did a pretty exhaustive search, uh, on MCTS to see if we could do better. And we're just gonna make every action that we took better by predict like having the policy network predict that outcome instead. And— and so this is, um, a very, very nice idea because you have one supervision target for every single action.

Dwarkesh Patel01:25:19

Yeah.

Eric Jang01:25:19

So the variance of your— your learning signal is very low compared to the alternative naive, uh, RL thing. So— so let's actually consider what let— let's consider a very naive algorithm, uh, uh, that looks a lot more like, you know, modern LLM RL today where— where we do something like, um, let's take the winner of a self-play game and encourage it to do more of that. Okay. So, uh, it's worth kind of thinking a little bit about like, okay, what are some alternatives that we could do to train self-play agents instead of MCTS, right? Like, you know, we use a lot of, uh, LLM style RL these days.

Like is that relevant? Could we do that instead? Um, so— so let— let's think through this a little bit. Let's suppose we have a very naive algorithm where we take a league of agents of different checkpoints and we play them against each other. And, um, for— for the games where a single player wins, uh, we're gonna reinforce those actions up and then— and then, uh, and— and train retrain the policy network to imitate those— those guys instead of, uh, instead of the MCTS objective. Um, so what ends up happening is let's say you have a chain of actions that led to a win and you're— you're you have a matchup between two agents that are basically the same.

So in fact, let's just assume that like, uh, you know, policy A and policy B are like evenly matched, right? So their true— their true win rate is— is like 50%. Um, so let's say you play a hundred games and then, uh, each game, let's say lasts, you know, 300, um, moves. And, um, you're doing some sort of like evolution strategy or some way to perturb these things to get them— get them to do different things, or maybe you don't and you just play them against each other and you see like occasionally this one might actually have a better strategy than this one, right?

Dwarkesh Patel01:27:12

Yeah.

Eric Jang01:27:12

And so— so let's say, um, you know, 51 games, um, policy A wins, and then 49 games, policy B wins. And this is just due to random luck or maybe you perturbed policy A in some way that let it do this.

Dwarkesh Patel01:27:29

Yeah.

Eric Jang01:27:29

And just to have a very, very simple model, let's pretend that for, uh, for like, uh, 49 of the games, they played exactly equally. Um, I'm sorry. For— for 50 of the games, they— they played exactly equally, right? Um, and on that one game where this one won, it— it played slightly differently. It made like one critical move that like, you know, normally it would have done differently, but due to some exploration or some random noise, it just happened to make a smarter move than it did previously.

Dwarkesh Patel01:27:58

Yeah.

Eric Jang01:27:58

So you have one supervision signal, like one true supervision signal for your policy network. Um, and then you have, uh, 99 games times 300 moves for which, uh, imitating those actions gives you exactly the same policy you had before. And so the, um, the scale of your variance is actually very bad because it's like you only have one label out of this enormous data set of— of actions, of supervision actions where you want actually, sorry, let— let me— let me clarify a little bit.

Dwarkesh Patel01:28:31

Okay. So we were just talking about how the good move, the out-of-distribution move is a small fraction of all the moves that are played across all the games on which you'd wanna train. And, um, this of course reminds me of how LLMs are trained with policy gradient methods. Uh, Karpathy, when he was on the podcast, called it like sucking supervision through a straw. Um, and, uh, and so yeah, it's interesting that this like this thing you're saying, which would be intractable and prevents you from actually getting beyond a certain level in Go, is just by default how LLMs are trained?

Eric Jang01:29:03

Right. So, um, in this case, this is not to say it doesn't work, right? Like if you imagine increasing the number of games to like, you know, millions of samples, you actually can get some meaningful supervision like samples so long as you find a way to sort of mask out the supervision from these guys. And then this is where things start to get pretty related to RL in terms of advantage and baselines and so forth.

Dwarkesh Patel01:29:25

Yeah.

Eric Jang01:29:25

So— so let's, um, let's look at the, you know, the gradient variance of a very naive approach like this where, um, I'm just gonna call it like gradient RL, and it's basically the, you know, sum of rewards.

Dwarkesh Patel01:29:50

All right. I see what you're saying.

Eric Jang01:30:04

So— so the sum of rewards is the return, right? So— so like, uh, in— in our naive setup here, we only have an indicator variable for the return where either you won or lost. So, um, so in the case where you lost, well, you just don't train on your gradient is zero. You don't train on those examples. And when you won, you try to predict those— those things, right?

Dwarkesh Patel01:30:25

Yeah.

Eric Jang01:30:25

So— so you can think about this setup as a— as a special case of this general formula here. Um, the, um, the trouble here is that this is very high variance because when you multiply these terms out, when you take when you try to compute the variance of this, and so— so variance of the gradient is equal to expectation of squared minus and just for simplicity, we can pretend this is like, you know, on average zero or something.

We if— if you're centering it at, at, you know, no signal. Um, and, uh, the variance here basically means that you're, you know, taking the square of this product term. And so you end up with, uh, a term that kind of grows quadratically with the with t. So— so variance, um, when you have a setup like this, this thing acts as a coupling effect on top of— of these terms here. So, um, um, let's actually map this to an LLM case and we can answer like why do LLMs only do one-step RL instead of a multi-step RL scenario?

Um, in LLMs, you have a decoder that might, you know, predict some words like "Hello world." And so in current LLM RL, they treat this entire sequence as a single action, just AT, and big T is just one, right? And so yes, it is true that, you know, the, uh, because of how, you know, transformers are formulated, uh, through the sort of product of conditional probabilities, um, we do have, uh, you know, probability of this sequence is equal to the sort of sum of log— log probability of the whole sequence is equal to the sum of the probabilities of like, you know, uh, individual tokens, right?

Dwarkesh Patel01:32:18

Yeah.

Eric Jang01:32:18

So— so in this case, I would, um, I would say something like, you know, log L plus log LOW plus log world. So, um, this is true. And if this term were one, then they would be the same thing. However, um, in sampling things, if you have a reward term assigned to every specific token, now you have these interaction effects between the cross-multiplication of these terms and these terms.

Dwarkesh Patel01:32:54

Right.

Eric Jang01:32:54

And so the problem becomes how do you ascribe the credit associated with every episode to all these different terms here?

Dwarkesh Patel01:33:02

I guess the thing I'm confused on is what would that even look like to do it that way in, um.

Eric Jang01:33:07

In LLMs.

Dwarkesh Patel01:33:08

In LLMs because you do you only do get a reward at the end of the episode. So.

Eric Jang01:33:13

You could imagine a reward that says like, "I'm gonna give you some process supervision."

Dwarkesh Patel01:33:18

Yeah.

Eric Jang01:33:19

Uh, where you get a reward for each of these actions on every step.

Dwarkesh Patel01:33:23

Okay. So you're saying it instead of doing it that way where you, um, well, I guess the way you've written it, it would be a sum at the end anyways. So it would they wouldn't have to be multiplied, but you're saying instead of doing it that way, you would just add up this process rewards at the end and then treat that as one single reward signal.

Eric Jang01:33:39

Correct. For one single log action.

Dwarkesh Patel01:33:42

But, um, isn't that how it's written to begin with anyways? Like the sum of the, uh, rewards.

Eric Jang01:33:47

Yeah. So— so the— the thing that's a little bit hidden here in the math is that we're assuming that when you decompose the problem to a multi-step problem, that you're now introducing kind of correlations between your actions through the computation of this guy. And, uh, so if you separate these things out, then there will be, um, this— this will magnify the variance of— of this one.

Dwarkesh Patel01:34:07

Right.

Eric Jang01:34:07

So in— in the case where you don't separate it out, if you just have t equals equals one, you just have a single estimate of log prob and a single estimate of reward.

Dwarkesh Patel01:34:16

Right.

Eric Jang01:34:17

Now, um, there are this— this term still shows up in, uh, L so in LLMs, it looks a little bit more like the naive reinforced estimator looks a bit like return of the single action plus, uh, times, you know. It looks kind of like this. This is sort of the very, uh, basic form here, but this is still a contributor to variance. Um, so you want to make sure that like you don't, uh, similar to how in this case we were training on a lot of neutral labels, you want to make sure that you're subtract you're— you're sort of penalizing the labels that don't help and only rewarding the ones that actually make you better.

Dwarkesh Patel01:34:59

Right.

Eric Jang01:34:59

Right? So intuitively, the analogy here is like, can we find a— a term in our training objective such that it's actually kind of discouraged from doing this or, you know, these don't have any effect on the gradient and this has an effect on the gradient?

Dwarkesh Patel01:35:12

Right. I— I guess if you applied that there, the only thing you could do is eliminate 49 of the games. So at least if the way you have it written there, it would be, um, 51 times.

Eric Jang01:35:26

Actually, the optimal case is to pull out discard all of these moves and only get a gradient on that single move that you got better.

Dwarkesh Patel01:35:34

Yeah. How— but how would you do that?

Eric Jang01:35:36

Right. So this is a pretty tricky problem in practice.

Dwarkesh Patel01:35:39

Yeah.

Eric Jang01:35:39

Um, and so this is where advantage estimation happens in reinforcement learning. So you want to subtract, um, you know, a term from— from, um, from your—

Dwarkesh Patel01:35:54

Right.

Eric Jang01:35:55

Your— your multiplier instead of an indicator function of like one and zero, you want something that kind of behaves like a zero for all of these guys.

Dwarkesh Patel01:36:02

Yeah.

Eric Jang01:36:02

And then a one for all these ones.

Dwarkesh Patel01:36:04

Yeah. But you so you could do that if they're, um, if you can say, "Hey, I won this game, so this is slightly above baseline performance."

Eric Jang01:36:12

Well, you won on a lot of games. Uh.

Dwarkesh Patel01:36:14

Exactly.

Eric Jang01:36:15

But— but you don't know which ones let you win because they were truly better.

Dwarkesh Patel01:36:18

Right.

Eric Jang01:36:18

Versus winning on accident.

Dwarkesh Patel01:36:20

So how would you design a baseline where it's truly better?

Eric Jang01:36:22

Yeah. So this is where in RL, people use things like TD learning.

Dwarkesh Patel01:36:26

Yeah.

Eric Jang01:36:26

To better approximate the quality function, the Q that we mentioned earlier. So you can try to subtract that from your, um, your— your— your return.

Dwarkesh Patel01:36:35

I see.

Eric Jang01:36:35

So— so ideally, what you really want to do is in RL, you want to, um, push up the actions that make you better than the average.

Dwarkesh Patel01:36:44

Yeah.

Eric Jang01:36:44

And, um, push down the actions that make you worse than the average.

Dwarkesh Patel01:36:48

Right.

Eric Jang01:36:48

And they call this advantage. There are multiple ways to compute it. I highly recommend John Schulman's, uh, General Advantage Estimation paper as like a good, you know, treatment on how to, um, how to like think about various ways to compute it. But the at the end of the day, you know, um, you— you want to reduce variance by trying to make this smaller and so that it doesn't magnify the variance of this one.

Dwarkesh Patel01:37:07

Right. Makes sense. So but, um, this requires you to have a very good estimate of what average performance from a state would look like.

Eric Jang01:37:15

Yes.

Dwarkesh Patel01:37:15

And this is this gets back it gets us back to the value function thing we were talking about earlier.

Eric Jang01:37:19

Right. And— and so this, uh, keep in mind that in this case, this model-free RL setting, uh, is trying to solve a credit assignment problem where you don't know which actions were actually good and which ones were bad.

Dwarkesh Patel01:37:29

Yeah.

Eric Jang01:37:30

Monte Carlo tree search is doing something very fundamentally different, which is it's not trying to do credit assignment on wins. It's trying to improve the, uh, the label for any given action you took. And so we can actually think about a completely different algorithm called neural fictitious self-play, which was used, uh, to great effect in, uh, systems like Alpha Star and, um, and OpenAI's DOTA model.

Dwarkesh Patel01:37:52

Mm-hmm.

Eric Jang01:37:52

Um, so— so let me talk a little bit about how, um, how you can kind of unify some of these RL ideas in the model-free setting as well as the self-play setting.

Dwarkesh Patel01:38:01

Mm-hmm.

Eric Jang01:38:02

Okay. So what happens if you don't have the ability to easily search a tree, right? Like in Go, it's a perfectly observable game. You can easily construct a pretty deep tree that completely captures the game state. In a game like StarCraft, where you don't have really complete control over the binary, it's— it's a little bit hard to do this. And I'm not even sure if it's a— it's a deterministic game, right? So— so that makes this, uh, kind of difficult from a data structures perspective. So, um, what is done instead is that the basic idea of supervising your actions with a better teacher is still there, right?

So— so if in a given neural fictitious so we're gonna talk a little bit about how neural fictitious self-play works. Same idea. We're gonna like come up with better labels for each of the actions we took, just like in MCTS. But how do we derive the better labels? In MCTS, we, uh, perform search to and assuming we have a good value function, the search will kind of give us a better result than our initial guess.

In, um, in a game where you can't easily simulate a search process, what they do instead is train what is known as a best response policy. So you fix your opponent. So let's say you— you're— you're currently training pi A against, um, a strong opponent pi B. Uh, in StarCraft, maybe like, you know, these are the Zergs and you're playing Pro Toss or something. Um, so you fix your opponent and you treat this as a classic model-free RL algorithm where your goal is just to beat this guy.

And so here you use your standard TD learning style tricks or use PPO or any actually like, um, you know, model-free RL algorithm to try to hill climb against winning this player. And so, um, you train you train, uh, basically you— you have a reward function that's like, you know, return is like, you know, one if wins against pi B. So this is no longer a self-play kind of problem, right? This is just like a fixed opponent and you're just, um, solving trying to maximize, um, a score against— against that and then, you know, zero otherwise.

And so you're you have a sort of fixed environment where all you care about is just beating this guy. And, um, once you have a good policy that you trained with, uh, you know, pick your favorite model-free RL algorithm, PPO or SAC or, you know, any kind of mixture of the or, you know, uh, VMPO or whatever, um, you now have a good policy that gives you a good label for what this one should do when playing against that player. And when you train multiple best response policies, you can basically then distill the RL algorithms into the labels for a given, uh, opponent.

So you might have, let's say, a best response policy against pi B and then maybe you have a collect a league of, you know, um, of opponents like pi B, pi C, pi D, and you're gonna take the best response policy that you train against each of these fixed opponents. And for this one, you're going to, uh, supervise them with the label that this one would provide. So it is kind of like this is almost like a proxy for your MCTS teacher, right? Instead of MCS teacher, you use a model-free RL algorithm to find the best search action that you could do to, uh, to— to kind of beat your opponent.

And then you're finally you're distilling the, um, the policy here into what is known as like a, a mixed strategy where it's trying to basically average across all possible opponents it could play against. And this is what gives you something that can do no bet no worse than like, you know, an averagely average selected opponent from the league. And— and so this gets around the problem of having to derive a teaching signal from MCTS, but it still fundamentally is about relabeling your— your— your states with better actions so that they improve your policy.

Dwarkesh Patel01:42:06

And just make sure I understand this is like if you win against again win again against this other policy, you sort of reinforce all the actions.

Eric Jang01:42:13

Yes.

Dwarkesh Patel01:42:14

On— on that trajectory.

Eric Jang01:42:15

Yeah. So— so here you can use a number of algorithms like PPO.

Dwarkesh Patel01:42:18

Yeah.

Eric Jang01:42:18

VMPO, um, you know, Q learning even if you want like, uh, the specific algorithm here, um, can be, you know, it's usually a model-free thing 'cause you don't have search. But there's an interesting connection from MCTS and Q learning that I— I wanna, you know, bring up. So in MCTS, you do something where you have a tree and through the resolution of your— your— your value function at the at the leaves of the tree or, you know, your approximate leaves of the tree, you can kind of back up through— through the, you know, the— the sequence of many sequences and then obtain some sort of mean value estimate, right?

Like your Q is kind of derived from the average of a bunch of simulations. Um, in model-free algorithms, there is often a component, uh, of estimating a Q value. And so, um, and Q— Q values are often learned through TD learning. Although in PPO, the— the way that they do advantage estimation is not necessarily through a Bellman backup. But, um, but in Q learning, there's this kind of a very cool trick where, um, you do, you know, QSA is backed up as R plus, you know, some discount factor times the max AQ of your next step.

So intuitively, how this works is like if you have a MDP and then this is like, you know, terminal. What this is sort of saying is that like the best action you can take at this, uh, state is equal to the reward you take for, you know, taking this action plus the best that you can do at the next state.

Dwarkesh Patel01:44:03

Yeah.

Eric Jang01:44:04

So there's a sort of recursive and dynamic programming property.

Dwarkesh Patel01:44:06

Yeah.

Eric Jang01:44:06

Of— of MDPs. And you can train neural networks to basically try to enforce this— this con this uh.

Dwarkesh Patel01:44:13

Yeah.

Eric Jang01:44:13

Consistency, right? So you can say like, "Well, once I know the Q value of this action, I can then use that to kind of compute something about the Q value for."

Dwarkesh Patel01:44:20

So when earlier I was like, "Hey, why are we training policy? Why don't we just train the value alone?"

Eric Jang01:44:25

Mm-hmm.

Dwarkesh Patel01:44:25

That— that is what this is.

Eric Jang01:44:27

Um, this is a algorithm for recovering value estimates of intermediate steps when you don't have the ability to do forward search.

Dwarkesh Patel01:44:35

Mm-hmm.

Eric Jang01:44:35

So you must collect a trajectory first of like n steps before you're able to do this trick. Um, but the intuition is kind of the same, which is that like knowing something about the Q value here can tell you something about the Q value here.

Dwarkesh Patel01:44:48

Yeah.

Eric Jang01:44:49

And indeed, you can recover a policy from a Q value.

Dwarkesh Patel01:44:51

Yeah.

Eric Jang01:44:51

Right? So— so the, um, you don't need to explicitly model the policy distribution.

Dwarkesh Patel01:44:55

Yeah.

Eric Jang01:44:56

You can actually recover the policy distribution by doing argmax over your, um, your Q values.

Dwarkesh Patel01:45:00

Right.

Eric Jang01:45:01

Yeah. So— so Q, uh, Q learning or, um, you know, this kind of like, uh, approximate dynamic programming kind of propagates what you know about the future Qs backward like this, right? And you can see that there's a sort of similar structure that goes on here where, uh, in— in this case, you're planning over trajectories your agent hasn't actually been to yet, whereas in this case, you're planning over trajectories your— your agent has visited.

Dwarkesh Patel01:45:23

Yeah.

Eric Jang01:45:23

Um, so— so importantly, why does Q learning, you know, why was Q learning a big deal, right? Like it's because historically, we just haven't had the ability to do search on fairly high-dimensional problems like robotics or whatever. So for a long time, we kind of make the assumption that like, "Okay, well, if we can't model the dynamics with like a world model or something, we're gonna instead just collect trajectories."

Dwarkesh Patel01:45:44

Ah.

Eric Jang01:45:44

And then plan with respect to the only number that really matters, which is reward.

Dwarkesh Patel01:45:48

Okay. So this is very interesting. And then to unify this with our discussion of LLMs. So with LLMs, you're doing something you don't have Q values, but you're doing this sort of backwards learning where, "Hey, let's find the trajectories which pass some unit test in some coding environment and then let's reinforce those trajectories."

Eric Jang01:46:06

Mm-hmm.

Dwarkesh Patel01:46:06

And then there's a huge difference between that and this forward approach with MCTS. And the reason you can do MCTS and it's much more preferable to do MCTS because you can do it per move, uh, and make each move better rather than having to learn per trajectory. Um, and hope, you know, as Karpathy said, hope to learn this like.

Eric Jang01:46:24

Like a straw.

Dwarkesh Patel01:46:24

Yeah. So you get this supervision through a straw. Uh, basically, you just upgrade all the tokens in a trajectory that might or might not have been relevant to getting the answer right. The reason you can do this much more sort of sample efficient, uh, much more favorable thing with Go is that because MCTS works in Go, you basically know that, "Hey, if I just do search locally here and this search is sort of truncated at the end by this value function that, uh, works even— even if I haven't unfolded my whole trajectory, I can just say this is my new policy, um, and I can improve in a more iterative like local way rather than having to ha— having to unfold all these trajectories."

Eric Jang01:47:06

So there was some research, I think, from Google in 2030 2023, 2024 where they did try to apply tree structures to reasoning.

Dwarkesh Patel01:47:15

Yeah.

Eric Jang01:47:15

And I think it's, you know, the jury's still out as to whether this can ever work. So I— I would say like, uh, it we probably will see like, you know, revisiting of this idea of forward search, um, in— in the future. But there's two things that make MCTS very simple for Go, which is that value estimation is kind of concrete and you can determine it.

Dwarkesh Patel01:47:37

Yeah.

Eric Jang01:47:37

For real. And then you can kind of, uh, uh, sort of use it to truncate depth, as you said.

Dwarkesh Patel01:47:42

Yeah.

Eric Jang01:47:42

And then the, uh, breadth is also, uh, determined. And what's kind of critical is that the action selection algorithm where you iteratively visit and grow the tree is, um, well suited for the size of problem that Go is.

Dwarkesh Patel01:47:55

Yeah.

Eric Jang01:47:56

And the depth of the problem. But for something like LLM reasoning, um, you know, Puct might actually not be a good enough heuristic. It might be too greedy with local tokens and it might do something like, "Oh, only give you, you know, uh, sort of obvious thoughts that are correct, but not really solve your final problem."

Dwarkesh Patel01:48:11

Yeah.

Eric Jang01:48:12

So I— I would say the jury is probably still out on how like what the final instantiation of reasoning for LLMs would look like. And I wouldn't rule out that like this stuff could, you know, come back. But it's been hard.

Dwarkesh Patel01:48:21

Don't— don't LLMs sort of natively learn to do MCTS where they'll try an approach and be like, "Oh, that doesn't work. Let's back up. Let's try this other thing."

Eric Jang01:48:31

Yeah.

Dwarkesh Patel01:48:31

And then go in the direction that proves to be more fruitful?

Eric Jang01:48:34

Uh, yeah. Cer— certainly. I think that LLMs manage to do something that looks like real human reasoning without having to do an explicit tree structure.

Dwarkesh Patel01:48:41

Yeah.

Eric Jang01:48:41

Um, that being said, I think the idea of doing forward search and simulation to get a better sense of what is valuable might make a comeback, um, even though not exactly in the same instantiation as, as, uh, AlphaGo.

Dwarkesh Patel01:48:54

Interesting. But, uh, just to make sure I understand the crux of it, like the— the breadth from the number of legal actions being wider and the depth from being able to not being able to train a value function as easily because.

Eric Jang01:49:09

So here's an example where LLMs break down. The CPUCT rule involves, you know, square root of n.

Dwarkesh Patel01:49:13

Yeah.

Eric Jang01:49:13

Over 1 plus nA. In an LLM, like you're most likely never gonna sample the same, uh.

Dwarkesh Patel01:49:20

Yeah.

Eric Jang01:49:20

Child more than once, right? So if you have, let's say, multi-steps of thinking.

Dwarkesh Patel01:49:23

Yeah.

Eric Jang01:49:24

Um, because language is so broad and open-ended, it's, uh, a sort of, uh, discrete set of actions is not really an appropriate choice for an LLM.

Dwarkesh Patel01:49:33

Yeah.

Eric Jang01:49:33

Uh, even though they're discrete tokens.

Dwarkesh Patel01:49:34

Yeah.

Eric Jang01:49:35

Um, it's just such a large number that this type of exploration heuristic is probably not the right thing to do to guide how to search down a tree.

Dwarkesh Patel01:49:42

Right. But I— I guess the crux comes down to the fact that in Go, you know that the MCTS is almost certainly better than your current policy, even though you haven't gotten even though you haven't explored the end of any trajectory.

Eric Jang01:49:55

Correct.

Dwarkesh Patel01:49:55

And then in, uh, in normal reasoning for LLMs or robotics, there's no way to just locally evaluate and improve your next move in a way that doesn't result in in a way that's independent of actually like solving the problem.

Eric Jang01:50:10

Uh, no way is a strong word. I think lots of people have thought about how to try to apply MCTS or its kind of successors like MuZero to continuous control spaces.

Dwarkesh Patel01:50:18

Yeah.

Eric Jang01:50:18

And I'm sure, you know, very cool research work is still ongoing to try to crack that problem. Um, but yes, the— the seeming challenge right now is that like most problems in much higher, uh, dimensional, um, you know, action spaces or something that's combinatorially much bigger like language, they— they don't seem as amenable to the kind of discrete action selection heuristics as well as, uh, kind of game evaluation type stuff that, uh, Go does. Um, but that's not to say the idea of like, you know, thinking into the future along multiple parallel tracks might not give you some information about like which way to search, right?

Like if you think about mathematics, I think mathematics often occupies a— a little bit more of like a logical search kind of procedure.

Dwarkesh Patel01:51:00

Yeah.

Eric Jang01:51:00

Where you kind of could back up, you could kind of see like which paths seem good or not. There's more of a rigid structure there, whereas maybe like in a, uh, you know, business negotiation or something, um, it's less of a tree and maybe, you know, something a bit different.

Dwarkesh Patel01:51:15

Okay. So we're now seated so I can ask you some more questions about AlphaGo and about AI research more generally. Um, in 2021, Andy Jones had a paper called Scaling Scaling Loss for Board Games. And, um, he basically anticipated inference compute or inference scaling by showing that you can trade off test time compute and, uh, training compute. That is to say that you can spend more compute on the forward the searching through the MCTS. Um, and if you do that, you can get the equivalent performance as having spent more time training the model.

Um, and so if you you know, you if you see this pattern, you might think, "Okay, well, with LLMs, you might do something like that in the future." And in fact, that's what ended up happening. Okay. So what is a kind of fun exploration one could do now to explore other axes of scaling in toy settings, which will be important to understanding what AI development might be like in a few years?

Eric Jang01:52:06

Sure. Yeah. Um, I think that indeed test time scaling and, uh, reasoning and how it interacts with model size are quite profound when it comes to like how much, uh, needs to be actually done as explicit search versus how much can be packed into the forward pass of a neural network.

Dwarkesh Patel01:52:25

Mm-hmm.

Eric Jang01:52:25

Right? And— and, um, how does a forward pass of a neural network sort of learn how to do something that should be a sort of sequential and.

Dwarkesh Patel01:52:31

Mm-hmm.

Eric Jang01:52:32

You know, recursive step? That's quite interesting.

Dwarkesh Patel01:52:34

Yeah.

Eric Jang01:52:34

Um, so the yeah, the Andy Jones Scaling Loss for Board Games paper is quite cool. There's another really nice result from that paper where they where he showed that, um, not only can you predict scaling loss of like, you know, the sort of LLM variety where, um, as you increase parameters, you can decrease the amount of compute for search or vice versa. Um, he also showed that you can actually predict, uh, how much compute is needed to solve a, uh, larger version of the board game, uh, for example.

Dwarkesh Patel01:53:00

Mm-hmm.

Eric Jang01:53:00

And— and so with Go, you know, which can scale from, you know, uh, 3x3 to an infinitely sized, you know, uh, Go board, you might actually be able to sort of revisit this question and try to reproduce, uh, whether this shows up. Um, you know, I actually started this project with this sort of a motivation that like does the bitter lesson or does our knowledge of scaling laws allow us to kind of execute a lot better on a sort of compute optimal Go bot?

Dwarkesh Patel01:53:21

Mm-hmm.

Eric Jang01:53:22

And can we can we kind of build a strong Go bot without all of the KataGo tricks, right? Just— just by really focusing on the bitter lesson and the scaling laws.

Dwarkesh Patel01:53:28

Hmm.

Eric Jang01:53:28

Um, I have not been successful so far, but I think it's— it's sort of a— a fact that like usually when you want scaling loss to work, you want to be in the regime where, um, the— the recipe already works and the data sets are good rather than trying to kind of figure out how to do scaling while also trying to figure out what the— the right data set are.

Dwarkesh Patel01:53:45

Okay.

Eric Jang01:53:45

So— so this is like the scientific understanding component in research often follows a step where you get something to work first.

Dwarkesh Patel01:53:51

Mm-hmm.

Eric Jang01:53:51

And then you use that, um, system to collect data that then helps you build a mental model of how things work, such as scaling laws, right? And— and so usually actually, if you wanna build a strong Go bot using scaling laws, you— you actually have to make a strong Go bot first and then use the scaling laws to kind of extrapolate a bit farther into the future.

Dwarkesh Patel01:54:08

Say more. So just so I understand, first of all, you're saying scaling laws did not work or you could not there was no scaling loss pattern that you could see in your Go bot?

Eric Jang01:54:17

Yeah. So, um, a— a mistake I made initially when I had some bugs around how MCTS labeling was working was I would, um, I would collect a bunch of data with an expert policy and then treat it as a supervised learning problem and try to identify scaling laws with, uh, expert data sets. Um, you can indeed plot things that look kind of like this, but if you're in a regime where, you know, your policy's not working well, you might be just studying scaling laws on like bad data, right? So— so just like one important implementation detail is that if you wanna study a scaling laws problem, you kind of have to have a problem for which like the data is good, the architecture is good, and there's no bugs, and then like you— you— you solve it there.

Um, X anti I wasn't able to like apply scaling laws to direct what to look on look— look at, um, until, you know, I had the rest of the system working. And— and this sounds obvious like to— to researchers, of course, you want to have like a working bug-free system before you study scaling. But, uh, just as a sort of advice for practitioners on like where I actually tripped up when I started this project was you don't necessarily want to kind of jump into the science of studying your manmade artifact before your manmade artifact is like interesting enough to be studied.

Dwarkesh Patel01:55:21

Speaking of compute, so if you you can look at these charts of compute used to train the best AI model in the world over time going back 10 years, and it's a very smooth line in log space, uh, that is exponentially growing year over year. Uh, except there's this huge aberration, and that aberration is AlphaGo Zero, uh, which is trained on way more compute than any other AI model at the time. It was like, uh, three E23 flops. It's sort of comparable to like a frontier LLM.

I mean, orders of magnitude off, but still. Um, and so yeah, the question is, especially with you being able to get something off and did you train it on your own?

Eric Jang01:56:01

Uh, I got a donation from Prime Intellect.

Dwarkesh Patel01:56:03

Okay.

Eric Jang01:56:03

For like about 10K. And then I spent, um, I spent maybe the first 4K doing, um, kind of exploratory research.

Dwarkesh Patel01:56:10

Yeah.

Eric Jang01:56:11

And then, uh, about 3K on the kind of final run.

Dwarkesh Patel01:56:13

Yeah.

Eric Jang01:56:14

Um, and then, uh, some— some of it remaining for serving the model for.

Dwarkesh Patel01:56:18

Cool. Yeah. Is there a sense that they were just did a bad job training it if you can do it in 10K now?

Eric Jang01:56:23

Uh, the compute required to be the first to do something is always like much larger than the compute it takes to catch up. And it's the same story playing out in LLMs, right? Like once someone else has done it, um, you could use tricks like distillation, you could use, um, uh, all sorts of like kind of, uh, crutches to kind of bootstrap your way to success. So with my own bot that I'm I've hosted online, um, I actually used, uh, sort of best response training against the KataGo models to kind of get a strong level performance. And, um, you know, as— as the time of recording, I'm— I'm validating whether this can be uh, I can kind of do that first step, which is to do the tabular ROSA, right?

Dwarkesh Patel01:56:56

Yeah.

Eric Jang01:56:56

Um, but importantly for research, you often wanna start from a good init, right? So— so the kind of simple thing I did first was train best response agents against KataGo.

Dwarkesh Patel01:57:03

Yeah.

Eric Jang01:57:03

Um, AlphaZero team, uh, they did not have any policy that they could train against, right? 'Cause they were trying to do everything tabular ROSA. So, um, and being the first to do it means that you're prioritizing getting the thing working rather than like, let's say, the most compute efficient, uh, possible implementation. So, um, this actually plays out in robotics as well. Like if you look at the kind of frontier of large models trained for robotics, um, there the scatter plot is all over the place and there isn't a very clean line the way that there is for frontier LLMs. And— and that is because the folks training these models often are not, you know, at the scale where every flop counts and they need to like kind of squeeze out, you know, the performance of every single flop as the dominating decision deciding factor in pretraining, right?

Instead, their focus is more like we want a certain capability to show up, so we optimize the, uh, training setup to kind of make it easy to derive that capability. And once you have that capability, well, um, invariably if you scale up the compute, you are forced to kind of make it compute efficient because this is like hundreds of millions of dollars we're talking about. But, um, but in the past when— when compute for experiments was kind of more plentiful or, you know, not— not, uh, not, um, uh, accounted in a way that the researcher was really responsible for, then you kind of end up with people optimizing for things besides kind of being on the compute optimal preto frontier.

Dwarkesh Patel01:58:15

I see. Like speed or something.

Eric Jang01:58:17

Yeah. Like time to result or just getting it to work.

Dwarkesh Patel01:58:18

Okay.

Eric Jang01:58:18

I think the first AlphaGo, like probably they had lots of compute and they didn't need to be um, they didn't need to worry too much about making it the most compute optimal thing.

Dwarkesh Patel01:58:26

Yeah. And how much of the improvements in compute efficiency are methods that did not exist as of 2017 versus things which you they could have done in 2017, but, um.

Eric Jang01:58:35

Yeah. Great question. So— so going into this project, I kind of knew in the back of my mind that like things always get easier to do over time, and I wanted to see like where— where is Go at given that like it didn't seem like there has been any major open source, you know.

Dwarkesh Patel01:58:48

Right.

Eric Jang01:58:48

Uh, strong bot after KataGo in 2020. Uh, and then, you know, reading the KataGo paper, there's a lot of clever ideas. I was kind of wondering like, okay, uh, let's look let's see if the bitter lesson has happened where like a lot of these kind of tricks just sort of go away because the NVIDIA made faster GPUs, right? And— and so, uh, roughly where are we on that? So, um, again, this is not a peer-reviewed claim, so this is just my preliminary, um, you know, vibe guess on like what I've seen based on my own experiments. But it seems like, um, you know, architecture choices don't matter that much. You know, Transformer versus ResNet, we're— we're at the sort of speed of GPU where the size of the model's not so big that this really matters.

Um, um, you can actually simplify the setup quite a lot. So instead of doing a distributed asynchronous RL setup with replay buffers and pushers and collectors, you can kind of do a— a dumb synchronous thing where you like collect, you just train a supervised learning model, and then you collect again.

Dwarkesh Patel01:59:41

Mm-hmm.

Eric Jang01:59:41

And— and so there's like opportunities to simplify infrastructure. Um, NVIDIA GPUs have indeed got faster. So whereas KataGo was trained on V100s, you can train on like half the number of, you know, desktop Blackwell GPUs and it still works. Um, and, um, some of the kind of auxiliary supervision objectives that KataGo developed aren't really necessary if you have a strong initialization.

Dwarkesh Patel02:00:04

Mm-hmm.

Eric Jang02:00:04

Right? So— so if you're initializing against, you know, best, uh, response training against KataGo itself, then your own model actually needs none of the tricks that KataGo needs.

Dwarkesh Patel02:00:12

Yeah. Yeah.

Eric Jang02:00:12

So— so then the core thing is like how can you get as quickly as possible to some strong opponents? And, um, that matters a lot more than the specific architectural innovations. But there are still some nice compute multipliers. So I found that training on 9x9 boards was very nice for resolving endgame value functions. And then like if you can co-train that on a architecture that can transfer between 9x9 and 19x19, then you can really cut down the warm start time to learn that from scratch. I think, uh, AlphaGo Zero, their plot was, um, first 30 hours or so are spent basically catching up to the supervised learning baseline.

Um, and you can cut down that time a lot by kind of pretraining on a small board and then— and then like, you know, warm starting that into your, you know, 19x19 board play. Uh, there was some other stuff like, you know, varying the number of sims between episodes. This turns out to be not that sensitive actually. Like you can kind of, you know, fix it or increase it. Doesn't matter too much. Um, but so anyway, it's kind of just nice from a scientific perspective just revisiting like an old paper and seeing like what really matters.

Dwarkesh Patel02:01:11

Yeah. Wait, this is sort of a tangential question, but why is it okay to have a buffer in AlphaGo? 'Cause every time I talk to any researcher, they're telling me about how bad it is to be off policy.

Eric Jang02:01:18

Mm-hmm.

Dwarkesh Patel02:01:19

But then the way a naive implementation of AlphaGo Zero would work is that most of the moves in a given backward step or in a batch of backward steps, um, would be not— not among the ones that were made by the most recently trained model. So why is that okay?

Eric Jang02:01:38

Great question. Yeah. And this— this gets into the sort of fundamental off policy versus on policy, uh, reinforcement learning kind of questions. So, uh, as you recall in MCTS, you take actions that you took and you relabel them to, uh, to take different actions on the same states, right? So— so the off policy part here comes where, um, what if you're relabeling states that your new policy would never visit, right? Like what's the point? You're kind of wasting capacity. And in the extreme limit, imagine your distribution of states in your training buffer are all states that you would never visit.

Then you're basically supervising them, uh, to, uh, take good actions on states you would never achieve.

Dwarkesh Patel02:02:16

Yeah.

Eric Jang02:02:17

And therefore your policy can get really bad, right? So this is where off policy can really hurt, um, uh, uh, AlphaGo. Um, however, if you interpret this sort of from like the dagger perspective, which is basically saying like a way to kind of correct yourself back to the optimal trajectory, uh, given some— some data, what— what you kind of want in an algorithm like this is to have mostly states that you would visit, but then you have a small percentage or maybe a reasonable percentage of states in this kind of high-dimensional tube around your optimal.

Dwarkesh Patel02:02:46

Mm-hmm.

Eric Jang02:02:46

You know, trajectories. And any of those states are given a supervision target to kind of, uh, sort of funnel you back into your optimal trajectory. Uh, so maybe I can just draw that quickly here.

Dwarkesh Patel02:02:57

Yeah. Great.

Eric Jang02:02:57

So in, um, sort of a dagger style setup, what your kind of optimal training data distribution is, is that here is your optimal states and actions. So this is like, you know, you wanna be in this state, you wanna be in this state, you wanna be in this state, and then you win here. Um, and then these are your optimal policy actions. So these are the— these are the things that you definitely wanna train on. But to make it robust to disturbances, um, you want to make sure that if you happen to drift off into some other states.

Dwarkesh Patel02:03:26

Mm-hmm.

Eric Jang02:03:27

You can kind of funnel yourself back into.

Dwarkesh Patel02:03:29

But why isn't this a fully general argument for off policy training?

Eric Jang02:03:32

This is actually why you wanna do off policy training sometimes.

Dwarkesh Patel02:03:35

Mm-hmm.

Eric Jang02:03:35

Is that like you— you don't want to have a compounding error where if you make a mistake, you don't have the data of how to return back to your optimal distribution.

Dwarkesh Patel02:03:43

Yeah.

Eric Jang02:03:43

And so, um, optimal control does not really say too much about like, uh, you know, how to, uh, you know, not accidentally get here because it's sort of making the assumption that like.

Dwarkesh Patel02:03:54

Yeah.

Eric Jang02:03:54

Once you learn the policy, you're gonna get here.

Dwarkesh Patel02:03:55

Yeah.

Eric Jang02:03:56

But in applications like robotics, right? Like— like, I don't know, uh, a gust of wind blows you slightly off and then now you need to like correct.

Dwarkesh Patel02:04:02

Yeah.

Eric Jang02:04:02

Right? Um, or the friction on one of your tires is kind of a little bit like lower than the other wheel, and then now— now your— your car's drifting and you gotta— gotta like correct it. So— so these kind of things in— in like more real environments often happen where like, um, actually there's a funny, uh, quote about chess and also Go. It's like the problem with, uh, the— the problem with Go and chess is that the other player is always trying to do some shit.

Dwarkesh Patel02:04:23

Uh-huh.

Eric Jang02:04:23

Right? Like, uh, so— so like, you know, things can kind of drift off.

Dwarkesh Patel02:04:26

Yeah.

Eric Jang02:04:26

And you always wanna be able to correct, uh, back to your— back to your winning condition. So— so your replay buffer really should have like your, you know, the states that your policy would visit plus some distribution of states that you might drift to, and then how to return back to your optimal states.

Dwarkesh Patel02:04:42

Yeah.

Eric Jang02:04:43

Now, if you take this to the extreme and you say like, well, let's, uh, we don't have any of this data, and we're gonna just like be labeling with MCTS, um, you know, states that are so far away from our optimal behavior, like this— this bag of states over here, well, like now, yeah, I mean, like each of them gets, uh, MCTS label.

Dwarkesh Patel02:05:10

Mm-hmm.

Eric Jang02:05:10

And your policy learns how to do take sort of the best possible action here, but you never get here. So like you're training your model on states you would never reach.

Dwarkesh Patel02:05:17

Yeah.

Eric Jang02:05:17

Uh, like— like this is— this is not there. So then this is a problem, right? And— and this is where off policy can really hurt.

Dwarkesh Patel02:05:23

Yeah.

Eric Jang02:05:24

Um, so actually, uh, as— as part of this project, I did try an experiment where I took a bunch of trajectories and to try to saturate the GPU as much as possible, what I did was I took, uh, you know, random states from the dataset and, uh, reran MCTS on just those states, right? So instead of playing a whole game where I'm doing MCTS on every move, I just ignore the sort of causality of moves and just pick random board states, and I just label those with my current network. And, uh, and I might revisit old states that I've labeled before and relabel them again with my current network, right?

Dwarkesh Patel02:05:54

Yeah.

Eric Jang02:05:55

And so in practice, this actually does work. You can actually say like, let's take some states that are reasonable and constantly be relabeling them, um, in, uh, in, um, while we're training. And— and so this actually starts to converge on a very robotics-like setup, which is very common, which is you have your— your dataset of trajectories, um, and then you have something like a replay buffer pusher. And— and these are off policy offline trajectories, right? So your replay buffer pusher pushes transition tuples to— to— to the replay buffer.

And then you have some job that's kind of continuously, um, replanning what the best action you should have done instead of taking this action as, right? And so in robotics, it's actually very common to use a— a, uh, that sort of minimized TD error. So like your Bellman updater, um, constantly is pulling things from here and trying to satisfy, you know, the QSA.

So— so, um, and then— and then from here you have your trainer, which is trying to fit the S to A or— or— or, uh, um, fit the, you know, Q to the Q target. So— so here you can think about this as a sort of planner, right? You visit old, um, states that you've been to, and you take your current model and you rethink like what could I have done better if I visited this? And, um, and so this is actually how like kind of off policy robotic learning systems are usually trained. Um, these days there's a sort of simpler recipe, but— but like, you know, in the Google QtOp days, we kind of did— did things like this.

Dwarkesh Patel02:07:47

So what is the trainer?

Eric Jang02:07:48

Oh, yeah. The trainer is, uh, you try to— you try to minimize, uh, QSA and Q target.

Dwarkesh Patel02:07:57

Wait, so can you explain the whole setup again? Like at the high level?

Eric Jang02:08:00

Yep. So you have your off policy data that came from various policies.

Dwarkesh Patel02:08:04

Yeah.

Eric Jang02:08:04

You're constantly pushing, uh, transitions that you saw before to a replay buffer.

Dwarkesh Patel02:08:08

Yeah.

Eric Jang02:08:08

And then you've got this thing called a— a Bellman updater, which basically replans instead of this action, what action should I have taken at S to have a better, you know, value? And— and the way you enforce that is you try to minimize the TD error. So— so actually you— you given this, you have S prime, right? You— you compute Q of S prime, and you find the action that should go with S prime that makes this Q value as high as possible.

Dwarkesh Patel02:08:32

Mm-hmm.

Eric Jang02:08:33

And then you add that to the reward here, and that gives you your actual target, right? So for this current S and A, your Q target is this. So now you have a— now— now you send back the Q target to— to this— this transition. So— so with this tuple, you pair with that a, uh, a Q target. And then here on the trainer, you simply just use supervised learning and you minimize your current network's QSA with its target.

Dwarkesh Patel02:09:00

Got it. Okay. So in the— in the background, you're just like, hey, let me— let me basically think through how valuable were all these actions actually.

Eric Jang02:09:07

Yeah. Uh, in a— in a more optimal policy where you're trying to maximize this, what is the Q target of this transition?

Dwarkesh Patel02:09:13

It's sort of like basically daydreaming.

Eric Jang02:09:14

Exactly. Yeah. You can think about it's like you're kind of going back in hindsight and being like, hmm, like, like given what I've seen in the historical buffer, um, like was there a better action I could have taken?

Dwarkesh Patel02:09:25

Yeah.

Eric Jang02:09:25

Now, the connection to Go here that I— I— I tried and it was, you know, moderately successful, but too complex to kind of like open source was, um, you replace this with like a MCTS relabeler where, um, instead of doing this kind of target network computation, you, uh, run MCTS on your transition, right? So in— in this case, you have, uh, your state, your action, and then whether you want or not at the game. Um, and actually you can just toss these two. You don't— you don't care about these ones. You just take your state and you just plan MCTS.

Dwarkesh Patel02:09:58

Mm-hmm.

Eric Jang02:09:58

To get your best policy, you know, pi, uh, on your current network, right? Not— not the network that took this action, but your current best policy network. You just rerun your search offline on these transitions. And, um, if these are transitions that your policy can get to, then this actually acts as a very nice stabilizing effect. And also the one other benefit is that you can like kind of fully saturate your GPU better because.

Dwarkesh Patel02:10:25

Yeah.

Eric Jang02:10:26

You're not like blocking on the Go game to kind of like give you board states. You just simply search across all board states at any depth in parallel.

Dwarkesh Patel02:10:34

Yeah.

Eric Jang02:10:35

So, um, and then here the trainer would be just, you know, predict the MCTS label as possible.

Dwarkesh Patel02:10:39

Yeah.

Eric Jang02:10:39

So— so again, like this kind of works and this is quite relevant in robotics where you're really you just have a lot of offline data and you can't simulate things like MCTS. But, um, in practice, like it does run into the problem where, you know, like if the current model is looking at states that it would never reach, then it's kind of wasting capacity.

Dwarkesh Patel02:10:57

Yeah.

Eric Jang02:10:57

And so you— you have to be a little bit careful here. So, um, the on policy thing and also much of RL has kind of converged to a much more on policy setup where they don't really try to like directly train on off policy data. At best, they use off policy data as a way to reduce variance, but not directly influence the objective.

Dwarkesh Patel02:11:13

Mm-hmm. I'm sorry, why have to converge to that?

Eric Jang02:11:15

It's just more stable.

Dwarkesh Patel02:11:16

Okay.

Eric Jang02:11:17

Yeah.

Dwarkesh Patel02:11:17

Yeah.

Eric Jang02:11:17

So— so like you— you might use the off policy Q as a way to do like, you know, advantage computation, um, like, uh, you know, Q minus like sum of Q.

Dwarkesh Patel02:11:28

Yeah.

Eric Jang02:11:28

That's kind of like your— your or— or sorry, like, you know, sum of, uh, like if there's n actions and then yeah. So— so like, uh, so— so like this is your value and then this is your— your kind of current Q value. So your advantage for that action is like the average value minus your current one.

Dwarkesh Patel02:11:43

Yeah.

Eric Jang02:11:43

So— so like people can try to estimate Q in an off policy way and then like just use advantage here and then— and then the— the sort of if there's a problem in these dynamics, the it doesn't like blow up your loss as much.

Dwarkesh Patel02:11:55

Yeah.

Eric Jang02:11:55

Um, and so in robotics, there's a kind of convergence towards more like a using off policy data to just shape your rewards, but not actually be directly here.

Dwarkesh Patel02:12:03

I— I'm reminded now of our earlier conversation of why MCTS is so favorable as compared to the kind of, you know, reinforce or policy gradient kind of thing LLMs do. And this might be totally wrong, but I wrote a blog post a few months ago about, um, how RL, at least policy gradient RL, is even more, uh, inefficient than you might think. And so the inefficiency one thinks about naively is the fact that you have to roll out a whole trajectory in order to get any learning signal at all. And so as these trajectories become longer and longer, as an agent has to instead of just previously like complete the next word in the sentence, it has to go instead to, hey, do two days' worth of work to figure out even if you even did this project correctly, the amount of information per, uh, flop has been decreasing.

As you had to unroll two days' worth of thinking in order to see if you even did something correctly to like did I implement this feature, the amount of samples per flop has been decreasing. But so you can think of, um, uh, you're trying to maximize as you're learning bits per flop, right?

Eric Jang02:13:05

Mm-hmm.

Dwarkesh Patel02:13:06

Um, and this is you can think of bits per flop as, um, samples per flop times, um, uh, bits per sample. And what I just mentioned, uh, a second ago is that the samples per flop go down as RL becomes more and more long horizon.

Eric Jang02:13:33

Mm-hmm.

Dwarkesh Patel02:13:33

But, um, at least this kind of naive RL is also terrible from a bits per sample perspective. And here's what I mean, at least compared to supervised learning. So early on in training, let's say you have a, uh, vocabulary size for an LLM that is 100K long. So there's 100K possible, you know, tokens that one could answer. And you have a totally untrained model and you have a prompt like, "The sky is." Um, with supervised learning, you what— what would happen is that the model would have some probability distribution over all the things it could say.

Um, there's a label that says actually the term here is blue, and it would figure it would learn basically through cross-entropy loss exactly how far its distribution is from correctly saying blue. Now, if you were doing this through RL, um, you would say the model would try, "The sky is halicon." Nope, that's wrong. "The sky is told." Nope, that's wrong. This is a totally untrained model, right? And so you would have to do this on the order of 100,000 times in order to just stumble on blue, then get some learning signal off of that.

So if you're in the supervised learning regime and you just get you have your distribution of probabilities, you get told, uh, that it's blue and you figure out how far off you are, the amount you learn is, um, is a function of your pass rate. So like the further away you are from blue, the more you've learned to go towards blue, uh, using cross-entropy loss. And so you can think of it as like your pass rate, your like prior probability of having said blue. And, um, as a function of that, like in supervised learning, uh, through cross-entropy loss, you would— you would learn negative log P, P being pass rate.

Eric Jang02:15:16

Mm-hmm.

Dwarkesh Patel02:15:17

Uh, bits once you get this label. Whereas in RL, if you're just randomly guessing shit and seeing if it works or not, that's, um, that's just basically going to be the entropy of a binary random variable, which is.

Eric Jang02:15:37

And what's also tough here is that actually the distribution that you're sampling under is your policy's distribution.

Dwarkesh Patel02:15:43

Mm-hmm.

Eric Jang02:15:44

Right? So— so it's like if your policy has no chance of sampling blue, then you will never get a signal.

Dwarkesh Patel02:15:49

Exactly. Right. Right. So that's— that's being modeled by the fact that your probability of sampling blue is extremely low. If you do sample it, you do learn as much as you would have learned in a supervised learning. In all other cases, like, you know, 99.99% of in an untrained model, you're, um, you're just learning incredibly little from like seeing halicon is not the correct word or told is not the correct word.

Eric Jang02:16:11

Yep.

Dwarkesh Patel02:16:11

Um, and that's what happens most of the time. So you're just like, um, learn very little. So if you try to graph, um, if you put on the X-axis your pass rate, um, and, uh, here you put the like sort of the bits you bits you're learning from a sample.

Eric Jang02:16:30

Mm-hmm.

Dwarkesh Patel02:16:31

If you have like 0% here, 50% here, and 100% here, so the end of training you're here. Um, if you have, um, supervised learning, negative log pass rate would look something like this. And then the, uh, entropy binary random variable would look like this. Um, and this is, uh, depending on whether you're doing NATs or bits.

Eric Jang02:17:00

Mm-hmm.

Dwarkesh Patel02:17:01

Yeah. If you do bits, it's like one, right?

Eric Jang02:17:02

Mm-hmm.

Dwarkesh Patel02:17:03

Here at the at the peak. Um, this is like a coin flip. You learn the most from a coin flip.

Eric Jang02:17:07

Mm-hmm.

Dwarkesh Patel02:17:08

Uh, this is supervised learning. This is RL. However, the problem is you spend most of training in this regime, right? Like in the in the low pass rate regime. And, um, in fact, if how fast you're learning is a function of how many bits per sample you're getting, uh, and you're getting very little signal here, if you chart the pass rate on a log scale, so you put the X-axis on a log scale where like at the beginning of training with a vocab size of 100K, the pass rate is 100 1/100,000, then 1/10,000, 1/1,000, uh, 1/100, and then, um, okay, what this graph looks like here where supervised learning would look like this.

And— and then RL, if you just basically crunch what I just showed there, it would look like that.

Eric Jang02:18:11

Yeah. And arguably you spend all your time here.

Dwarkesh Patel02:18:15

Exactly.

Eric Jang02:18:15

Potentially never even getting a single success, right?

Dwarkesh Patel02:18:17

Exactly.

Eric Jang02:18:17

Like, uh, so— so it's— it's a sort of depressing plot in the sense that like once you're here, it's not at all obvious how you get to here.

Dwarkesh Patel02:18:24

Yeah.

Eric Jang02:18:24

Um, you know, once you're here, you have something, but like.

Dwarkesh Patel02:18:26

Yeah.

Eric Jang02:18:27

You actually in many RL problems spend all the time here.

Dwarkesh Patel02:18:30

Yeah.

Eric Jang02:18:30

Uh, so— so that there's a sort of question of like how do you initialize so you're at least not at zero, but like at a non-zero pass rate.

Dwarkesh Patel02:18:36

Yeah.

Eric Jang02:18:37

Um, one more thing I'd like to add about bits per sample that's very relevant to, um, uh, you know, you know, any kind of machine learning problem is that, um, and— and there's a connection to soft targets and distillation where if you have access to the logits, right, not just the one hot like this— this is a sort of one hot, uh, token answer.

Dwarkesh Patel02:18:55

Yeah.

Eric Jang02:18:55

Um, if you have access to the soft targets, um, the entropy of this distribution is far, far higher.

Dwarkesh Patel02:19:02

Mm-hmm.

Eric Jang02:19:02

Than— than the one hot. So there's actually way more, uh, there's way more information in bit and bits per sample, um, in a soft label.

Dwarkesh Patel02:19:10

Yeah.

Eric Jang02:19:10

So that's why distillation is so effective.

Dwarkesh Patel02:19:12

Yeah.

Eric Jang02:19:13

Uh, per sample is that it's actually giving you way more information per sample.

Dwarkesh Patel02:19:16

Ah, yeah. Well, I wonder what the equation would be, but obviously it's.

Eric Jang02:19:19

It would just be the entropy of this distribution.

Dwarkesh Patel02:19:20

Right.

Eric Jang02:19:21

So the entropy of this is zero.

Dwarkesh Patel02:19:22

Yeah.

Eric Jang02:19:23

Um, the entropy of this is like, you know, the entropy equation. And— and this is also why like, you know, AlphaGo is quite beautiful. In AlphaGo, you don't train the policy network to imitate the MCTS action. You train it to imitate the MCTS distribution.

Dwarkesh Patel02:19:37

Yeah. Interesting.

Eric Jang02:19:37

But both of these are actually valid. And if you wanted to do a scientific experiment of like how important are this kind of soft label.

Dwarkesh Patel02:19:43

Yeah.

Eric Jang02:19:44

Dark knowledge distillation, you can run an experiment where you— you, uh, retrain the policy network on the action MCTS selected rather than the soft target.

Dwarkesh Patel02:19:51

Interesting. Earlier I was sort of stumbling around. This intuitively, why is this ability to do, um, iterative search where you don't necessarily need to be able to win the game in the beginning? You just need to be able to improve your current policy. Why is that so powerful a capability in learning as compared to how LLMs currently run learn RL? And, um, and yeah, it's exactly this thing of, uh, this is considering your pass rate of the entire trajectory.

I actually don't know a formal way to think about this. Maybe you should help me out here.

Eric Jang02:20:26

Why is AlphaGo an elegant RL algorithm?

Dwarkesh Patel02:20:28

Yeah.

Eric Jang02:20:28

Like, so, um, uh, the major reason is that you never have to initialize at a 0% success rate and solve the exploration problem of how to get a non-zero success rate.

Dwarkesh Patel02:20:39

Mm-hmm.

Eric Jang02:20:39

And— and this is what allows you to hill climb this beautiful supervised learning signal.

Dwarkesh Patel02:20:43

Mm-hmm.

Eric Jang02:20:43

Where and if you look at the actual implementation of AlphaGo, um, every step of the way, there's no, uh, there's actually no, um, you know, TD error learning or dynamic programming, uh, at least explicitly. Um, it's just supervised learning on a value classification as well as a policy, uh, you know, KL minimization. So it's just a super supervised learning problem on improved labels.

Dwarkesh Patel02:21:06

Mm-hmm.

Eric Jang02:21:06

And so the training is very stable, right? You can train like as big of a network as you want. You can kind of retrain this on the data set. Everything will just go stably. The infrastructure is very simple to implement as well. Um, you don't need a complex distributed system to kind of keep everything on policy.

Dwarkesh Patel02:21:19

Yeah.

Eric Jang02:21:20

Um, at the end of the day, you're just saying like, "I have some improved labels. Let's retrain my supervised model on these targets."

Dwarkesh Patel02:21:27

Yeah. Yeah.

Eric Jang02:21:27

And— and so you're— you're always in this beautiful regime where you're just trying to improve the policy rather than, uh, escape this kind of like, uh, sort of local minima where every— every signal's flat all around you.

Dwarkesh Patel02:21:37

Yeah.

Eric Jang02:21:38

Um, so— so one way to draw the curve is like if you draw the sort of win rate of an MCTS policy versus the raw network, um, let's say this dotted line is the raw network, the MCTS policy kind of looks like— like this. And so every step of the way, this supervision signal is very clean.

Dwarkesh Patel02:21:55

Right.

Eric Jang02:21:56

You're never in a situation where, you know, the MCTS is kind of like giving you no signal.

Dwarkesh Patel02:22:01

Yeah.

Eric Jang02:22:01

Unless your MCTS distribution converges to exactly what your policy network predicts.

Dwarkesh Patel02:22:06

Yeah. Yeah, yeah, yeah. Okay. That— that's a great way to explain it. Um, cool. Okay. Maybe we sit down and I ask some questions about automated research.

Eric Jang02:22:17

Sounds good.

Dwarkesh Patel02:22:18

One thing I really wanted to talk to you about is that you did a bunch of the research for this project through this kind of automated, uh, LLM coding assistant loop. And, um, there's an idea that if you fully automated AI research, you could have some sort of singularity. Uh, obviously, we're not there yet, but to the extent that we have early indications of what this process might look like, I am curious what your observations about, um, what the AI is good at, what it's not good at, what you think about this scenario's likelihood eventually, what thoughts you have about this in general.

Eric Jang02:22:54

For sure. Yeah. I think automated scientific research is one of the most exciting, um, uh, skills that, you know, the frontier labs are developing right now. And I think it's important for everyone who's doing any kind of research to get a good intuition of like what it can do now and what it can't and how might the sort of science process work in the future once we're having AIs automating a lot of this— this investigation. Um, so, uh, in brief, I mostly use Opus 4.6 and 4.7 throughout the, uh, working on this. And, um, what works is that the models can do a very good job of doing hyperparameter optimization.

Dwarkesh Patel02:23:28

Mm-hmm.

Eric Jang02:23:28

So in the past, people would kind of come up with a search base of hyperparameters like learning rate and, you know, weight decay and maybe how many layers are in your network. And, um, they would just kind of do a grid search or a sort of Bayesian hyperparameter optimization, uh, approach, and then it would find some tuned parameters. Um, the kind of really cool thing that automated, uh, you know, uh, coding can do now is that it can search a much more open-ended set of problems, right? It can say like, "Well, um, I've identified that like the gradients are kind of small on this layer, so let me change it up here. Let me rewrite the code so the data loader has a new augmentation I came up with.

Let's, uh, let's, uh, sort of try to find the best way to kind of fit the constraints of the optimization problem." And— and you end up with this much more flexible and kind of high-level, almost like grad student-like, uh, ability to just, you know, grind a performance metric.

Dwarkesh Patel02:24:15

Mm-hmm.

Eric Jang02:24:15

And— and so this can squeeze out quite a lot of performance. You can, you know, on a fixed data set with a fixed time budget, um, improve perplexity by quite a lot on— on a sort of classification problem like LLMs or, um, or Go. Um, and, uh, it is also fantastic now at basically executing any experiment, right? So I have a— a Claude skill that I wrote called experiment where, um, I give it a description of what I wanted to plot and like I just describe, "Here's the X-axis I want. Here's the Y-axis. Answer this question for me." And it'll go run off and do all the experiments, compile the plot, make a report, and suggest like, you know, what might have caused it or— or so forth.

Um, so— so that's what works quite well today. And I think we can expect that these abilities get better in the future, but it's also kind of useful to know, you know, what— what is it not doing so well today? Um, so on my, uh, blog version of this tutorial, I have a— a plot of basically all the kind of experiments I did grouped in a sort of tree where, um, you know, every node kind of represents a failed, successful, or sort of mixed experimental result. And then from there, it branches off into a child where it's like the follow-on experiment. Um, occasionally, I'll kind of rabbit hole down a track like this off-policy MCTS relabeling, do a few experiments, and then realize it's probably not worth it.

So then I'll kind of jump to a completely different track, right? And I call these kind of things like rows, right? So— so what— what I find is that current, uh, you know, closed models that we can access, the public can access today, um, they don't seem to be that great at selecting what the next experiment should be in a given track. And they don't seem to be able to kind of step back and do the lateral thinking of like, "Wait a minute, this track doesn't really make sense. Like, let's go back to sort of first principles and— and think about, you know, what the bottleneck might be or like what are we trying to achieve, right?"

And— and so often I had to catch infra bugs myself by prompting the right question to Claude to like investigate, you know, why— why— what— what is causing this discrepancy? And then it'll answer the question. Um, I think with like, you know, Mythos-class models or Mythos++ models coming online, um, maybe this just completely changes and these— these problems just fall to— to just improved scaling. Um, but at the same time, I think there's a lot of like rich opportunity to, um, develop RL environments that might incentivize this kind of lateral thinking. And— and so one of the motivations for setting up this Go environment was that I think that, you know, Go captures a lot of very interesting research problems, often overlapping with, you know, LLMs or robotics.

And yet it's like very quick to verify. Um, the outer loop is ultimately like, does the agent do what I think it does? And— and you can kind of check the outcome of a Go game quite easily. Um, and then the inner loop involves all this kind of like, you know, research engineering around distributed systems, uh, predicting whether an idea is going to work or not, um, predicting the, you know, the difference a particular modification to your training algorithm might make. Um, and I think there's a rich library of subtasks and sub-environments that you can kind of train an automated scientist to work on, uh, with Go as a sort of outer verification loop that then once you acquire these skills, maybe you can apply them to like other domains like, uh, you know, um, biosciences or robotics.

Dwarkesh Patel02:27:08

Or automating AI research.

Eric Jang02:27:09

Or automating AI research.

Dwarkesh Patel02:27:10

Which is— which is the real crux or the, um, scary/uh, incredible thing of just making AIs making future versions of AIs. And you're suggesting the outer loop here could just be your win rate against Katego basically?

Eric Jang02:27:22

Um, that's one of them. Um, I think there's a lot of deeper questions that one could tackle, right? So for example, um, let's say you have an idea on, um, how to improve a scaling law compute multiplier.

Dwarkesh Patel02:27:34

Yeah.

Eric Jang02:27:34

Um, the outcome isn't necessarily like, "I— I, uh, achieved the best Go bot ever." The outcome might just be like, "Can I predict what the win rate of my Go bot will be?"

Dwarkesh Patel02:27:44

Yeah, yeah.

Eric Jang02:27:45

Uh, or, "Can I predict the scaling law plots that emerge from my idea?" But then you can verify that you haven't kind of reward hacked anything by using a very verifiable game like Go on the outer loop.

Dwarkesh Patel02:27:54

I— I think there's a couple of interesting follow-on questions. There's questions on the inner loop and the outer loop. On the inner loop, there's a question of how locally verifiable any modification you might make is. That is to say, could would you know whether something is actually improvement or a degradation, some idea you try out? Would you know that if something isn't working as a result of, um, a bug or is it the result of the idea itself being wrong? Um, Ilya was talking about why having one of the reasons he— he thinks he's a good researcher is he is a good researcher.

One of the things he thinks makes him a good researcher is that, um, he has intuition about he has strong belief in what the correct idea is, and he is able to persevere through bugs and know which things are bugs versus mistakes in the fundamental idea based on his high-level belief about this idea should work, so therefore it's this there has to be bug versus the other way around. Why don't we start with that question actually? Yeah. How locally verifiable are things which are good ideas?

Eric Jang02:28:56

Yeah. I think as in the case of the success story for deep learning, you can think about this as like a decades-long idea that took like took a lot of faith to get it to work.

Dwarkesh Patel02:29:05

Mm-hmm.

Eric Jang02:29:06

Um, and so this presents a very challenging long horizon, you know, RL problem where, you know, every— every step of the way you have like a committee telling you that this is a bad idea and then ultimately you break it through, right?

Dwarkesh Patel02:29:17

Yeah.

Eric Jang02:29:18

And so like how do you design RL environments that maybe give you some feedback, uh, uh, earlier? Um, and— and I think this is a very tough open question that I don't have an answer to, but, um, but, you know, ultimately to play a very strong Go bot, you probably did need to discover deep learning.

Dwarkesh Patel02:29:34

Yeah.

Eric Jang02:29:34

Right? And so, um, um, I think that like having a challenging game that cannot be, you know, cheated easily on the outer loop could be used as a sort of outer loop signal for something like discovering the principles of deep learning. Now, of course, like to make it tractable and this is where research taste really matters, um, like you have to come up with ways to initialize your problem so that you don't solve a sort of very intractable problem, right? Like— like maybe you can leverage LLMs as— as a sort of a universal grammar in the middle to kind of give— give you some sort of local feedback.

Dwarkesh Patel02:30:07

Yeah.

Eric Jang02:30:08

Um, um, the— the fact that LLMs are universal grammar means that they can kind of move at almost any level of the stack, right? They can think very locally as well as step back and think like in very broad steps. And I think that's where a lot of, uh, um, the— the lateral thinking ability of humans kind of come from.

Dwarkesh Patel02:30:23

Yeah.

Eric Jang02:30:24

Like— like how to know if the track that you're pursuing or the objective that you're pursuing is not right and you should be asking a different question.

Dwarkesh Patel02:30:29

Mm-hmm. The, uh, the other question is how stackable local improvements are in the attempt to get to a better result on the outer loop. Um, I've heard rumors that at some AI labs, the thing that has gone wrong is that people will individually pursue good ideas, um, but those don't end up stacking well. And so the training run falls because of some weird, uh, interaction between two seemingly good ideas. And having a single top-down vision of how things should work is very important. Um, having worked at, uh, different AI labs and also playing around with, I guess, parallel agents trying different ideas, what is your sense of how parallelizable, um, AI— AI innovation is?

Eric Jang02:31:07

Yeah, great question. Um, I think the research taste for executing well on, you know, the bitter lesson is that you need to know how much the bitter lesson can buy you and, uh, how much is too much to ask for.

Dwarkesh Patel02:31:22

Mm-hmm.

Eric Jang02:31:22

At any given moment, right? Like, of course, in the fullness of time, compute kind of is the single most important determinant on like how things work. And, uh, and— and, uh, it's almost like inevitable that as you scale up energy and compute, uh, and parameters.

Dwarkesh Patel02:31:36

Yeah.

Eric Jang02:31:36

Intelligence will just fall out of that. And that's super, super beautiful, super profound. No algorithmic detail really matters beyond that. But, um, in present day, we don't have infinite compute and parameters and— and in and arbitrarily good initialization, so we have to come up with like heuristics that kind of give us that. But these heuristics are probably somewhat redundant. So that's probably why you see this effect where like a lot of these compute multipliers don't necessarily stack, is that like they— they might have some correlated benefit. Um, and then— and then, you know, three years down the line when the NVIDIA GPUs have gotten even stronger, maybe— maybe they stack even less well, right?

Like maybe like at any given point in time, the— the— the sort of benefit of any given compute multiplier is transitory, which is what I sort of suspected with the Katego paper. Like there was many algorithmic ideas kind of applied. And then you can see that like with, you know, modern Blackwell GPUs and Ada-class GPUs that are much better than the sort of V100, uh, grade GPUs that— that paper used, um, you can see that like some of these algorithmic tricks to speed up convergence just don't matter so much compared to something else. And I think that's a matter of taste in a— in the present time.

Dwarkesh Patel02:32:41

Yeah, interesting. How about the outer loop? How verifiable for making AI smarter? With Go, you do have this outer loop of, um, win rate against the best open-source model out there. And even there, as you were saying, there are other outer loops of did you discover a new phenomenon, which is actually very hard to if you didn't know scaling laws were important, if you're back in when was Chinchilla or Kaplan's scaling laws released? Like 2019.

Eric Jang02:33:09

2018.

Dwarkesh Patel02:33:09

Yeah. So if you're back in 2015, would you there's not an automated procedure one can easily imagine of, uh, knowing which paper is the scaling laws paper versus which is just like another random plot. And so that even in the Go case is a hard to verify outer loop. And the whole— the whole idea of an outer loop is to have like some backstop on, um, on improvement. Uh, but let alone for general AGI where of course we have a bunch of these benchmarks, but there's a problem that like we know the things we can measure and we improve on the things we can measure, but we're we care about this broader ability to do economically useful work, which is, um, at least until you automate everything, easy not— not super easy to measure.

Um, so yeah, there's a— there's a question of, okay, how— how good is the outer verification loop for, uh, for AI self-improvement and does that matter?

Eric Jang02:34:02

Yeah. Um, I'm going to give a non-rigorous argument, but one that I kind of intuitively believe, which is that, you know, um, DeepMind, the AI research lab, um, they started as a sort of focus on games, right? Like they— they kind of used games as their outer loop. And then the researchers learned, uh, from experience of solving games. And then like now they're working on LLMs. And presumably there was some positive transfer from their time working on games and like Atari and Go and— and, uh, you know, StarCraft that like now helps them make good LLMs. Like I— I assume that there's like positive transfer in some regard, whether it's coding or general research ability or project management, right?

Like all these things kind of like probably help them do well. Um, and so if that's the case, why wouldn't it also be true for automated AI researchers? Like they should be able to positively transfer experience tackling quick to verify, uh, quick to iterate on environments to something more ambitious and economically useful like, uh, you know, automating drug discovery or so forth.

Dwarkesh Patel02:35:00

I mean, I don't know. Isn't the— hasn't the issue with, uh, historically until Gemini 3 or whatever been a couple years ago people were saying, "Look, Google hasn't, uh, isn't catching up in LLMs because they're too tied to the old approach." And yeah, there's gains, but there's also, um, there's ways in it which actively hinders you. Um, so it's actually not obvious to me that there's like.

Eric Jang02:35:27

The jury's still out, right?

Dwarkesh Patel02:35:28

Yeah.

Eric Jang02:35:28

Like I— I think like who knows if the, you know, let's say currently Google's doing quite well. Who knows if the, uh, initialization on training on games is ultimately going to hobble their ability to be the winner in the long term, right? Like— like, uh, it's— it's hard to say for sure.

Dwarkesh Patel02:35:41

Yeah.

Eric Jang02:35:42

Um, and, uh, you know, likewise, who knows if the late seeming late start was really just them kind of pretraining for longer on how— how— how to like scale up TPUs, right? They— they invested all their tech tree in like, uh, getting TPUs to be good, which seemed not that useful in the short term, but then in the long term it becomes maybe like a.

Dwarkesh Patel02:35:59

Yeah.

Eric Jang02:36:00

Uh, so— so it's— it's even hard for humans to reason about what the optimal research strategy should be, right?

Dwarkesh Patel02:36:04

Yeah.

Eric Jang02:36:04

Um, even with, uh, the— the data we have today.

Dwarkesh Patel02:36:07

Yeah. Yeah. Cool. Um, okay. We should let people know how they can find out more about this project, whether to fork it themselves, whether to check out your blog because you're doing an excellent job explaining many of these ideas. Um, uh, where do people go next?

Eric Jang02:36:21

Great. Yeah. So my— my website is, uh, evjang.com. The there's a blog post that kind of links to a interactive version of this tutorial. Um, and on my GitHub, uh, which is the username is just Eric Jang, uh, there's a— there's a, uh, AutoGo repo that people can fork and reproduce the, uh, training results.

Dwarkesh Patel02:36:36

Yeah. And I also highly recommend people check out this blog post, "As Rocks May Think," which we touched on some of the ideas in this conversation, but it's this grander, uh, uh, you know, um, thesis of what happens when you have, uh, thinking as a primitive in.

Eric Jang02:36:50

Computer science.

Dwarkesh Patel02:36:51

Exactly.

Eric Jang02:36:51

Right.

Dwarkesh Patel02:36:52

Uh, and so I highly recommend people check out that blog post as well.

Eric Jang02:36:54

Yeah. And I encourage the, you know, the audience to, you know, think about the relationship between thinking and Go, you know, via MCTS and search and how it relates to LLMs. I think there's something quite like profound there, um, and probably under-explored just because Go has been relatively under-explored compared to, you know, the boom in LLMs.

Dwarkesh Patel02:37:11

Yeah.

Eric Jang02:37:11

Um, it's not to say that I think we should have trees in our— in our LLMs, but, um, but— but there is some very interesting duality between them. And you can actually do a lot of research on Go, um, MCTS and— and reasoning with, you know, very small budgets.

Dwarkesh Patel02:37:24

Yeah.

Eric Jang02:37:24

So that's very exciting.

Dwarkesh Patel02:37:25

Cool. Awesome. Eric, thanks for doing this.

Eric Jang02:37:27

It's— it's an honor to be on the podcast.

Episode description

Eric Jang walks through how to build AlphaGo from scratch, but with modern AI tools.

Sometimes you understand the future better by stepping backward. AlphaGo is still the cleanest worked example of the primitives of intelligence: search, learning from experience, and self-play. You have to go back to 2017 to get insight into how the more general AIs of the future might learn.

Once he explained how AlphaGo works, it gave us the context to have a discussion about how RL works in LLMs and how it could work better – naive policy gradient RL has to figure out which of the 100k+ tokens in your trajectory actually got you the right answer, while AlphaGo’s MCTS suggests a strictly better action every single move, giving you a training target that sidesteps the credit assignment problem. The way humans learn is surely closer to the second.

Eric also kickstarted an Autoresearch loop on his project. And it was very interesting to discuss which parts of AI research LLMs can already automate pretty well (implementing and running experiments, optimizing hyperparameters) and which they still struggle with (choosing the right question to investigate next, escaping research dead ends). Informative to all the recent discussion about when we should expect an intelligence explosion, and what it would look like from the inside.

Watch on YouTube. Read the transcript.

And check out the flashcards I wrote to retain the insights.

Sponsors

* Cursor‘s agent SDK let me build a pipeline to generate flashcards for this episode. For each card, I had an agent read the transcript, ingest blackboard screenshots, generate an SVG visual, and run everything through a critic. A durable agent is much better at this kind of work than a chain of LLM calls, and Cursor’s SDK made it easy. Check out the cards at flashcards.dwarkesh.com and get started with the SDK at cursor.com/dwarkesh

* Jane Street gave me a real deep-dive tour of one of their datacenters. I got to ask a bunch of questions to Ron Minsky, who co-leads Jane Street’s tech group, and Dan Pontecorvo, who runs Jane Street’s physical engineering team. They were willing to literally pull up the floorboards and take out racks to explain how everything works. Check out the full tour at janestreet.com/dwarkesh

Timestamps

(00:00:00) – Basics of Go

(00:08:17) – Monte Carlo Tree Search

(00:32:04) – What the neural network does

(01:00:33) – Self-play

(01:25:38) – Alternative RL approaches

(01:45:47) – Why doesn't MCTS work for LLMs

(02:01:09) – Off-policy training

(02:12:02) – RL is even more information inefficient than you thought

(02:22:16) – Automated AI researchers

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.dwarkesh.com