Transcript of Reiner Pope – Chip design from the bottom up

01:20:30 67 min read 0 views Published 6 months ago

Copy link
Episode summary

This episode of the Dwarkesh Podcast features a blackboard lecture with Reiner Pope, CEO of the chip startup MatX, on how chips work. Starting from logic gates, the conversation builds up to multiply-accumulate circuits and explains why multiplier area scales quadratically with bit width.

Pope argues that moving data between registers and logic often costs far more than the arithmetic itself, which motivates systolic arrays. The discussion also covers clock cycles and pipeline registers, FPGAs versus ASICs, caches versus scratchpads, and why GPUs can be seen as many small TPUs.

Listeners get a grounded picture of why GPUs, TPUs, FPGAs and even brains are organized the way they are.

In this episode Dwarkesh Patel, Reiner Pope

This transcript of Reiner Pope's chip design episode was made with HappyScribe technology, the same technology behind Teams AI note taker.

Dwarkesh Patel00:00:00

I'm back with Reiner Pope, who is the CEO of Matex, which is a new AI chip company. Last time we were talking about what happens inside a data center; now I want to understand what happens inside an AI chip. How does a chip actually work? Full disclosure, by the way, I am an angel investor in Matex, so hopefully you have designed a good chip.

Reiner Pope00:00:18

I hope so.

Dwarkesh Patel00:00:19

Okay. Also, if you're listening to this on an audio platform, it's much preferable to watch this Blackboard lecture on a platform where you can see what's happening. So switch over to YouTube or Spotify.

Reiner Pope00:00:31

So I'll start with, uh, sort of the very smallest fundamental unit of, of, of chip design, and then we'll sort of build up into what an overall, like, actual production chip, uh, what are the components of that.

Dwarkesh Patel00:00:41

Yep.

Reiner Pope00:00:41

At the very bottom level of a chip, the primitives that we work with are logic gates, which are very simple things like "and" or "not," and then these are connected together by, by wires that have to be laid out physically as metal traces on a chip. The main function that, that AI chips want to compute is multiplication of matrices, and really inside that is the fundamental primitive is multiply accumulative, just like of, of pairs of numbers. So we're going to sort of demonstrate what that calculation looks like by hand, and then sort of infer what, what a circuit would look like for that.

It'll turn out to be sort of easiest if, if I do multiplication accumulative, something like a, um, a 4-bit number with another 4-bit number. And then we're going to, the actual clearest primitive is actually multiply accumulate. So there's a multiply these two terms, and then we're going to add in, sorry, product of these two terms, and then we're going to add in an 8-bit number.

Dwarkesh Patel00:01:51

And, and can I ask a clarifying question? Why is this the, um, natural primitive for, um, you know, whatever computation happens inside a computer?

Reiner Pope00:02:02

Yeah. So, um, there's a few reasons for this. Uh, it's a little bit more efficient, but the, the reason it's natural for AI chips is that if you look at what's happening during a matrix multiply, the, uh, what is matrix multiply in very short, it is, um, uh, there's a for loop over i and over j.

Dwarkesh Patel00:02:24

Yep.

Reiner Pope00:02:24

And over k of output i k plus equals to, um, input i j times other input j k. And so multiply accumulate happens at every single step of, of a matrix multiply.

Dwarkesh Patel00:02:48

Makes sense.

Reiner Pope00:02:49

And then the other observation is that, um, the precision will almost always be higher in the accumulation step than in the multiplication step. This is maybe specific to AI chips, uh, but you're, you're multiplying low precision numbers, but then when you accumulate, errors accumulate quickly, and so you need more precision here. So this is why we've chosen to do a 4-bit multiplication and an 8-bit addition.

Dwarkesh Patel00:03:09

Let me make sure I understood that. There's two ways to understand that. One is that the value will be larger than the inputs, and the other is that if it was a floating point number, it would be. Maybe that, that part is, like, less intuitive to me, but it's maybe the same principle.

Reiner Pope00:03:25

It, it is really the same principle. Um, I guess the, sort of, I mean, I guess the, the separate principle is that as you are summing up this number, you are summing up a whole bunch of numbers, and so you've got a lot of rounding errors accumulating. Whereas in this case, there's, like, there's, there's only one multiplication in that chain, and so there's not a lot of rounding errors accumulating in the multiplication.

Dwarkesh Patel00:03:43

Why are you summing up a whole bunch of numbers? It's just two numbers, right?

Reiner Pope00:03:45

Just, I mean, the summation happens, it's repeated.

Dwarkesh Patel00:03:48

Ah, I see, I see.

Reiner Pope00:03:49

J and j multiplies.

Dwarkesh Patel00:03:49

So yeah, any errors accumulate. I see.

Reiner Pope00:03:51

Yes.

Dwarkesh Patel00:03:51

Makes sense.

Reiner Pope00:03:52

So how would we perform this calculation by hand? Um, I mean, as a human, we would probably separate it into two, but we can sort of do it all, all in one, um, using long multiplication. So the multiplication term first. We're going to multiply this number, this 4-bit number here, by every single bit position in the other 4-bit number. So we write that out. Um, first, 1001 multiplied by this bit position. Um, that is this number itself.

Then shift it across by one, we're multiplying by zero. That gives us an all zeros number. Shift it, uh, across even one more to multiply by this one. We get 1001. And then finally, for this last, last bit position, we get an all zeros, uh, number again. So this, this sort of gives us a bunch of terms that we're going to have to add for the, um, for the multiplication.

And then while we're doing that, uh, summation of this, we might as well add in the, the, the actual accumulated term as well. And so we just copy that directly across. So, so this is the sum. It's a, it's a five-way sum that we're going to want to compute. So firstly, what did, what logic gates did it, uh, take us to even get to this intermediate step? We needed to, uh, produce all 16 of these, um, partial products.

Dwarkesh Patel00:05:27

Mm-hmm.

Reiner Pope00:05:28

How do I produce one of these partial products? So let's take this, this number one, for example, here. It is one. So what was it? How do we produce this number by multiplying this number by this one over here? We can actually produce that by an AND gate. This, this number is one if both this bit is one and this bit is one.

Dwarkesh Patel00:05:47

Yep.

Reiner Pope00:05:47

If either of them is zero, then the multiplication of one times one, uh, of, of zero times anything is zero. So to produce all of this stuff, we ended up consuming 16 AND gates.

Dwarkesh Patel00:05:58

Mm-hmm.

Reiner Pope00:06:00

Or in the general case, if I were doing a, like, a P bit multiply times a Q bit multiply, then this will be, like, P times Q, many ANDs.

Dwarkesh Patel00:06:17

Exactly.

Reiner Pope00:06:21

Finally, I sum them. Actually, most of the work is going to happen in the summing. Um, and so let me describe, uh, sort of the other logic gate that we use here. AND is almost the simplest logic gate that exists on a chip. It's, it's, it's, it's almost the smallest. Um, at the other extreme, typically the very largest logic gate that you'll use is something called a full adder. And what this does is sort of, it does, like, coming from software, you might think that a full adder, like, adds 32-bit numbers together.

Uh, in this case, it just adds three single-bit numbers together. And so you can think of it as, like, adding zero, one, and one together. Now, when I add these together, the result can be zero, one, two, or three. So I can express that in binary using just two bits. So it, it, it as input, it has three bits, and as output, it has, it has two bits, which in this case are one and, like, uh, the number two in binary is one zero. So, so this is also known as a three-to-two compressor.

Because it takes three bits of input and produces two bits of output.

Dwarkesh Patel00:07:30

The two inputs are an x and a y value, and then some carry that, that came in from, like.

Reiner Pope00:07:37

The, sorry, the, the, the three inputs are, um, all, all bits that are in the same sort of bit position, like three bits that are in a column here.

Dwarkesh Patel00:07:45

Yeah, yep.

Reiner Pope00:07:45

And then the two outputs, um, I, I have sort of drawn them vertically here and horizontally here to kind of ma-match this vertical versus horizontal layout here, which is expressing that, um, things that are in the same column are in the same, like, bit position.

Dwarkesh Patel00:07:58

Yeah.

Reiner Pope00:07:58

Um, whereas things in that are in adjacent columns, like, this is a carry out, whereas this was a sum.

Dwarkesh Patel00:08:04

So if the inputs in the full adder are, let's say, like, 101, then the output would still be 10. If it was 111, it'd be 11. If it was 000, it'd be 00. If it was, like, 010, it'd still be 01.

Reiner Pope00:08:17

So yeah, it's just counting, essentially, the number of, uh, the number of things and expressing that in binary. So this circuit actually can sort of capture what we as humans naturally do, um, when, when we're, when, when we're doing summing along a column. So, uh, I can show that sort of, I'll show sort of one iteration of using the full adder to sum. Um, the way I sum here is going to be a little bit unnatural for humans. Humans, we would sort of sum along the column and then remember the carry. Um, but, uh, instead of remembering the carry, we'll actually just explicitly write it out.

So in this, we proceed from the rightmost column towards the left. On the rightmost column, we sum the one and the one, and that produces, like, a, a zero here and a carry of one. So we've, we've sort of used a, this full adder circuit on this pair of bits and produced a pair of bits as output. Now we can do the same thing with this column. We've got a column of one, two, three, four numbers. And so maybe we'll, like, take the first three of them, um, run a full adder on them, and that gives us a zero and a zero as output.

Um, so, like, the sum of these is, is zero, zero. And so that's the full compressor, uh, full, full adder applied to all of these bits. As, as I've used up bits, I'm going to sort of just cross them out to indicate that I've, I've, I've handled them. Let's just keep going a little bit more. Um, so we'll, we'll, we'll go here. I take these three numbers, I add them. That gives me a, a one and a zero. I've dealt with these three numbers. And now I take one, two, and I can even take these three numbers, for example, right now, uh, and add them, and that gives me a, a one and a zero, and I've dealt with these numbers.

So I can sort of, like, the way I should view this is that I have this whole grid of numbers that need to be added. Uh, I'm going to just keep applying full adders, uh, to all the bits that are here, constantly removing three numbers from a column and then writing out, uh, two numbers as output. Um, keep going with this over and over and over and again until I eventually get, like, some just one single number coming out here. Um, something like that. Uh, this is probably the wrong sum. So, so this approach that I've described here, this is called a, um, data multiplier.

Um, uh, and this is sort of, like, the, the standard for how, how you do, uh, area efficient multipliers using, uh, full adders. Let's try and quantify the circuit size of this, just so we have got a sense of, like, how big things are so we can compare to them later. How many full adders do I, did I use? I started with how many numbers? I have the 16 partial products, which is the product of all of these terms with all of these terms, plus the eight, um, terms that I'm adding here.

So I, I, I started off with 24 bits. And then on, I produced eight bits on the output eventually. And in every step, I was sort of crossing off three numbers and writing two numbers out as a, a result. And so every single use of a full adder eliminates one of the bits here.

Dwarkesh Patel00:11:32

Mm-hmm.

Reiner Pope00:11:32

And so how many full adders? It must be the 24 minus the eight. So there were 16 full adders in, in this circuit. In general, um, this is true in the general case as well. There will be P times Q many, um, full adders, um, uh, in this circuit.

Dwarkesh Patel00:11:47

Let me make sure I understand the logic of that. So the input bits, 24 is P times Q plus P plus Q.

Reiner Pope00:11:56

That's right.

Dwarkesh Patel00:11:57

And the output bits is just, uh, P plus Q. And so P times Q plus P plus Q minus P plus Q equals P times Q.

Reiner Pope00:12:09

That's right. So I think this explains sort of, uh, or at least hints at the second reason why we chose to do a, um, multiply accumulate. Um, first reason being that's actually what shows up in matrix multiplication, but second reason being it gave us this very slick, um, P times Q, very simple algebra.

Dwarkesh Patel00:12:26

Mm-hmm.

Reiner Pope00:12:28

So we've sort of described, like, this, this whole procedure, every single, uh, atomic step that I took here becomes a logic gate, and then sort of the wires connect it together. Like, when I had these three inputs that, that I summage to produce these two outputs, like, if I think of mapping this to a physical device, there would be a wire that runs sort of connecting all three of these things together into a logic gate that produced this output. Okay. So this is the, the main primitive at different bit widths, um, that, that is, that is inside an, an AI chip.

Um, uh, we're going to build up from here to, uh, how would you use that to run all of the other operations you may want.

Dwarkesh Patel00:13:08

Right. This might be the wrong time to ask this question, but whenever NVIDIA reports that this chip can do X many FP4 or half as many FP8, it seems to imply that those circuits are fungible, uh, that there's not as dedicated, like, FP4 versus FP8. Um, but the way you're, you're mapping it out here, it seems like you would need, if it has to be mapped out in the logic, you would need a dedicated, um, FP4 multiply accumulate and then a dedicated FP8 accumulate.

Basically, can you, can you funge them?

Reiner Pope00:13:44

As drawn, they're actually not particularly fungible. Um, this is actually one of the main choices you have to make when designing a chip, which is how much of FP4, how much of FP8 do I have? And then sometimes I'll make that consideration from the point of view of, like, uh, what do I think is the customer requirement? Another way to take an angle on that is to say, um, what is the, what is the power budget for, uh, equalize the power budget between FP4 and FP8.

Dwarkesh Patel00:14:05

But so then when they report those numbers, and they just happen to be the case that, like, it does two X as many FP4 as FP8, they just happen to choose, like, give, um, equivalent die areas to all the floating points. And as a result, it ended up being.

Reiner Pope00:14:21

Like, why is the ratio exactly two X?

Dwarkesh Patel00:14:23

Yeah, yeah, yeah, exactly.

Reiner Pope00:14:23

Yeah. So, um, part of it is, I mean, the, the, surely that won't be exactly, like, exactly equivalent to die area. Um, there's, uh, there's a data movement reason actually, and we'll maybe come back to this when we sort of, uh, look through how it goes into and out of memories.

Dwarkesh Patel00:14:37

Yeah.

Reiner Pope00:14:37

Um, there's something really nice just from a software level of the fact that I can pack two four-bit numbers into, into the same storage as an eight-bit number. And so when I store that to a memory or something like that, uh, it, the, the sizing of the, um, the buses that I wire out within the chip, um, actually makes that work out really, really nicely.

Dwarkesh Patel00:14:55

Actually, come to think of it, it's not just two X. It would be, uh, the, the amount of area it takes, it sounds like, is, um, quadratic.

Reiner Pope00:15:05

It's quadratic, in fact.

Dwarkesh Patel00:15:06

With the, yeah.

Reiner Pope00:15:06

With the bit length. So, uh, that's why smaller precision is, like, even more favorable than you might think.

Dwarkesh Patel00:15:12

This is a really big reason. So, um, in fact, NVIDIA made a change. Um, historically, uh, up until B100 or B200, every time you have the bit precision, you, you double the, um, the flop count. Um, that ratio is exactly, like, for the reason you said, because of this quadratic scaling, that ratio is actually slightly wrong. Um, it should be, like, an even bigger, you should get an even bigger speed up.

Reiner Pope00:15:35

Yeah.

Dwarkesh Patel00:15:35

Than, than you, than you might otherwise think. Um, NVIDIA's, like, product specs have sort of started acknowledging that in B300 and beyond, where the FP4 is three times faster than the FP8.

Reiner Pope00:15:47

Right.

Dwarkesh Patel00:15:47

Though it should be four X.

Reiner Pope00:15:49

Yeah. What I've shown here is, like, the simplest case of integer multiply. When you're, when you're dealing with floating point, as you do in FP4 and FP8, um, there's this sort of other term, which is the, the, the exponent that just complicates this, this calculation.

Dwarkesh Patel00:16:03

Oh, yeah.

Reiner Pope00:16:04

So, so what can we see already from this? It's like, I think the, the big observation you've made is that there's this quadratic scaling with bit width, um, which, which is, like, very effective and, and is the single reason why, um, low precision arithmetic has worked so well for neural nets. Um, but the other thing we're going to do now is we're going to compare sort of the area spent on the multiplication itself with all of the circuitry that is around it. So we'll, we'll walk back in time a little bit and, and see how did GPUs prior to TensorCores work, um, as which is the same way as the way that CPUs worked, in fact.

Dwarkesh Patel00:16:41

Hmm.

Reiner Pope00:16:42

So which is like, where do we stick this multiply accumulate unit? So generically, I'll describe, like, a CUDA core or a CPU. You'll have some register file, which stores some number of entries. Maybe, maybe it's like eight, um, eight entries, um, uh, of, of, like, in this case, I guess four-bit numbers, but typically like 32-bit numbers or something like that of, uh, which are numbers. Um, so this is the, like, inside the CUDA core, I'll have some register file of some depth, and then I will have my multiply accumulate circuit, multiply, uh, and accumulate circuit.

And what it's going to do, it's going to, like, it's going to take three arbitrary registers from this register file, perform the multiply accumulate, and then write back to the register file. So it's going to maybe write to this one, but it, it was able to read from this one, this one, and, and another random one. So it'll take three inputs like this.

Dwarkesh Patel00:17:44

Mm-hmm.

Reiner Pope00:17:46

So this is the core data path of, of many, um, processors. Most processors look like this. You've got some set of registers and then you've got some set of, uh, logic units or ALUs. We want to analyze the cost of the data movement from the register file to, to the ALU and back. So ultimately, there's going to be some circuit that says, well, I, I don't always have to select this guy. I might select any of the registers at any point in time. And so sort of a first question is, how can I build a circuit?

The circuit that I'm going to look for is, uh, is a MAX. So, um, in this case, it's going to have eight inputs, one from each entry of the register file. Um, and it's going to have one output, which is actually, um, producing this output. And then, like, what is the cost of this thing? It's, it's like, all we have to, to build it out of is and an or. Uh, and so how do we build it? We, we do the dumbest thing possible. We, like, form a mask saying we, okay, when we want to read, like, the third entry, um, we're going to and every single entry with either one or zero based on whether that's all we want to read.

And then we're going to or all of them together.

Dwarkesh Patel00:19:01

Okay. Just to make sure I understand the basics, what the MAX is doing is it's just like selecting.

Reiner Pope00:19:06

Just selecting.

Dwarkesh Patel00:19:06

Just selecting an input.

Reiner Pope00:19:08

Yeah. So like invisible to software is like, you say I want input number three, that means there's a MAX here.

Dwarkesh Patel00:19:12

Yeah.

Reiner Pope00:19:13

And so like, what is the cost of this MAX? So an N input MAX, uh, operating on P bits. Well, I'm going to, so I have N, N rows, that's this eight rows, and I've got, uh, like each row is P bits wide. Well, I have to and every single bit. So I get N times P many and gates. Every single input, I have to say, am I going to, like, mask it out or not?

Um, and then, and then I'm going to or them all together. And so, uh, there's going to be like N minus one times P many or gates, which is saying, uh, I've got all of these different things, almost all of them are zeros, but I need to sort of collapse them down into, into, like, from my eight options down into one option. And so every step I need to or like one row into, into an existing row.

Dwarkesh Patel00:20:09

Got it. Yeah. It's actually kind of funny that you would sort of, um, you don't think at the level of hardware, you sort of just think like, oh, I'll just select element three. And something as simple as that is, uh, a sort of like in and of itself a quite complicated circuit.

Reiner Pope00:20:24

Yeah. I mean, this is the first step of all of the hidden data movement costs that, that show up.

Dwarkesh Patel00:20:27

Right. Yeah.

Reiner Pope00:20:29

Um, and so like, the thing, like we're, we're just going to like compare, like I have to pay this cost and I've got one MAX here. And then in fact, I have two more copies of that for each of the three inputs to my multiply accumulate operation. And so I have this cost, which is like, like three times N times P and gates over here, um, compared to this P times Q, um, like, uh, sort of gates in the actual circuit that I, that is doing the thing I care about. And if we plug in actual numbers, like this N being eight, um, like I get like 24 times P gates over just, just in the data movement compared to like if Q is four, like four times P gates just in the, in, in the adder, multiply adder.

Dwarkesh Patel00:21:14

And sorry, where's the three coming from?

Reiner Pope00:21:16

Three different inputs here.

Dwarkesh Patel00:21:18

Got it. Okay.

Reiner Pope00:21:20

So the case like really just what I'm hinting at here is that like all of this work, which scales like as, as the size of the register file, and this is a very small register file, all of this work just moving the data from the register file to the, to the, to the logic unit, um, is many, many times more expensive than the logic unit.

Advertisement00:21:39
Dwarkesh Patel00:22:52

It might be helpful to just see what a MAX looks like. Maybe like a two-bit or a four-bit MAX.

Reiner Pope00:22:57

Yeah, great. So we'll take some inputs. We'll, we'll have maybe, um, uh, like, we'll just do a, a, a two-way MAX. So we've got, um, two different numbers. Um, we've got these two inputs and then we have a, um, so these are the inputs, um, that are being selected between. And then we have a selector. Which says, which can either be like, I want this one or, or it could be, I want the other one.

So this is a one-hot encoding. Um, so, so this is what, what we all start with. And then we, the output we want to produce, like, let's focus on this case. So, so this is the actual input we got.

Dwarkesh Patel00:23:46

Yeah.

Reiner Pope00:23:47

Um, we just want to produce this guy as, as, as the result. Um, and so like, sort of very laboriously, what we do is we and this bit with all of these. And so, um, that produces like anding this bit with this row. And likewise, we and this bit with this row, that produces all zeros. So this was the, there's four ands here. There's four ands here.

And then finally, we just or these two together. This gives a one. We or these two together. This gives a one. We or these two together, gives a zero. We or these two together and gives a one. And so this is the four ors. So like, this actually ends up looking a little bit like addition, in fact. Like we, we did exactly the set of same, same set of ands here. Um, sort of we've anded all of these things together, but then instead of collapsing it by using these full ladder circuits, we just get a very simple collapsing with or gates.

Dwarkesh Patel00:24:49

And how, but that, I guess that doesn't look like N times P.

Reiner Pope00:24:52

Um, so, so yeah, so this was, this was with, um, N equals two, um, inputs.

Dwarkesh Patel00:24:57

Ah, great.

Reiner Pope00:24:58

Um, in the general case, we will have, um, N, uh, like, and, and sorry, this, this is N rows. Um, and then we'll have, uh, P bits, um, uh, per row. Um, so that gives us the, um, the N by N times P many and gates. So this circuit I've described here, almost all of the cost, like, uh, seven eighths of the cost, uh, is, is in the reading and writing the register file.

And only a tiny fraction of the cost is in the logic unit itself. So this is the problem to solve. Um, this, this essentially was the, the state of play, um, prior to the Volta generation of NVIDIA GPUs. This is what, what this kind of thing is what was inside the CUDA cores. Um, and this, and this sort of problem statement is, uh, what motivated the introduction of TensorCores, which are more generically called systolic arrays.

Dwarkesh Patel00:25:54

Hmm.

Reiner Pope00:25:55

So if we think about how are we going to solve this problem, like we, we're spending almost all of our circuit area on something that we just really don't care about and is, and is like hidden to the software programmer. And, and the thing that we actually care about is, is, is not much of the area. Um, well, make this one bigger somehow while, while keeping this at the same size. That's the goal. So sort of the evolution was like, uh, we had baked this much into hardware, um, uh, in this stage that this single line is a multiply accumulate. Um, and this was, this single thing was baked into hardware. Um, the idea of a systolic array is to sort of go two levels of loop up and, uh, and, and bake, uh, this entire loop out here into, into hardware.

Dwarkesh Patel00:26:30

Mm-hmm.

Reiner Pope00:26:31

And so the idea being that if we have a much bigger granularity fixed function piece of logic, um, maybe the taxes we pay on the input and output are much smaller.

Dwarkesh Patel00:26:40

It sounds like you're suggesting that if you have, um, if you go up one step in the, uh, in the matrix multiply loop, that there's some, you, you can tilt the balance more towards compute than communication.

Reiner Pope00:26:54

That's right. Sort of there's two effects that we're going to take advantage of here. One is just that we can do more stuff before per every trip through a register file.

Dwarkesh Patel00:27:02

Mm-hmm.

Reiner Pope00:27:03

Um, and then the other thing we're going to take advantage of is, in fact, um, uh, in, in, in some of this loop, we can take advantage of, for example, uh, some, certain things staying fixed. So, uh, let's sort of visually, we're going to look at this matrix multiplication. So, so this portion of the loop corresponds to a matrix vector multiplication, in fact. So we'll, we'll take a matrix, um, and multiply it by, uh, uh, a vector. So how do we do this? We take every column gets multiplied by the vector and then sum.

So we're going to sum sort of along columns. Um, and so this zero and three gets multiplied by the three and seven and gets summed. And then the one and two gets multiplied by the three and seven and gets summed. So there is a multiply accumulate associated with every single one of these, um, entries in the matrix. So we'll just draw out these four multiply accumulates.

Dwarkesh Patel00:28:04

And just to make sure I understand why there's four multiply accumulates. So if each entry in the, uh, column that corresponds to the output vector is, uh, a dot product, and in this case, it'll be like two multiplications and then the addition of those two multiplications. So like you're accumulating.

Reiner Pope00:28:28

Yeah. So the addition, so really there's only one addition per dot product, but like we like to start with zero.

Dwarkesh Patel00:28:33

The initialization of zero.

Reiner Pope00:28:34

Yeah.

Dwarkesh Patel00:28:35

Yeah.

Reiner Pope00:28:35

So the, the, what we're going to aim for is to have, um, so we've got to, we want to have quadratically more compute. We do, we have, we've got sort of, um, X times Y, uh, as much compute as, as we had before. Um, um, but we're going to want to somehow aim for having only X times as much, um, like, uh, communication. And, and this is sort of the, the intention so that we get this, uh, advantage, uh, term going as Y.

So we've laid down the multiplications, bringing in, like we're going to want to bring in a vector of size two. And so that sort of already is in line with our comms target. That's fine. However, we need to somehow manage the communication of this matrix, which, which, which does, which exceeds our budget of, of X. And so, uh, the idea is that in the, in an AI context, this, this matrix is actually going to stay fixed for a long period of time. And so instead of like bringing it in from the outside, so we've got some register file sitting over here, we, we don't want to have like the amount of stuff coming out of this register file.

This is the term that we want to go, uh, sort of as, as X in some sense. Um, we don't want to bring this full matrix in from the register file every, every cycle because we don't have enough, um, that would cost too much in, in terms of wiring from the register file. And so instead we're going to store, um, our, our key trick is that this matrix can be stored locally, uh, to the systolic array. And so, um, where we'll store these numbers zero, one, two, and three in, in just like a gate called a register that like physically stores these numbers and we're going to reuse these numbers, uh, over and over again for, for a large number of different, uh, different vectors.

Dwarkesh Patel00:30:32

And so the optimization here is that like the nature of matrix multiplication is you can store this like, uh, square quadratic thing directly where the logic is happening.

Reiner Pope00:30:46

Mm-hmm.

Dwarkesh Patel00:30:47

Um, and which is like higher dimension than the, uh, or has an extra dimension compared to the inputs, which you keep swapping in and out.

Reiner Pope00:30:56

That's right.

Dwarkesh Patel00:30:56

I mean, this is the nature of what a matrix multiplication is, is that you do, uh, a lot of multiplication to get one value out. Like a dot product is the result of a lot of multiplications. And so that optimization means that you, you're just like, you can stuff a lot of like multiplication in before you get some value out of it.

Reiner Pope00:31:14

That's right. That's right.

Dwarkesh Patel00:31:14

Yeah.

Reiner Pope00:31:15

So like just to complete the picture here of, of concretely how that looks, um, uh, I swapped the three and the two here. Um, three and two. Um, so just like this zero and three is going to multiply by the three and seven. And so, um, uh, we're going to form a dot product sort of along columns here. So somehow we're going to feed a three and a seven in here. These participate in sort of this feeds into this multiplication and also feeds into this multiplication.

Likewise, the three feeds into here and also into here. And then we're going to sum, uh, sort of along here with like starting at the top of a column, we feed in zeros and then coming out the bottom, uh, we get results coming out. So sort of just to visually see what we've got, um, there's a dot product that is performed along columns in a matrix and that, that sort of maps exactly to what is done spatially in the systolic array here. Um, so this is one dot product summed vertically and this is a second dot product also summed vertically.

Um, and then what is the data that needs to go into and out of the register file? We have X amount of data that's coming out here on the output. Um, and we also have sort of, uh, this data coming from the input, X amount of data from the input. Um, and so with respect to the input and output vectors at least, um, we, we've sort of met our goal of having only X as, uh, as much data going in and out of the, um, uh, the register file.

This leaves open the question of like, I said that the, the weight matrices, weight matrix is stored locally in the systolic array. How did it get there in the first place? Um, is sort of a, like at some point you need to boot your chip and populate this data. And so where did that come from? The trick is just we just do it very slowly. So we, we very slowly trickle feed it in, into the systolic array. The, um, sort of the simplest strategy is that we, we sort of run this daisy chain that says like feed a number into here and then, and then on the next clock cycle, it'll move down to the next entry of the systolic array.

And so we can do that in every, um, in, in every column in parallel. And that gives us, um, sort of, uh, this is also going to come from here and this is going to be another factor of approximately X units of, of bandwidth coming in.

Dwarkesh Patel00:33:42

Uh, can you, can you just, would you mind repeating that sentence one more time?

Reiner Pope00:33:45

So like we're sort of like, we know that we're going to be bringing in, uh, numbers only rarely into the matrix.

Dwarkesh Patel00:33:51

Yeah.

Reiner Pope00:33:52

Um, and so we just want to come up with any construction at all such that the amount of wiring that actually feeds into sort of crosses this boundary of the systolic array, like this boundary right here, we just want to keep that bounded to X and not be, not go as XY. And so a particularly simple strategy is that we, um, we sort of bring in a, a number into the top row of the systolic array. That's what we can do in one clock cycle. And then for like, uh, for Y consecutive clock cycles, we're going to be bringing in the top row every time and then sort of shift all of the other rows down by one.

And that keeps the, the sort of the wiring that needs to come from this expensive register file, uh, only down to a factor of X rather than XY.

Dwarkesh Patel00:34:35

I see. Okay. So, uh, there's two questions in terms of communication. There's like communication time and then there's communication bandwidth.

Reiner Pope00:34:44

Yes.

Dwarkesh Patel00:34:44

And you're saying since we're only going to be loading this in once, let's maximize, let's minimize bandwidth.

Reiner Pope00:34:50

That's right.

Dwarkesh Patel00:34:50

Because bandwidth equals di area.

Reiner Pope00:34:52

Mm-hmm.

Dwarkesh Patel00:34:53

And let's just like load it in slowly over like smaller lanes because we're just going to keep this value in there for a while.

Reiner Pope00:34:59

Exactly. Exactly.

Dwarkesh Patel00:35:00

Interesting. So it's interesting to me that, uh, when we were talking last time about inference across many chips, the big high level thing we're trying to optimize for is increase the amount of compute.

Reiner Pope00:35:11

Mm-hmm.

Dwarkesh Patel00:35:11

Per memory bandwidth. That is to say per communication. And here also we're trying to increase the amount of like, uh, actual multiplies or actual additions relative to transporting, um, information from registers to, uh, to logic. So in both cases, you're trying to maximize compute relative to communication.

Reiner Pope00:35:30

Yeah. Um, this, this shows up sort of all the way up and down the stack. Um, this is sort of close to the bottom, um, sort of like to, to the gates. Um, there's sort of a, a version that's maybe even closer to the gates of just like, um, even the precision of number format that you choose to use. Um, we saw that same effect. Uh, there's like a square cube law or a like squared versus linear term going on both in just purely the precision of this, um, ALU, but then also in, in terms of the size of this, the, the matrix.

Dwarkesh Patel00:36:01

Yeah. Very interesting.

Reiner Pope00:36:03

So, um, so this unit is sort of the next bigger unit. We had like the multiplication circuit and then, uh, on top of that, we have, um, a, like a pretty large systolic array. I drew it as two by two, but in like, um, for example, older TPUs, they were described as 128 by 128, uh, of, of this circuit, uh, shown here. Um, and this circuit ends up being, um, this is the, the most efficient known, uh, mechanism for, uh, circuit for implementing a matrix multiply.

Dwarkesh Patel00:36:31

I see. So we, we've talked about, um, sort of it seems obvious that you should try to maximize compute relative to communication. Um, what are non-obvious trade-offs that, um, actually you are, you, you know, keep you up at night about should we do X or should we do Y? And it's not obvious what the answer is.

Reiner Pope00:36:50

Yeah. So, I mean, I think most of the decisions in chip design are, um, sizing decisions.

Dwarkesh Patel00:36:57

Mm.

Reiner Pope00:36:57

And so, uh, already in what we've drawn so far, um, like, so AI chips all have this, this circuit in it. They have a, a systolic array and then somewhere near it a register file, uh, providing inputs and outputs. Um, the two sort of, uh, like even within this scope, sizing questions that you have are how big should I make my systolic array and how big should I make the, um, the, the register file? The, um, so, and, and then the trade-off for the size of the size of the systolic array, actually these two questions are coupled is, um, one way to think of it is to say I'm going to like have a budget for, um, uh, how much, what percentage of my chip area I want to spend on data movement.

So maybe I just say that I want this to be 10% and the systolic array to be 90%. Um, and then sort of I can size my register file. Bigger register files are more flexible. I, they allow me to run sort of more, I can get more application level performance out. Um, but, uh, but then they sort of take away from this area spent on the systolic array.

Dwarkesh Patel00:38:05

Yeah. Makes sense.

Advertisement00:38:06
Dwarkesh Patel00:39:11

Where does the clock cycle of a chip come in? What, what determines what that is?

Reiner Pope00:39:18

Yeah.

Dwarkesh Patel00:39:18

And what is a clock cycle of a chip?

Reiner Pope00:39:20

So I guess at, at baseline, it's, it's sort of worth observing that chips are incredibly, incredibly parallel, right? You've got, uh, 100 billion transistors in a chip. A key thing that you need to do whenever you have, uh, like massive parallelism is you need to synchronize between the different, uh, parallel units. Um, in software typically you, like you, uh, you have these very expensive synchronization methods like a, a mutex. So one thread will finish what it's doing. It will ink, like it will, uh, grab a lock somewhere stored in memory and then notify the other thread that it's done. Um, on chips, we take a very different approach and say that, uh, uh, every like nanosecond or so, um, all, all circuitry in the chip will kind of pause for a moment and then, and then synchronize every.

So it actually synchronizes every single nanosecond or so. And so that is the clock cycle.

Dwarkesh Patel00:40:07

Mm-hmm.

Reiner Pope00:40:08

Um, the entire chip typically all in sort of one fell swoop, um, goes in lockstep to the next, uh, operation that happens. And so what this looks like in circuitry is that, um, you will have this typically drawn. So the clock is sort of mediated by registers, which are these storage devices that we've drawn elsewhere. Um, and the way to think of it is that I have, I have some storage, uh, which is storing like a bit, which might be zero or one. And then I have some sort of cloud of logic, which maybe is like this systolic array or this multiplier or something like that.

Um, and then I've got some, and that's going to produce some output. So my inputs, I've got a bunch of inputs feeding to this cloud of logic. And then eventually later there's going to be some output register that this writes to. There is a global clock signal which drives all of these registers. And it says at a certain instance in time when the clock, uh, uh, strikes, um, uh, whatever value happens to be on this wire at that instant, that's what's going to get stored in there.

And so the, the sort of the challenge here is like I, I would like to have my clock speed run as fast as possible because if I can run at 2 GHz, I can get twice as many operations done per, per, per second than if I run at 1 GHz. Um, but what that ends up meaning is that, uh, I am very sensitive to the delay through this cloud of logic because, uh, any computation that, uh, is going to happen in here needs to sort of finish before the next clock cycle hits.

Dwarkesh Patel00:41:48

Right.

Reiner Pope00:41:50

So, uh, a major point of sort of optimization on, on any chip then is, is to make, is to sort of make, uh, this delay, um, delay from here, uh, uh, as short as possible.

Dwarkesh Patel00:42:04

Interesting. And, um, is there ever, because the constraint here seems to be that if you add too much logic, then you might risk missing the clock cycle.

Reiner Pope00:42:15

That's right.

Dwarkesh Patel00:42:16

But if you don't add enough, then you're, you know, leaving potential compute on the table. Um, is there ever a situation where you're like, you'd take a probabilistic chance that a compute computation finishes and or is it just like, no, either it's going to finish by clock cycle or not?

Reiner Pope00:42:35

Yeah. In standard chip design, you, you margin it such that, I mean, there is a probability, but it's like many, many standard deviations, like way standard deviations out such that for all intents and purposes, it is a reliable part. It will, it will always meet the clock. Um, there are some weird exceptions to that. There are clock domain crossings where you go from one clock to the other clock and then you actually do have to reason about this probability. But.

Dwarkesh Patel00:42:58

Interesting.

Reiner Pope00:42:59

Um, uh, in the main path, uh, you just like you margin it such that you'll get there like 25% of the clock cycle in advance. Um, uh, so that it's very unlikely that.

Dwarkesh Patel00:43:09

In this, in this, um, the clock, uh, where the, uh, clock synchronize, I guess where the registers are, this is not something you determine as a chip designer. This is sort of just like an artifact of, hey, I want whatever sequence of logic and then the software you use to convert your Verilog into the thing you send to TSMC that just determines like, hey, in order to make this work, you got to kind of, you got to put a register here, here, and here to make sure that there's a, there's no one step that is like too long such that it makes the whole clock cycle of the entire chip longer than it has to be.

Reiner Pope00:43:47

Yeah. So this is actually a huge part of the work of, of designing a chip actually is, is inserting them. So it is done in a combination of manually and automatically.

Dwarkesh Patel00:43:54

Mm.

Reiner Pope00:43:55

Um, so, I mean, like just like to show the very sort of dumb, uh, version of like what you can do here, you can take this logic and split it in half. And so like say actually instead of just one, uh, cloud of logic, I'm going to, um, have two smaller clouds of logic, um, which do the same thing, but split them up by a register.

Dwarkesh Patel00:44:16

Right.

Reiner Pope00:44:17

Um, feeding in like this. Um, and this is like, like if you split it like in the middle, you can hit twice the clock frequency. That's great. You get twice the performance, um, at the cost of this extra register. And so at the cost of some more, uh, storage.

Dwarkesh Patel00:44:32

And so stepping back, why do we need to synchronize the whole chip? Like if you have like, if you imagine playing Factorio or something, there's no like global clock cycle. It just shit is done when it's done. There's iron on the plate. You can take it if you want.

Reiner Pope00:44:44

Yeah. So, uh, taking that analogy, um, the, the, the, the thing that you need to be mindful of is if I've got two different paths through some logic, so I have to do a computation like F here, um, and then computation G here. Um, and then they're going to come and meet for computation H somewhere here.

Dwarkesh Patel00:45:01

Yeah.

Reiner Pope00:45:02

Um, and so, uh, there's going to be manufacturing variance here. Uh, in some chips, F will take a little longer. Maybe in some chips, G will take a bit, a little bit longer. And so if I've got some signal that's propagating through here, um, and the result from F and G have to sort of meet up at H, um, what can the, the thing that can go wrong is that F can get there early and it meets like the previous value of G or the next value of G or something like that.

Dwarkesh Patel00:45:29

Ah, and H needs to know when to start.

Reiner Pope00:45:31

Exactly.

Dwarkesh Patel00:45:32

Like when, when has this next iteration of, and so this explains why different chips made at the same process node, the same like TSMC, uh, technology, um, can have different clock cycles. Like two chips made at three nanometer might have different clock cycles based on, uh, whether they were able to optimize making sure that like there's no one critical path that is so long that, um, it, it slows down the whole chip's clock cycle.

Reiner Pope00:46:01

That's right. This optimization that I, that I showed here, this is just the, uh, this is sort of, um, pipeline register insertion it's called.

Dwarkesh Patel00:46:08

Yeah.

Reiner Pope00:46:09

Um, we've inserted in the middle of the pipeline register here. Um, this is a sort of pure trade-off between clock speed and an area.

Dwarkesh Patel00:46:15

Yeah.

Reiner Pope00:46:15

Um, this is the easy case. There is a harder case too, which is, um, uh, sort of I've drawn it as a pipeline of logic here, but in, in other cases you may have some, uh, uh, some calculation which actually feeds back in on itself. So, uh, it runs some function F and then writes back to, uh, itself like this. So for example, this might be this addition, like you've got some number that you're adding to into every clock cycle. Um, and so this, this could be like a, um, this could be like plus, uh, where adding in some number every clock cycle.

Dwarkesh Patel00:46:52

Mm-hmm.

Reiner Pope00:46:53

So this, like this little circuit, uh, um, it essentially it's just going to sum all of the numbers that get presented on, on different clock cycles. And the challenge is if this plus takes too long, what can I do? If I like split it in, if I try and put a pipeline register like right in the middle of it, um, like, like here in the middle of it, um, this will end up changing the computation that's done. Instead of forming a running sum of everything that comes here, I will actually have two different running sums.

I'll end up having a running sum of the even numbers and a running sum of the odd numbers. So sort of this constraint where I have, um, a loop in my logic, which all chips have somewhere, um, this is actually the thing that is the hardest thing to, to, to address and, and sets the clock cycle.

Dwarkesh Patel00:47:39

I don't understand why it'd be a problem to have that. Or I'm not sure even what it would mean to have a lot a register there. Like is it, is it sort of, uh, atomic operation, right?

Reiner Pope00:47:47

Yeah. Well, so plus is not really atomic. Uh, like I think, um.

Dwarkesh Patel00:47:50

As you just demonstrated.

Reiner Pope00:47:51

Yeah. Yeah. It took a whole lot of work to do a summation. And so like you can take the early parts of that work and then, and then stick a register in the middle and then do take the late parts of that work.

Dwarkesh Patel00:47:59

Got it. Okay. Yeah. Uh, and I guess it's then up to, so TSMC offers a, uh, PDK which says, hey, here's the primitives of, um, logic that we can grant you in the, in the chip. And it's up to them to determine that no, no primitive is bigger than like the clock cycle they're hoping a process node targets. But other than that, is there like what further optimizing? Can't you just say like, hey, here's all the primitives from TSMC and if keep adding registers in between the primitives as much as is needed until you get to your desired clock cycle?

Reiner Pope00:48:34

Yeah. As a logic designer, like the chip architect set the clock cycle. So just for, for one example, the primitives you get from TSMC are on the order of like AND gates or full adders.

Dwarkesh Patel00:48:45

Mm-hmm.

Reiner Pope00:48:46

Um, they depends a lot on voltage and frequent and, and, and which library you choose and so on. But generally they, you can typically have about like 10 or 20 or 30 of these in, in a, in a clock cycle sequentially. So these primitives are very, very fast, like 10 picoseconds or something like that. Uh, and so as a logic designer, I mean, like in principle, if you literally just had like, um, like register and then AND gate, um, kind of in a loop like that, you could get an insanely fast clock speed, like more than four or five, six gigahertz, something like that.

Um, but if you take this, this like really sort of like simple circuit, um, and you look at the area you're spending here, like, um, this is maybe like one, I mean, this is, this is called one gate equivalent, um, in size. So like unit of one in area and this thing is like unit of eight in area or something like that. And so like this is just almost again, almost all of your cost has been this like synchronization or communication cost compared to the actual logic. And so, uh, so this would be a case where you've gone too far.

You've made your clock speed really, really fast at the cost of spending almost all of your area on.

Dwarkesh Patel00:49:56

Right.

Reiner Pope00:49:56

On, on, on, on pipeline registers.

Dwarkesh Patel00:49:58

Interesting. So what you're hinting at is a dynamic where you can have really fast clock speed, but you're not getting that much work done.

Reiner Pope00:50:05

Yeah. Yeah. Yeah.

Dwarkesh Patel00:50:06

Um, and so you can have like low latency, but, but low bandwidth or throughput rather.

Reiner Pope00:50:12

Yeah. The, it, it hurts your throughput in fact, because like, uh, like the, the throughput of your chip you can think of as the product of, um, how much do I get done per clock cycle, which is based on this area efficiency thing times how many clocks I get per second.

Dwarkesh Patel00:50:26

This is, this is actually so similar to the thing we were discussing last time about like batch size, um, where if you have a low batch size, then you can, any one user can receive their, uh, next token really fast, but the total number of tokens that are processed in, say, an hour will be kind of lower than it could otherwise be.

Reiner Pope00:50:45

Yeah, exactly. You get, you get less parallelism out if you, if you drive your clock speed up really high.

Advertisement00:50:49
Reiner Pope00:51:51

Okay. So I remember talking to, um, an FPGA, uh, engineer at Jane Street at Clark, um, who actually helped me prep for your, the, the previous interview we did together. And he was explaining why they use FPGAs and, um, uh, I, I imagine that for high frequency trading, the throughput is less important than latency. And so having very specific control over the clock cycle in a deterministic way is the most important thing. I, um, maybe it'd be interesting to talk about how you, why you can't just achieve that with an ASIC or why FPGA is the, uh, yeah, why you might use an FPGA to have deterministic clock cycles for like high frequency trading.

Speaker 300:52:34

Yeah. So I mean, firstly, like let's consider the sort of the business case for an FPGA versus an ASIC. Um, FPGAs and ASICs use largely the same sort of, uh, conceptual model, which is that I have, um, a series of gates built from ANDs or ZIGSORs, those like very small primitives, um, connected together with a fixed clock cycle, um, uh, and connected together with wires, uh, that are running in a fixed clock cycle. So anything you can express in an FPGA, you can express in an ASIC too, and it'll be about an order of magnitude cheaper and, and like, uh, better energy efficiency on an ASIC than an FPGA.

Uh, the trade-off is that the first FPGA costs you $10,000, whereas the first ASIC you make costs you $30 million because, uh, because of, uh, it requires an entire tape out. So sort of the business use case for an FPGA would be that I want something that has this very deterministic latency and, and, and fast runtime and high parallelism. Uh, but, um, uh, but you know, is, is I, I'm going to change it very frequently, change what I do every month or something like that. Uh, and so then I don't want to pay the tape, tape out cost every time.

Now, how does an FPGA actually implement? It's sort of like it emulates the ASIC programming model, but in, in, in a fixed piece of hardware. And so how does that work actually? So what it has, uh, at the, at the base is, um, it's got a, it's got a, um, it's got the two components we just talked about. It's got these registers as, as, uh, as storage devices. And then it's got, uh, these, these are called LUTs, lookup tables, um, uh, which actually provide all of the gates.

Dwarkesh Patel00:54:17

Mm.

Speaker 300:54:18

So, and then, and then we're going to see even the, the sort of the third component, we, we then have like, uh, sort of a, a swarm of these, um, registers and LUTs. Um, and all of these are available and then they're connected by sort of this big set of, um, sort of, uh, MUXes. So in front of, uh, every single one of these, we've got, uh, something like one of these MUXes, which selects one input from everywhere else, um, sort of selecting from all of these different things.

Um, you know, we've got a whole bunch of different options feeding into, into, into, into all of these things. So, um, so what, what, what this allows is essentially a, uh, when I program my FPGA, I can say that I'm going to take all of these components and I'm going to sort of superimpose on top of this a particular wiring, which like goes through this LUT and then feeds into this LUT and then goes to this register and then feeds into this LUT or something like that.

Dwarkesh Patel00:55:24

Mm.

Speaker 300:55:26

So what I've drawn in orange is like how you like FPGA means field programmable gate array. This is the, the orange is what has been programmed in the field, whereas the white is all of the wires that must exist in the FPGA in order to actually take, uh, to make the device in the first place.

Dwarkesh Patel00:55:41

What does it mean to be programmed into a field?

Speaker 300:55:43

Like programmed in the field. So like the device has been deployed in a data center, it's sitting in the field and then you can.

Dwarkesh Patel00:55:48

Ah, that field isn't like electric field.

Speaker 300:55:50

No.

Dwarkesh Patel00:55:50

Field as in like out there in the world field.

Speaker 300:55:52

Yeah.

Dwarkesh Patel00:55:52

Okay. Um, and so if I see, look at the, how the, uh, the field programming comes out of the first, uh, lookup table and goes in a second one, how is it, how, yeah, how?

Reiner Pope00:56:08

Like, like where, where are the wires that made that happen? I guess.

Dwarkesh Patel00:56:10

Yeah.

Reiner Pope00:56:10

Yeah. So I, I, I got a little bit like lazy in drawing all of these. Every single device here has a MUX sitting in front of it. Um, uh, uh, which can select from all of the like nearby, um, like, uh, circuits that are available.

Dwarkesh Patel00:56:26

Yeah.

Reiner Pope00:56:26

Um, and so, uh, the actual configuration of the FPGA is like amounts to it, it is the MUX control. So like we, in this MUX here, we have the data inputs and then we have like the control that selects.

Dwarkesh Patel00:56:41

Yeah.

Reiner Pope00:56:41

Uh, and so like there's a little storage device sitting next to every single one of these, um, uh, MUXes saying, this is where you're going to source your input from.

Dwarkesh Patel00:56:53

Got it.

Reiner Pope00:56:53

Um, and so programming it consists of like configuring every single one of these MUXes.

Dwarkesh Patel00:56:58

So that makes sense. What is happening inside the lookup table?

Reiner Pope00:57:00

Yeah. So the purpose of the lookup table, so it's going to also have a little bit of control feeding telling it what to do as well. The purpose of the lookup table is to function, to be able to configurably take the role of an AND gate or gate XOR, any of those different things. So there's many ways you could consider doing that. Um, the way it is done in sort of traditional FPGAs is to say, um, it will support, so it will be a, a lookup table will be, um, it will have four bits of input, one bit of output.

How many different functions are there from four bits to one bit? There are 16 different functions. Um, and so, uh, you can actually just tabulate this as like 16 different, like 16 different numbers. You've got a table of 0111001, 16 entries. And so what it does is, um, this, this table is stored in this blue configuration bit. Um, and then, uh, it views these four bits as binary, looks up the relevant row of the table and emits that bit.

So this is a truth table view of, of lookup tables essentially.

Dwarkesh Patel00:58:22

Okay. So the lookup table, if you think about an AND gate, OR gate, NOR gate, XOR gate, these are all like take as input.

Reiner Pope00:58:29

Those are like two input functions.

Dwarkesh Patel00:58:31

Yeah.

Reiner Pope00:58:31

Sometimes we have like more complicated, like a three input function would be a three-way XOR.

Dwarkesh Patel00:58:35

Right.

Reiner Pope00:58:36

Or a four-way XOR.

Dwarkesh Patel00:58:37

And in this case, how many it just depends how, how big it is, but.

Reiner Pope00:58:41

Typical size for LUTs is four input.

Dwarkesh Patel00:58:43

Got it.

Reiner Pope00:58:43

Which is sort of just a sweet spot between, um, there's another compute communication trade-off like here, like if it, if it has too few inputs, then you need to use more LUTs.

Dwarkesh Patel00:58:52

Yeah.

Reiner Pope00:58:53

If it, if it has too many.

Dwarkesh Patel00:58:54

But basically the lookup table is like a truth table.

Reiner Pope00:58:56

It's a truth table. Yeah.

Dwarkesh Patel00:58:57

And with a truth table, you can program in any gate you want.

Reiner Pope00:59:00

That's right. Yeah.

Dwarkesh Patel00:59:01

And so instead of lookup table, it just thinks like a programmable gate.

Reiner Pope00:59:04

That's right. And so, I mean, one of the things you can do here is you can see why, why the rule of thumb that, uh, an FPGA is like an order of magnitude more expensive than an ASIC comes from, um, is to count how many gates would be inside this lookup table.

Dwarkesh Patel00:59:18

Yeah.

Reiner Pope00:59:20

So we can view this lookup table essentially as one of these MUXes. Um, and so it is a MUX with, has to select between 16 different values. Uh, and so it is a MUX with, uh, sort of N equals 16, uh, options, P equals one bits. And so what we saw way earlier is that, um, this circuit costs like N times P many gates.

And so it's like, um, so it costs like N times P equals 16 AND gates and also 16 ORs.

Dwarkesh Patel01:00:00

This circuit being the MUX.

Reiner Pope01:00:02

Uh, yeah, exactly. The MUX, the MUX is the core.

Dwarkesh Patel01:00:04

The MUX that goes into the lookup table.

Reiner Pope01:00:06

Uh, so the lookup table itself you can think of as being actually a big MUX that like selects from all 16 rows down to one output.

Dwarkesh Patel01:00:14

Yeah. Okay.

Reiner Pope01:00:15

That is the lookup table.

Dwarkesh Patel01:00:16

But the way you've drawn it here, there's like a MUX and then a lookup table.

Reiner Pope01:00:19

It's, it's MUXes all the way down. So I mean, there's a, there is a second MUX that is inside here. This MUX is, is this MUX.

Dwarkesh Patel01:00:25

Got it. Okay. And then the, the other MUX is just saying, uh.

Reiner Pope01:00:29

Where it came from in this sort of mess of, of, of gates.

Dwarkesh Patel01:00:32

Right. And then the second MUX is, okay, now you have one value, but that value is still, um, still a four-bit value.

Reiner Pope01:00:40

Yeah. So I've selected four bits from the soup.

Dwarkesh Patel01:00:43

Right.

Reiner Pope01:00:43

Um, and then, and then I use those four bits to select which entry in the lookup table.

Dwarkesh Patel01:00:47

Yeah.

Reiner Pope01:00:47

Uh, I'm going to use.

Dwarkesh Patel01:00:49

Right. Okay. And just so I'm making sure, so like suppose in the first MUX there's like eight nearby, eight, you're, you're pulling from eight nearby registers as input.

Reiner Pope01:00:58

Mm-hmm.

Dwarkesh Patel01:00:58

Um, and so that's like a total of like 32 bits going in. Um, and then out of that, uh, four bits come out. Those four bits go into the second MUX, which is inside the lookup table.

Reiner Pope01:01:11

So actually I would say in, yeah, in this case, these registers are single bit registers. So if there are eight, like eight nearby registers and lookup tables, then I have eight bits total coming in, in, in nearby. I select from eight down to four different values. So there's actually like four different MUXes, one associated with each of these input, like. Little MUX associated with each of these input bits. Each of them is selecting one out of eight.

Dwarkesh Patel01:01:37

And where are those eight coming from?

Reiner Pope01:01:39

Nearby registers and other LUTs.

Dwarkesh Patel01:01:41

And each register is one bit.

Reiner Pope01:01:42

Yes. Yeah.

Dwarkesh Patel01:01:44

And so I guess you, AMD or whoever makes these, um, uh, FPGAs still has to be opinionated about what, uh, what registers are connected to which registers. Uh, and then you can program in the actual gates, but they had to wire in the connect, like the communication topology, right?

Reiner Pope01:02:03

Yeah. So there's the sort of like you get flexibility in a local grain thing. There's a sort of nearby neighborhood where you can select from. Um, but then more grossly, like more coarsely, longer distant connections, they, they form an opinion on that.

Dwarkesh Patel01:02:16

Right. Yeah. And the reason it's 10x slower is why?

Reiner Pope01:02:19

So if you look at the cost of like building this lookup table, it's like 32 gates.

Dwarkesh Patel01:02:25

Yeah.

Reiner Pope01:02:26

And then it can give me the equivalent of like what's one an interesting thing I can do here, I can do a four-way AND gate. And so that's like I am using 32 gates of lookup table to sort of implement like a four-way AND means like.

Dwarkesh Patel01:02:40

What is a four-way AND? I would do like AND, AND, and then AND of AND.

Reiner Pope01:02:46

So to implement, like this is a circuit that I could implement in an ASIC directly using these three AND gates. Um, but using a LUT, I can also implement it, but it's going to take like these 32 gates instead of three.

Dwarkesh Patel01:02:57

Right. And so the overhead is really coming from the, like, um, the fact that the lookup table, the MUX in the lookup table is. There's a more concise way to describe a truth table than listing out every single possible combination of inputs.

Reiner Pope01:03:15

Yes.

Dwarkesh Patel01:03:16

Uh, which is just to like write out the gate.

Reiner Pope01:03:20

Yeah. Like to, to like place down the polysilicon and the, and the wires.

Dwarkesh Patel01:03:23

That's right. That's right. Yeah. Interesting. One important point you made to me is that the reason they prefer FPGAs to CPUs is because they get deterministic clock cycles. They know when a packet will come in and go out. Um, why is it not a guarantee in CPUs?

Reiner Pope01:03:38

Hmm. So you can actually design a CPU that has deterministic latency as well. Um, uh, and in fact, like the, the processors that are inside a lot of, uh, AI chips actually also have deterministic latency too. Grok has advertised this. TPUs have that in the core as well.

Dwarkesh Patel01:03:54

Right.

Reiner Pope01:03:54

Um, the challenge is getting sort of deterministic latency and high speed at the same time. And so, um, where does the non-determinism in latency come from? Um, non-deterministic latency, um, comes from specific design choices in a CPU. Um, uh, it's actually possible to remove those design choices and make a CPU that has deterministic latency. Um, those are not very attractive in the market and so people don't make those CPUs anymore.

Um, but, uh, but, but actually in some sense like, uh, deterministic latency is maybe a sort of a simpler designing start, starting point. Um, and then, and then like some chip designers have added, uh, things in to, to be non-deterministic. To take a concrete example of that, um, the, probably the most important example is on a CPU just like the, the, the, uh, the CPU cache itself. So, uh, in a CPU you have the, the CPU, this is, this is the CPU die itself and then there's a memory, uh, off on the side.

This is the DDR memory, um, off on the side. And then you have a cache system here inside it is the cache, uh, that sort of remembers recent accesses to DDR and, and, and, and stores them. And so, uh, when, when I'm running through my CPU instructions, uh, every time I, uh, I have an instruction that accesses memory, it first checks in cache, uh, was, was the data stored in cache? And then if not, it goes, uh, fetches out to DDR.

Dwarkesh Patel01:05:31

Yep.

Reiner Pope01:05:33

This is a huge optimization. The cache is like two orders of magnitude faster than the DDR. Um, if you never, uh, if, if you never like use the cache, like all, basically all programs would run a hundred times slower. So, uh, the presence of a cache is absolutely necessary for a CPU to, to run at reasonable speed. Um, but whether or not you get a cache hit is dependent on the sort of ambient environment of the CPU, like what other programs are running, what has run recently, what is the random number generator inside the cache system doing? And so, so that is a big source of non-determinism in, in the runtime of a CPU.

Dwarkesh Patel01:06:05

Yeah.

Reiner Pope01:06:06

Um, so this is sort of the memory system for a CPU. Um, the, the big thing that you can do differently is, uh, instead of having the hardware say, I'm going to like read, uh, read memory and then decide the hardware decides whether or not it comes, comes, comes from cache or not. You can actually bake this in, this decision into software. So, uh, a different design philosophy is, is to, um, so, and you see this in maybe, for example, TPUs. Um, the, the TPU instead has, I mean, I'll draw the same diagram, but I'll call it a scratchpad.

And so the, the main difference is, um, so this would be like a TPU and then like HBM in this case rather than DDR, but it's still an off-chip memory. Um, and instead of like the software saying first access like memory and then the hardware decides, um, you've got some instructions that, that go here. This is like one kind of instruction and then a totally different kind of instruction that goes to HBM.

Dwarkesh Patel01:07:05

Yeah.

Reiner Pope01:07:06

And so this style is, is generically known as scratchpad, um, instead of cache. The key distinction being that you have like one kind of instruction that says read or write scratchpad and a totally different instruction that says read or write HBM.

Dwarkesh Patel01:07:21

So it's scratchpad being the cache.

Reiner Pope01:07:23

Yeah. This, this thing here is, is the scratchpad. Um.

Dwarkesh Patel01:07:26

Just to be clear. So stepping way back, people say computers have the quote unquote John Moyne, uh, Von Neumann architecture where there's this, uh, serial processing of information. And maybe just because we've been talking about parallel accelerators, but I just don't like the FPGA is super parallel.

Reiner Pope01:07:45

Yeah.

Dwarkesh Patel01:07:45

Uh, um, the, the kinds of AI accelerator TPUs are super parallel. Um, even CPUs are super parallel if you think about all the cores they have. And so is it actually like, in what sense is modern hardware actually the Von Neumann architecture? Is that actually a fair way to describe modern hardware?

Reiner Pope01:08:03

Um, I think it's a fair way to describe CPUs. Like just the amount of parallelism, like on a CPU, the amount of parallelism you get is about a hundred cores times maybe like 16 wave vector units. So 106, uh, about a thousand wave parallelism on a CPU.

Dwarkesh Patel01:08:16

Yeah. I, I, I so one question is what is the, there is a die that is being used for the CPU.

Reiner Pope01:08:22

Mm-hmm.

Dwarkesh Patel01:08:23

And if there's fewer threads, just as a matter of like transistor, uh, voltages or like switching on and off, is it just that there's like literally one control flow, like a small part of the die where like voltages are switching on and off or like in what, how, how do you actually occupy the die area of a CPU if there's a, as opposed to.

Reiner Pope01:08:45

If there's so few cores, like what do you, am I spending a lot of time?

Dwarkesh Patel01:08:47

What is happening in there?

Reiner Pope01:08:48

Yeah. The cores are just much bigger and more complicated. Um, so, uh, I mean, like, so I guess we, we should compare like a CPU core which takes up one sixteen, one hundredth of the die to like, I mean, to a LUT, like a LUT is just only these 16 gates.

Dwarkesh Patel01:09:03

Right.

Reiner Pope01:09:03

So like it's clear why there are so many more LUTs in an FPGA than cores in a CPU. Um, but then sort of maybe the, the like why are there more CUDA cores, for example, than, than, than CPU cores I think would be like why a, like what's the difference between a CPU and a GPU or something like that would, would be a, a big difference. Um, inside the CPU you have, um, so one big use of, so the sort of the top unit, uh, uses of area in a side of CPU are the cache. Um, very little is actually the ALUs.

Um, like mostly it's like these register files rather than the logic units. Um, and then the, uh, both of these things have equivalents in, in a GPU. And so that's not a big difference. But the thing that does not have an equivalent in a GPU is the, um, sort of this branch predictor. And so there is a whole big area in the CPU, which is, uh, sort of, uh, just a whole bunch of predictors that are saying, um, when will my next branch be and where the, where's the branch target for that?

And so, uh, stripping a lot of that out, uh, as, as well as sort of making these register files tighter in, in a sense, um, is, uh, driving a lot of where the, like GPU gains on the CPU.

Dwarkesh Patel01:10:19

And what is the purpose of the branch predictor to like execute both branches at once or what does it do?

Reiner Pope01:10:24

So the issue is that when I've got a series of instructions like, um, like instructions, instructions, instructions, instructions, I, uh, if I have a branch like here, if this instruction is branch, um, the, the actual processing step of processing an instruction, um, takes a really long amount of time. It takes like maybe five nanoseconds or something like that. Um, so like, uh, the time to actually notice that I've got a branch and then like, uh, uh, evaluate the boolean whether it's true and then, and then update the program counter to the new target and then read from the instruction memory for that.

Um, that, that could take like, like actually five nanoseconds to, to finish. And so in reality, this may finish somewhere down here. Um, I don't want to like, but like I want to run a clock speed that is much faster than what five nanoseconds allows. Like I, like five nanoseconds is 200 megahertz clock speed. I would like to run at one or two gigahertz or something like that. And so, um, I need to run other instructions while the branch is, is being evaluated. And so I, like I really just want to keep running the following instructions that happen after me.

Um, but that might have been wrong. Like if the branch ended up being taken, then I need to know that instead of evaluating these instructions, I actually need to like jump to wherever the target is and, and run, run these instructions instead. And so the purpose of the branch predictor is like, like genuinely to predict based on like before you even get to this instruction to be like five cycles earlier to predict there was going to be a branch that's going to happen.

Dwarkesh Patel01:12:00

So if I think about how the brain works versus what you're describing here, at a high level, the differences might be that while you can do structured sparsity in these accelerators and then save yourself some, um, area that you would have otherwise had to dedicate to these gates, in the brain, there's unstructured sparsity. You know, any neuron can connect to any other neuron and not in like ways where they'd be column aligned or whatever.

Reiner Pope01:12:26

Yeah. Yeah.

Dwarkesh Patel01:12:27

Um, then there's the fact that memory and computer co-located. I guess you could say in a way the memory and computer co-located on.

Reiner Pope01:12:35

This, this is exactly the co-location in some sense of the memory and computer.

Dwarkesh Patel01:12:37

That's right. That's right. Yeah. Yeah. So maybe, maybe that actually isn't a big difference. And the other maybe a big, a big difference is that the clock cycle on the brain is much slower than on, um, on computers. And partly that's to preserve energy because the faster the clock cycle, the bigger the voltage, uh, needs to be in order to identify for the signal to settle and to like identify what state of transistor is in.

Reiner Pope01:13:05

Yeah, that's right. That's right.

Dwarkesh Patel01:13:06

I don't know if you have other high level takes about like how any commentary on what, you know, what the brain might be doing versus how these chips work.

Reiner Pope01:13:16

Yeah. I mean, so, so let's take the clock speed one, uh, first actually.

Dwarkesh Patel01:13:20

Yeah.

Reiner Pope01:13:20

Um, the clock speed is quite high, uh, on, on a chip because that, I mean, drives higher throughput. Um, like when we compare a, like a GPU running some, some workload, it's running batch size 1000 or something like that. Whereas like the brain is not running batch size 1000. There's only one of me. And so you could sort of imagine saying, well, take a GPU and like instead of running at a gigahertz, run at a megahertz or something like that. And, and that would start to look maybe a little bit more like, like, um, uh, sort of equivalent things that you're talking about in the brain.

Um, there is in the way that silicon works, um, there are like that does not give you an 1000X advantage in energy efficiency. So, um, what it ends up looking like is you can, uh, like you sort of just end up running this circuit, uh, once to stabilization and then it'll sit idle for a long period of time. Um, it doesn't consume a lot of energy while it's sitting idle because, uh, most of the energy is consumed in, in sort of toggling, uh, bits from zero to one and back.

Um, uh, so actually let's, let's talk about the energy consumption, uh, of, of a circuit like this. The way to think of, uh, a bit being stored is you've actually deposited some charge in, in a capacitor somewhere sitting somewhere in, in the chip, uh, implicitly. So it becomes charged when it is, um, it becomes a one and then it becomes discharged when it next goes to a zero.

Dwarkesh Patel01:14:52

Yeah.

Reiner Pope01:14:52

Um, and that cycle of like charging the capacitor and then dumping that charge out to ground, that is where the energy is consumed.

Dwarkesh Patel01:14:58

Yep.

Reiner Pope01:14:59

This is called the dynamic or switching power. Uh, this is most of the energy consumption of a chip. There is some other energy consumption just coming from the fact that, uh, um, uh, insulators aren't perfect insulators, but we'll just discard that. Most of the energy consumption, uh, actually comes from just the charging and discharging of, or like toggling from zero to one and back to zero.

Dwarkesh Patel01:15:18

Right.

Reiner Pope01:15:18

Um, so if you run a chip much slower and you only clock it once every thousand clock cycles or something, you will have a thousand times fewer transitions. It'll be about a thousand times less energy consumption. Um, but, but like, but not a substantial advantage in energy efficiency.

Dwarkesh Patel01:15:33

Um, okay. So you described how a TPU works at a high level. Um, what is the difference, uh, at a high level between how a GPU and a TPU work?

Reiner Pope01:15:43

Yeah. So, I mean, I think there's sort of a high level organization principle that is different. Um, and then there's sort of inside the cores what are different. But we'll look at sort of, uh, outside the, uh, like at the high level. Um, so we'll take a GPU, um, and a TPU and what does like sort of the top level block structure look like? Um, if you think of this as the whole chip, um, in each case, um, the organization of, of the GPU, um, uh, is mostly a bunch of, uh, almost identical units, which are these, um, these are the SMs.

Um, and then they've got a, an L2 memory in the middle and then a bunch more of these SMs on the bottom. Um, and so, uh, there's sort of this fairly regular grid of, of cores. Um, uh, and then like if we look at a, at a TPU in, uh, in comparison, um, you, you, uh, you end up with much coarser grained units of, uh, logic. And so you, you end up with something like, um, uh, some large number of, um, maybe like maybe just a few, uh, matrix units.

These are the, um, these are the big like, um, systolic arrays. And then in the middle you've got some vector unit and then you've got your, um, matrix units at the bottom. So, um, now sort of like matrix units with a vector unit in the middle, sort of this is the whole TPU chip. You can sort of think of take scaling this thing down into a really tiny unit with a smaller matrix unit, smaller vector unit.

And that is sort of what a, uh, an SM is. So sort of at a very high level point of view, the, the GPU has a lot of tiny, tiny TPUs, um, sort of tiled across the whole, uh, whole chip.

Dwarkesh Patel01:17:36

Oh, interesting. So like you're suggesting the tensor core within a streaming SM is analogous to an MXU.

Reiner Pope01:17:42

Yeah. It's very, very similar. Yeah.

Dwarkesh Patel01:17:44

I see. And so if you had more like, more lack of structure, having a bunch of tiny TPUs makes a lot of sense. Whereas if you kind of just have like huge matrix multiplications, you're like, why don't we just, why don't we avoid the cost, the cost of having the individual SMs with their own registers and warp schedulers and things like that? Why don't we just like make a huge thing and like amortize those costs across the whole thing?

Reiner Pope01:18:12

Yeah. And I mean, I think this shows up in how large you can grow things. We've, we've sort of seen this theme, like especially with a systolic array where larger systolic array amortizes the register file costs better.

Dwarkesh Patel01:18:22

Yeah.

Reiner Pope01:18:22

Um, this sort of design allows you to have larger systolic arrays. This, whereas the sort of GPU design constrains you to having small units, uh, of everything. Um, there is a tradeoff, however. Um, the, there ends up being, um, because of this sort of coarse grained separation of things there, you need to move a lot of data from the vector unit to the matrix units. Um, and so like you need to move a lot of data through, through a sort of, uh, like two, two lines of perimeter here.

Um, whereas if you sort of look at the equivalent thing here, you've got vector units everywhere and you need to move data through this line, through this line, through this line, through this line, through this line, through this line. So the amount of data you can move between a vector unit and a matrix unit is actually much higher in, in a GPU than in a, in a TPU because, because like it's like instead of having to like move all the data through these just two lines, you're moving all these data through like 16 lines or something of, of, of, of, of wiring instead in a GPU.

Dwarkesh Patel01:19:27

Right. But also you might have to move across less area.

Reiner Pope01:19:32

Which I mean is also a saving like it's an energy saving as well. So, so data ends up moving like if you can operate entirely within an SM.

Dwarkesh Patel01:19:39

Yeah.

Reiner Pope01:19:39

The data movement is much smaller, but then the moment you want to operate across SMs, like it becomes sort of more, more complicated and expensive.

Dwarkesh Patel01:19:47

So you don't have a comment, but one might expect that a thing that Maddox might try to do is to get the GPU-like smaller structure of, um, systolic arrays surrounded by, um, SRAM, but also at the same time make it so that like the things you need in an SM to support the CUDA architecture, uh, but take a bunch of, bunch of space you might discard.

Reiner Pope01:20:14

Yeah. Um, we've talked publicly about something which we call a splittable systolic array, which is sort of in, in some sense you can think of as like big systolic arrays that can be small systolic arrays as well.

Dwarkesh Patel01:20:24

Cool. Um, okay. I think that's a good note to close on. Reiner, thank you so much.

Reiner Pope01:20:29

Thanks, Rakesh.

Episode description

New blackboard lecture with Reiner Pope: how do chips actually work - starting with basic logic gates, and working up to why GPUs, TPUs, FPGAs, and the human brain each look the way they do.

Reiner is CEO of MatX, a new chip startup (full disclosure - I’m an angel investor). He was previously at Google, where he worked on software efficiency, compilers, and TPU architecture.

Watch this one on YouTube so you can see the chalkboard. Read the transcript.

Sponsors

* Crusoe was one of only five GPU clouds that made the gold tier in SemiAnalysis' most recent ClusterMAX report. Gold-tier providers like Crusoe delivered 5-15% lower TCO than silver-tier clouds, even with identical GPU pricing. This is because optimizations like early fault detection and rapid node replacement don't necessarily show up in the sticker price, but still matter a ton in the real world. Learn more at crusoe.ai/dwarkesh

* Cursor is where I do most of my work—from reading research papers to visualizing technical concepts to coding up internal tools for the podcast. Most recently, I used it to build two different review interfaces for my essay contest, one that anonymizes submissions for scoring and another that lets me see applicants' essays next to their resumes and websites. Whatever you're working on, you should try doing it in Cursor. Get started at cursor.com/dwarkesh

* Jane Street let me ask Ron Minsky and Dan Pontecorvo, two senior Jane Streeters, a bunch of questions about how they use AI. We discussed everything from the types of models they're training to how they think about the future of trading to why they're more bullish than ever on hiring technical talent. You can watch the full conversation and learn more about their open positions at janestreet.com/dwarkesh

Timestamps

00:00:00 – Building a multiply-accumulate from logic gates

00:16:31 – Muxes and the cost of data movement

00:26:10 – How systolic arrays work

00:39:11 – Clock cycles and pipeline registers

00:51:51 – FPGAs vs ASICs

01:03:25 – Cache vs scratchpad

01:07:27 – Why CPU cores are much bigger than GPU cores

01:12:00 – Brains vs chips

01:15:33 – A GPU is just a bunch of tiny TPUs

This is a public episode. If you would like to discuss this with other subscribers or get access to bonus episodes, visit www.dwarkesh.com