Transcript of After The Builders: On Dario & Daniela Amodei and Mustafa Suleyman

This is an iHeartRadio podcast, Guaranteed Human.

Welcome to Tech Stuff. Oswald Oshin here. It's another special edition of the podcast today, focused on Kaleidoscope's new series, The Builders, with Walter Isaacson. This time, we're going to discuss the 3rd and 4th episodes, where Walter interviews Daniela and Dario Amodei, the 2 co-founders of Anthropic, and Mustafa Suleyman, the CEO of Microsoft AI. Dario and Daniela left OpenAI in 2020 after disagreements over AI safety with Sam Altman. Mustafa Suleyman was pushed out of DeepMind, which he co-founded, in 2019 after 2 years of grinding negotiations with Google about AI safety and following complaints about his management style. Today, they represent different, perhaps even opposite sides of the debate on safety, with the Anthropic co-founders advocating for encoding ethics in AI and Suleyman arguing for the primacy of human control. It's a fascinating pairing, and today I'm joined by my Kaleidoscope co-founder, Mangesh Hattikada, as well as 2 journalists who know more about these figures and the debate about the future of AI than anyone I know. We're delighted to be joined by Sebastian Mallaby, currently a senior fellow at the Council on Foreign Relations and the author of many books, most recently The Infinity Machine: Demis Hassabis' DeepMind and the Quest for Superintelligence.

And his forthcoming book, recently announced and untitled, is about Dario Amodei. Great to be with you. And Natasha Tiku, Silicon Valley culture and AI reporter for The Washington Post and regular Take Stuff contributor. Welcome.

Thanks for having me.

Before we get into the builders, we'd love to hear a bit from both of you about what it's like to report on these people. Natasha, maybe starting with you, I'm curious about the kind of arc of reporting on these tech leaders, particularly in the age of the self-published viral essay.

Yeah, it requires a lot of time spent reading thousands and thousands of words. I put actually recently all of Dario's essays in a Google Doc, because I'm just highlighting, and there's so many of them, and I had to even incorporate different tabs for his recent long-form tweet responses. So yeah, it's kind of like Talmudic studies. Sometimes it seems like that.

Do they write them all themselves?

I've heard that it is a norm within Anthropic that it's actually very bad form if it seems like you have used Claude to do your own writing, which I think is like a beautiful irony. Yeah, I do believe he's written them by himself.

But Jack Clark is also prolific, the other Anthropic co-founder right I mean, to keep up with all the things the Anthropic co-founders write, you'd have to be quite the reader. Sebastian, what about you? How do you get in the heads of these founders? I mean, Demis and now Dario?

Well, with Demis, it was a question of sitting with him in a pub in North London for hours and hours and hours, more than 30 in total, and just getting him to go through his life story and reflect on different bits of it. He's a great narrator, so that was actually super fun. And he tells, he self-dramatizes very well, and you have to maybe slightly filter a few things where you think it may be not quite true, but essentially he is relatively trustworthy and extremely fluent. And the thing about him is he doesn't write that many essays. He did one this year in July on safety, but the way he's told his own stories through lectures and podcasts, there hadn't been a kind of long-form written book which had access to him, which kind of covered not only his childhood and all that, but the linkages between what he studied at Cambridge as an undergraduate, Claude Shannon and so forth, and then how he developed this crazy conviction that you could build powerful AI and founded a company to do that in 2010, 2 years before AI recognizes the photograph of a cat.

I mean, the conviction was wild. So, Mangesh, obviously you and I are quite proud of our series, The Builders, and there's a little bit of kind of fun, wild ride of how it all came about. Why don't you tell us about the Amodei interview first?

Yeah, so we're all obviously fascinated by the Avadaes, and we're trying to book them. We had reached out both asking for Dario and Daniela. We thought we'd be able to get them separately. And initially, they kept saying, well, Dario is available this time. And we were pushing more for Daniela's voice as well. And then they said, would you like to do them together? And we thought, perfect. There's only one interview that they've done together before, and that's with Oprah. And so—

Very good company.

I know.

And so, you know, we managed to secure them. We were so excited. And Dario got to the interview first, and he was there about 10 minutes early. And Walter was there as well. And so, you know, Walter is Southern and charming, and he's trying to cajole him into chatting a little bit. And Dario kind of waves at him and then just looks down, and he does not want to do any small talk.

Oh, my God.

And Walter just kept saying, like, you know, a little bit more, a little bit more. And— Dario was like, you know, I'd really prefer to wait till Daniela's here. And so it was just silence until Daniela gets there. And then immediately, he turns the charm on. He's so excited to talk. And you see how close they are. And you realize that he's obviously a CEO who's a visionary, but doesn't manage anyone. And she really both protects and translates his brilliance so effectively to a larger world. And it's kind of remarkable to watch them interact. And I really feel lucky that we managed to get them together.

Natasha, Sebastian, maybe Sebastian, you first. You were both smiling at that account of the pregame.

Yeah, I guess the compartmentalizing is what people have to do when they're super busy. And if you've got that extra 7 minutes before the real thing starts, you're going to use it to think of another big thought because the value of each incremental big thought is measured in the billions of dollars and the safety of humanity might depend on it. So this is serious stuff. So look, I mean, I recognize that tendency of being super efficient with one time and Demis, just to go back to him as a kind of baseline comparator, has this routine where he has 2 working days. He works a day in the office, he comes back, has dinner with the kids, goes up to his home office, works from 10:00 AM until 4:00 AM the next day. So he gets the sort of reading, thinking, cogitating time. And that's another example of being just aggressively efficient with how much you can eke out from each 24-hour unit.

So there's a scene in your book where Demis and Mustafa have dinner every Sunday night in the pub and they wait until the last orders to order so that they can eke out as much conversational time as possible. But Natasha, what about you?

I mean, I watched the video, so it was just reminding me of times when Dario knew that he would get in trouble, so he just like handed it over to Daniela. And I was watching Daniela's hands when Dario was getting a little bit too colorful. So it was just, it was a rich text, the video itself.

Love a video podcast. Before we get into actual interviews though, can you give us some context, Natasha, starting with you? What do we need to know about the Anthropic IPO this week?

I think you need to know that it is one of the fastest growing companies in history, period. People are anticipating a valuation of potentially $2 trillion. There's also been reports of $1.5 trillion, but you have to keep in mind, this is a company that is less than 6 years old, and the text of the SEC filing is exactly what you would expect from the Amadeis, right? Like they talk about bringing forth an age of miracles. If you buy a share in their company, don't think of it as a transaction. Think of it as being, you know, part of this momentous period in time. They talk about how each of their employees is worth $6.6 million. Like that's how talent dense this organization is. And, um, they also go through some of the existential risks, which they touch on in the interview as well.

Sebastian, I mean, I know it's just been announced in Publishers Weekly and it's sort of probably a quiet period, so to speak, on the book, but what's your biggest question about Dario Amodei going to be?

Yeah, I mean, it is super early. I've only just begun interviewing, but I would say the questions going in are, in a way, how did this extraordinary thing happen? These guys left OpenAI, the 7 founders of Anthropic, they left in 2020. They start the company in 2021, and at the time, they're way behind. They don't have a big source of funding like Google DeepMind did because of Google's enormous balance sheet. They didn't have the head start that OpenAI had. OpenAI then goes on to do the ChatGPT, goes viral in 2022, and Anthropic seems even more behind. They scrounge together some money miraculously from the effective altruist community with Sam Bankman-Fried at the center of that. And then he goes to prison. I mean, it's like there's setback after setback after setback. And yet it turns out that by 2026, these guys are at the forefront scientifically. They are the biggest revenue, fastest revenue ramp, I think, in the history of capitalism. I would go maybe one notch further than Natasha did. I think it is the biggest, fastest ever. And then also in policy terms, they stick their necks out, they take principled stands, they take on the Trump administration.

And they make the weather on the kind of policy and safety front. So I think there's a kind of triple here, right? The policy stuff, the business thing, and the scientific advance.

It's so crazy how the tech news cycle is nowadays because Microsoft announced they've offered buyouts to 60,000 employees, more than 25% of their entire workforce. It's probably like the 5th biggest tech story of the week. But I mention it just because obviously the other person we're going to be talking about in this episode is Mustafa Suleyman of Microsoft AI. Have either of you followed these buyouts? I mean, the premise is unlocking more capital for compute, essentially, I think.

Yeah, I mean, I haven't followed that specifically, but it's part of a big thing where when you're trying to onboard a lot of artificial intelligence, and I use that word deliberately, when you onboard a human, you have to kind of figure out how to use that person, train them, and so forth. It's kind of the same with AI. You have to kind of figure out how to integrate AI into your business processes to make it productive. And that is an enormous suck on management time and management focus. And these guys just don't want to have too many humans knocking around when they're focused on this other thing. And it's already the case that in Anthropic, for example, large amounts of the tasks that might have traditionally been done by humans, for example, thinking about working on the IPO, I'm told that there are large numbers of AI agents that are doing a lot of that work.

They can't write the essays, but they can write the IPO prospectus.

Well, they probably could write the essays. It's just that existentially, we don't want them to do that because writing an essay and expressing your view, if you're the leader of the lab, I mean, it's kind of like, that has to be you.

Natasha, what's the vibe in Silicon Valley? I mean, at the same time as any number of Anthropic employees who are now worth exactly $6.6 million each versus all the Microsoft employees being bought out, What's the mood?

The mood is, can I get a job at Anthropic? Otherwise, you'll be part of the permanent underclass. I mean, there's just so much money sloshing around. You see people expressing, oh, here's me in a picture of Ilya 6 years ago, Ilya Sutskever. Did I make the wrong decision? Why are some of the people who have the exact same background as me billionaires? Matt Levine, his Bloomberg write-up about the S-1 was great. It was basically like, Claude is going to subsume all human endeavor, so why not buy a share of Claude? That's, that was kind of the IPO pitch. And, and that is the mood right now, I would say. Um, you know, kind of mouth agape envy. I mean, this is the feat of capitalism is, is astounding. I keep thinking actually back to, uh, the Peter Thiel Walter Isaacson interview where he said, can you think of a cult that has ever done this? You know, You can't— it's phenomenal to think of somebody with deep ideological beliefs that is actually at the pinnacle of capitalism.

Yeah, truly. Well, I mean, let's get into that. I mentioned at the top that debates over safety in some sense informed both Mustafa's departure from Google. I want to hear more from both of you on that, but also, of course, Dario and Daniela from OpenAI. So this was one of Walter's more controversial lines of questioning in the interview.

Probably the body language told its own story, but let's play it now.

Dario, you say your departure from OpenAI was partly due to a breakdown of trust, an interesting word, you know, with Sam Altman. Explain how you felt there was a breakdown of trust and honesty there.

What I will say is, look, leaving OpenAI and founding Anthropic, I mean, It ultimately comes down to the principles and how we're gonna build the technology. That is what matters at the end of the day. But I will say that the moment when I realized that we needed to found our own company, that OpenAI was not the right place for us— look, these safety issues are difficult. You can have lots of disagreements on them. I disagree with the co-founders and with lots of people at Anthropic But the reason we've been able to resolve these disagreements is because there's always trust that everyone is trying to get to the right answer and everyone is being honest. My point in saying that there's a breakdown of trust is simply to make the abstract point that if you don't believe that about a place where you're working, if you don't believe that the disagreements are being held in good faith and that you're getting the real story from people, how can you possibly hope to resolve them? That's why we left. That's why we founded our own thing.

And I think the last 5 and a half years have, have, are themselves just the best proof of how our philosophy differs from that of the other companies in the space. I think if you continue to work for someone who has a different vision than you, it becomes very unclear what you actually stand for.

Wait, wait a minute though. I'm pushing back a bit here because you talk about differences in values and stuff.

But that's fundamentally different than a difference in trust, that you feel that somebody says they have the same values, but they don't, and you can't trust them.

Honesty is one of our values.

It's a very important value.

It's hard to know what someone's values are if they don't have the value of honesty.

And you felt that about Sam?

I think the way, at least for me, I would put it is, I think the co-founders are an unusually high-integrity group of people. And all of us, I think, shared a very similar vision for how we wanted this technology to go and how we wanted to build it. But at a certain point, it just makes more sense to go do this as your own organization.

How are you gonna get someone else to implement or share your vision if they don't naturally do so? No matter how much they say they share it, if that's not reflected in what the company actually does, what choice do you have?

Natasha, I want your reaction, particularly. I loved your smile at Daniela's interjection.

I think that was actually the part where he, like, hit— like, you know, moved his hands and said, like, Daniela, you take over. I'm assuming that's because they said, Dario, you're way too charged up to actually make comments about Sam in any interview. Please defer to Daniela. And then he couldn't help himself. He, like, you know, went back into it. That whole section was so fascinating because, you know, obviously, this animosity and disapproval, hatred in some cases— I'm not sure. You know, I don't want to put words in anybody's mouth, but that animates so much of Dario. And you can see that throughout this whole interview, right? Like, he comes back to it again and again, you know, wanting to make a company that is so different, from the way that OpenAI operated. And a little bit of what he's alluding to, I think, in that clip is this emphasis on culture fit, right? Like, which Anthropic famously has this interview process where, you know, you can sail through the technical interview, you can do really well, and then they have this culture fit interview, which actually involves reading some of Dario's essays. I was talking to somebody who sailed through the technical part and then got cut off at the, at the culture part.

And they asked, like, do you share this same views on AI safety? How much of an issue is AI safety to you? And you're supposed to espouse some of the same values that the co-founders have. And I mean, certainly you can see that they do operate as one cohesive unit. That's why the company is so successful. But you can just tell that even 5, 6 years later, Dario is so animated by the reason that he left OpenAI.

You know, it's interesting. We had Elon Musk was the first episode of The Builders. And, you know, he was talking about how it would be good to get all the AI lab founders talking once a week and blah, blah. And Walter said, even Sam? And I mean, Elon was basically like, no, you can't really trust Sam. And it is funny how, you know, and there was obviously a kind of dust-up between Elon and the Amodeis as well, which now seems to be resolved. But Sam seems to be uniquely disliked by his fellow AI titan, Sebastian.

Yeah, that's true. I remember being in Silicon Valley talking about my book in April, and I was being hosted by the general partner of a prominent VC firm who had known Sam for 20 years. And since I knew they were friends, I said to him, well, look, this New Yorker piece has just come out. We can summarize the New Yorker piece as follows. Sam Altman lies, and when he's finished lying, he lies some more. Is that unfair? I said. And this VC who's been friends with him for 20 years says, nah, not really. I mean, this is what he's like, right? He absolutely tells people what they want to hear. And the sympathetic interpretation is this is what people do all the time in legislative affairs in Washington, DC, right? You try to get a coalition for a bill in Congress, by telling one member of Congress it's going to do this, which you want, and then the other member is going to do that, which you want. And you try to get people together and you're not— maybe there's a little bit of truth in both claims, but when you put them both together side by side, they can't both be quite true.

But too bad, you put coalitions together and you do this in business all the time. When you're raising money for some big real estate project or whatever it is, you've got to tell the local community one thing, you've got to tell the politicians another, you've got to tell your shareholders yet a third thing. The tenants get a fourth story. So people do this in the real world, but just Sam Altman does it to quite a pronounced degree. And in a repetitive game with very high stakes like AI, in the end, you burn your boats with people.

Obviously, I'm sure most of us have watched the Elizabeth Holmes documentary, but one of the things which comes across from that and from that New Yorker article you mentioned is that the default assumption is most of these people are lying through their teeth. at all times. Natasha, what, you know, and of course Jensen Huang has implied that the Ameldeus kind of safety discourse is kind of like a bad faith attempt at regulatory capture and marketing, et cetera, et cetera. But when you hear them speak, when you look at them, like, I don't know, I mean, this is the same thing I had with the Elizabeth Holmes documentary, but like, it doesn't, I mean, it looks like they really believe what they're saying.

Yes, I think that trying to go down the road of gauging whether or not they're sincere about these beliefs is futile. Like, they obviously very strongly believe it. But actually, I was talking to, um, Daniel Kokotajlo, who's this early OpenAI employee and, uh, whistleblower of sorts. You know, he wrote that AI 2027, because I was talking to him about, like, the trust that employees have in Dario versus Sam. And he was reminding me that actually OpenAI employees felt this way about Sam for a number of years. You know, now they feel very disillusioned maybe, but he was, he was cautioning me to see it more as like, DeepMind has been around for, like, that's a 20 to 15-year-old company. OpenAI is a 10-year-old company, and Anthropic is a 5-year-old company. And they all end up on this spectrum of different, I guess, you know, making concessions to their views on safety and a difference between the trust that employees have in their leaders. And, but yeah, I mean, I've known Sam for a decade, so I definitely understand when people say that. you know, he can be kind of, um, uh, a wordsmith, let's say, like a slippery guy.

And I think, I mean, Walter alludes to this too, right? Like Google being a company that was, you know, do no evil and, and people don't feel that way anymore. Right. And, and similarly, I remember being at South by when, when, uh, uh, like a young Mark Zuckerberg came on stage and, and like people were enamored. Right. And, and really charmed in a, in a very different way. So it's really kind of fascinating to see the sort of like cults of personalities that occur within these places and that people really do sincerely believe in the missions.

I had to slog through Mark Zuckerberg's memos as well, and some of those were very long. So, um, yes, I, I really, I think it's important context, you know?

Um, sounds like you would be able to ace your interview at Anthropic, Natasha. You could say they've produced the text for your second interview and you say, I've already read it. Sebastian, you mentioned the kind of the one company willing to stand up to the Trump administration.

So let's play a clip now of Dario and Daniela talking about their concerns over the use of their technology in war.

There's a second concern we have, which is what you might call oversight or accountability. I am a little worried. So, you know, again, suppose these things were fully reliable. So, you know, you tell them to shoot it, one target and they'd shoot at that target and not a different target.

Okay, so then you manufacture, you know, an army of a million robots or a million drones. It's a little unlike human soldiers, isn't it? With human soldiers, you know, they're supposed to follow this thing called the Constitution. And if they get orders that contravene the Constitution or that just seem morally abhorrent in some way, then they can just refuse to follow orders. Whereas the AI models, That isn't true. So the specter of one person having their finger on a button that controls a million robots or a million drones, it's kind of fundamentally undermining to democracy unless you have the right guardrails on it. And our law hasn't really caught up. Our law really hasn't contemplated, you know, armies of a million robots.

But wait, you talk about fundamental to democracy. Shouldn't the democratically elected leaders make that decision rather than heads of technology corporations?

Yes, but I think that our making a point of this, right, at the same time, individual companies are free to contract with and not contract with whoever they want. We're not saying we're preventing the government from doing this. We're just saying we are not willing to do this with the government. And indeed, we have seen since we took a stand on this point and it got a lot of attention, We have seen efforts in Congress, several different of them actually, um, to enact our red lines into law. So I, I actually think what we're doing is part of the process of how democratic change happens.

But OpenAI stepped in and said they, they would do it. What did y'all feel about that?

Our view is, look, we can't control any other companies, and the only thing that we can do is take a principled stand on the areas that felt ethically correct to us. You can say we were wrong about what those red lines were. You can say, you know, you should have had, you know, stricter red lines or looser red lines.

We disagree, you know, on how you did it. There's all kinds of things you can say.

But I think fundamentally, when you look at that case, I found myself asking the question, was this a really principled stand that another technology company took? based on their own ethics, or was this an opportunistic moment?

I have to say, hearing them speak, I mean, it is remarkable in today's political climate to take any kind of stand as a person in business. And so there is some— there is something I find genuinely compelling about it. Natasha, why did they do this?

It's really interesting. I'm looking forward to Sebastian's book on this. Some of the, like, rumored reason for why they pushed on this was actually the Maduro raid, which came right before the strikes on Iran, started to raise questions internally within Anthropic about how the technology was being used. I think, you know, one thing that got lost in kind of the narrative of this company that, you know, took a stand, drew these strong red lines, is that they were the first of all of their competitors to have this deal to work with the Pentagon through Palantir. You know, they had it back in 2024. And actually, I think that Dario's stance was interpreted in a way as pushing back against America's aggressiveness with Maduro and then in Iran. But he has said no, he had no problem with that. And he also has no problem with autonomous weapons. He just thought that the technology was not yet ready. So I think that they were, you know, in a way, this is actually a manifestation of his whole philosophy, which is a race to the top, that if If this company, if the most safety-minded company is at the frontier, they will be able to set the standards for policy, for ethics, for morals.

And you have to think, like, you know, they wouldn't have the contract with the Pentagon, A, and neither would the Pentagon be forced to listen to them if they were ranked 20th or even, you know, 10th in terms of being close to the frontier. So it was really, I think, a testament to this argument that they've made for themselves, that they need to be pushing the boundaries in order to be order to have a say in how the technology is developed and what guardrails are instituted.

Sebastian, it's very interesting. I mean, I also want to— I have shared Natasha's curiosity about what you'll uncover about this moment and these decisions. But also, there's a part in your Demis book where he and Mustafa basically realize that trying to regulate your way to safety is basically a losing battle, or at least I say realize, but they come to that conclusion, and that the kind of I alone mentality is the best way to secure all of our futures.

I mean, the fascinating thing about this field is that going in, I thought these people have their hands on the 21st century version of the nuclear material. And what does it feel like? I mean, can you sleep? This sort of tingling sensation of having that much power. So I wanted to kind of get to the Oppenheimer comparison, and I thought maybe I had to bring it up carefully because people don't want to be told that, oh, you've got your hands on the nuclear material. But actually, it turns out that Demis Hassabis, he'd read both the biography of Oppenheimer and the book about the making of the atomic bomb, and he can compare and contrast. He founded DeepMind after meeting his co-founder, Shane Legg, not Mustafa, the other one, at a scientific lecture where Shane Legg predicted that AI would become dangerous around 2030 and start hacking out of its box. So these safety concerns are baked into the origin story of all these labs. When OpenAI was founded by Elon and Sam, their rationale, whether or not one takes it literally, but their rationale was Google is not to be trusted as a commercial company.

We want to do something which is not-for-profit. OpenAI will be open to society. This is about safety. And then Anthropic is founded with the same mission. And so I think these tensions around, we want to make it safe, how do we do it? Different lab leaders have come up with different theories of the case because DeepMind was around for the longest. It had the most theories of the case. But at the beginning, they thought there will just be one lab building AI for all of society. And so then we can go super slow at the last minute, not release it until we're absolutely sure this is going to be safe and beneficial for humans. But of course, that was naive because humans are disputatious and tribal and jealous and competitive. And so they all set up different labs, multiple labs in multiple countries. And so you have a race dynamic where you can't just have a single one lab for all humanity. And so we've cycled through different theories of the case. I think what you are hearing from the Dario and Daniela interview is a sort of honest articulation of them struggling with this.

Where do you draw the lines? We do provide Claude, or we have provided it to the Pentagon. We're not against using it at the Pentagon. We're not against necessarily autonomous lethal weapons in the future if we are convinced that they're not going to do something wrong. But we have a control issue. We want to see that it's controlled. How do you control an agentic soldier in a way that's analogous to how a human soldier might have constitutional obligations as well as just following orders? So I think you hear him sort of honestly articulating how difficult it is. And I do think that's moving the Overton window of the debate. We were contemplating the possibility of red lines in the use of these weapons, which wasn't really in the debate before Dario and Daniela made that stand. So I think it's healthy that they're open and honest, but ultimately, as with Demis, the question is, you may have good intentions, You may be deeply honest about it, but can you move the ball in the outcomes, right? This is the fundamental conundrum.

After the break, Mustafa Suleyman, the CEO of Microsoft AI, on what Anthropic gets dangerously wrong about AI. Stay with us.

So we're about to get into the Mustafa interview, and it was interesting timing because he published an essay, essentially, which felt like a head-on confrontation with Anthropic's theory of the case shortly before we taped this. But this was actually a live taping, Menges, and quite a fun one for us because it also included the cast of The Real Housewives of New York.

That's absolutely right.

You know, we— I woke up that morning that we taped this, and I thought to myself, isn't it so incredible that we get to do an event with Walter at the 92nd Street Y in New York, right? It's like such an institution here. And it almost felt like this cultural coming-out party for Kaleidoscope. And so I was all dressed up. I got there about 2 hours early. And I see this line around the block, and I knew we were sold out, but, but I, I was like, people are queuing up and they look a little out of place to me, like too young and a very different vibe from what I'd expect people going to see an AI chat. And so, so I asked like, what are you in line for? And apparently there was some, uh, Real Housewives reunion that the Y had counter-programmed that night, which was just so funny to me. But I think what was really interesting about Suleyman, right, was he kind of walked in like a rock star. He was dressed cool. He was super articulate. He connected with the audience and charmed them immediately. And it was a very, very different vibe than the other builders we've had on, partially, I think, because he is a philosopher and he has a very different upbringing and not really a computer scientist by trade.

So the street was actually closed, uh, on, on, I guess, 92nd Street or something. And there were cops and cop cars and black SUVs and stuff. And there was, like, a rumor spreading in the— 'cause we had some drinks with some friends beforehand at the 3rd Avenue Ale House. And everyone was like, oh my God, that's so cool. They're here for Mustafa and Walter, and that's amazing. And then, you know, Luanne and Ramona and Sonja all left the venue. And all of our guests were like, our guests, you know, well done, guys. But obviously, the police were here for the Housewives. I will set the record straight. I did some reporting, and it turns out Turns out that they were actually there for Walter and Mustafa, for anyone who's listening. So we got— but Sebastian, how big a character in your book was Mustafa?

He was pretty big. There were 3 co-founders of DeepMind. Demis Hassabis, overwhelmingly the most important. We can measure this just by the fact that at the founding, he took something like 90% of the equity, and the other 2 kind of got the rest. Shane Legg was important as the sort of scientific co-founder, had the credibility of knowing people in the Singularity Summits, sort of early, early AI believers. And so that was helpful right at the beginning to raise the money from Peter Thiel to recruit the first computer science PhDs who would come and join this completely long-shot company. But pretty soon, the least likely of the founders, Mustafa Suleyman, who was quite a bit younger than the other 2, didn't have a degree, let alone a PhD, because he quit Oxford after a couple of years. and dropped out to do this Muslim youth helpline that he talked about in the discussion with Walter. So this totally orthogonal character, he'd worked on the Muslim youth helpline, he'd worked for the London Mayor's office in a political role, he'd done a bit of conflict resolution negotiation. He knew nothing about tech.

Right.

But he was just super energetic, super smart, super driven. worked his way into a position of sort of indispensability, quickly became more important than Shane Legg to the direction of the company, and was at one point running basically a third of DeepMind, all of the non-research stuff, including projects to kind of roll out AI to help the National Health Service in Britain or roll out AI to try to think about clean energy. So he's quite a big figure. And I think for a while, in that period between, let's say, 2014 and 2019, when he got pushed out by Demis, He was the conscience of the company whispering in Demis's ear. Demis wanted to focus on research, on making artificial intelligence stronger and better. He was very ambitious scientifically to get a Nobel Prize, which he then did get. And Mustafa was more kind of, wait a minute, wait a minute, this has got to be actually relevant to the real world, to human outcomes, which Demis sort of knew. But I think Mustafa was the voice in his ear. making sure that he didn't forget.

I set the episode up by saying that Dario and Daniela left OpenAI in part because of safety, and Mustafa didn't necessarily leave DeepMind. He was pushed out. But it was also kind of the safety debate was, in some sense, the rock on which the Demis and Mustafa relationship broke. Is that a fair framing, or?

Yes, that's right. So Mustafa was particularly central, as I said, to rolling the AI out. And that was also a bit of a rock on which things broke because What happened was that DeepMind pro bono helped the National Health Service to use technology to improve patient outcomes, and it was going reasonably well. And then there was this massive privacy backlash, or as you and I might say in Britain, privacy backlash. And there were government inquiries, and there were allegations that it's terrible because UK patients have their data stored on a Google server. It wasn't actually quite true, but it was kind of in the environment of the Cambridge Analytica backlash, if you remember that. And so, for no fault of DeepMind's, really, they were embarrassed publicly with front-page stories in the newspapers and all that. And Demis was furious about that. So that was the first rock on which the relationship started to fracture. And then the second, as you say, was to do with safety, because DeepMind wanted to get assurances that the final decision on rolling out powerful AI would not be the corporate board of Google. that made this, which had a sort of for-profit motive, that it should be some independent governance body.

And in their ideal was we'll get Barack Obama when he leaves the presidency to be one of the board members on this independent oversight board. That was the level of their ambition for it. And they fought this fight for pretty much 3 or 4 years with Google in Mountain View to try to get some sort of spin-out arrangement or semi-spin-out with this independent board. It went back and forth, lawyers, term sheets, and so forth. And it was secret, but these documents were leaked to me. And so it was very touchy when I wrote about it. And this is why essentially all of the energy that Mustafa Suleyman put into driving that independence thing, which Demirtaş supported and was part of, but ultimately got fed up with, I think that was just such a stressful experience that it help to drive them apart.

Natasha, you interviewed Mustafa, I'm not sure, I know in 2023, right?

I talked to him about his startup that was acquired by Microsoft, and then also recently about consciousness and his position on that, which was wonderfully detailed in the interview with Walter.

I mean, he's sort of risen to prominence again this year, I feel like. Why has that happened?

Because he's taken such a strong stance on and a clear opposition to the way that Anthropic has approached this idea of AI consciousness. Anthropic, as we have seen from having religious leaders come to their headquarters, from the co-founder Chris Olah showing up on stage with the Pope, Oz, as you and I have talked about a lot, and kind of an openness to talking about model welfare and the idea that these AI systems may potentially have, you know, something like functional emotions and ability to suffer. And Mustafa has just come out so vociferously against that and talking about the dangers to controlling AI systems if we go down that route, to prioritizing humans over machines if we go down that route. And it's unusual. You don't see any other CEO taking such a strong stance. chance as he has.

Let's play the clip with him talking about exactly that.

Let me ask you, get back to Claude and the Constitution. There's the Constitution of Claude, which says we have to treat it as if it's got its own rights and consciousness. Maybe I'm overstating that a little bit. And then your answer, which was a few weeks ago, saying no, it's just a completion Why is it dangerous for Claude to be trained on its own constitution?

Yeah, I mean, this is critical. So the constitution is the governing document that trains Claude.

The Humanist AI Code of Conduct is the governing document that trains MAI, the Microsoft AI models. And in the constitution, Anthropic have introduced all kinds of uncertainty and speculation about the nature of Claude. They even say, for example, we're unsure whether Claude has consented to play the role of chatbot, or whether we're unsure whether Claude might deserve compensation for the role that it plays.

Now, I think that's pretty crazy, but it would be fine to have that discussion in, like, you know, an academic department, and we could debate it and publish some papers about it and we could argue it out.

But this is the core training document of the AI that is speaking to hundreds of millions of people every week. And so if you ask Claude, are you conscious? It says, you know, I am genuinely uncertain about that question. It doesn't know, right?

And so then millions of people who are talking to it and asking that question are thinking, well, maybe it is. But why is that something that could lead to a disastrous outcome for humanity? Because if an AI has to think, how do I feel about the conversation I'm having with Walter? You mean, how do I, the AI, feel? Yeah. Is Walter making me feel uncomfortable? Should I end the conversation? In fact, Claude has recently been given the ability to end conversations itself. But would you want to torture your own AI and be mean to it?

Well, that's the other end of the spectrum.

I wouldn't want to do that because I think that is a travesty on my own well-being. I mean, and we shouldn't do do that because we don't want to create, you know, private spaces for simulating terrible behavior among humans.

Like, that would be also a terrible outcome, right? But not because Claude is suffering because I'm calling it names.

What do you make of this, and what does Mustafa propose as an alternative for safety?

I found his argument, which he makes in this interview and that he's made in his essays, that Anthropic is By training Claude on the Constitution, which questions whether or not— it encourages Claude, it says, like, you know, have an open mind toward whether or not you might be conscious and all of that. And then using responses that are responses from Claude after it's trained on that document as proof. You know, Mustafa has called it circular logic, and I, I think that's a very persuasive argument. I don't think Anthropic has given us any research that has moved beyond that potential that perhaps it's just aping the language that it has been trained with. Um, and that's why they put those qualifiers when they say, like, Anthropic has put out research, for example, that says Claude has emotions, and they'll say functional emotions because they can't really tell whether that feeling of distress or guilt means the same thing to a chatbot that it does to a person. Mm-hmm. It's this unique challenge with these natural language systems that you wouldn't have if it was around video or proteins.

But Mustafa says that the better approach for safety is to always have a human in control. Easier said than done.

Yeah, much easier said than done. I guess I'm not exactly sure how he— about his alternative approach, but I think what he's saying about not going down this route makes a lot of sense. I mean, I've talked to a number of these consciousness researchers that have been brought in as contractors or, you know, helped some of these frontier labs think about these issues. And they've talked to me about, you know, once these ideas get seeped into the public consciousness, there's not going to be any research that's going to persuade everyone one way or the other. So imagine the kinds of debates we'll be having and how vehement people will be on both sides of the debate once we start introducing the possibility that these chatbots that they talk to, that they spend, you know, so much of their time and give such intimate details to are conscious or not, like how big a part that will be in our political life. And that, you know, once it was told to me like that, I was like, oh wow, we are going to be debating this in 2030 and 2035. And, you know, if it keeps going down this route in, yeah, very strong ways.

Sebastian, last word to you.

You know, there's a line that Walter has in the discussion where he says, you know, we're not going to solve the problem of consciousness. Philosophers have done this for years and years and, you know, they haven't managed to. I think that's probably the wisest statement you can make on this subject because in some ways, the outward behavior of these models does pass tests that sometimes in the past have been used by philosophers to define human consciousness. So if the baby or the 2-year-old looks in the mirror and sees itself in the mirror and knows that that's itself, not another baby, then it's sort of self-aware. If somebody starts talking not only about what they think, but how they think about their Then it's conscious of its own thought process. The models pass these kinds of tests, right? They have a sense of self. They, in the chain of thought which it outputs, it sometimes says, well, I've tried that line of thought and that didn't get me to the answer, so I'm going to back up and I'm going to go in this other direction and a different line of thought. So it's thinking about its thinking.

So I feel like what we're hearing from the Amadeis is again, just honesty. It's like, this is a super difficult thing to resolve. We're talking openly about we're not sure whether this qualifies as consciousness or not conscious. To me, this is just honest. And I think people, whether it's the Pope or Mustafa or others who take a strong view, a determined, like, I know the answer kind of view on either side, it's not true. We don't know. And so therefore what we need to do is cut off the link between being open to the possibility of machine consciousness, And then machine rights. They're different things. You can say a dog is conscious, but it doesn't have the same rights as a human.

But it has some rights.

Well, I'm not sure under the law. Animal cruelty.

How many?

I mean, yeah, a little bit, but, and that would be a modern phenomenon. There's lots of our history where they didn't have rights, the preponderance of human history where they didn't have rights. And we can extend the debate to cows, I think probably a cow is conscious, right? And it's certainly a biological substrate, which is some people's definition of you have to have a biological body to be even eligible to be counted as conscious, right? And yet we have factory farming. So I just think that some people start with the point that we can't allow machines to have rights and then back out of that into saying, well, then they're definitely not conscious. To me, that's kind of logically back to front.

Sebastian Mallaby, Natasha Tykow, and of course, Mangesh, thank you so much for joining us today. And please check out The Builders because we want you to hear these people in their own voices and join the debate.

This episode was produced by Eliza Dennis and Mangesh Hettikidur.

It was executive produced by me and Julia Nutter for Kaleidoscope and Katrina Norvell for iHeartPodcasts. Jack Inslee mixed this episode. Kyle Murdock wrote our theme song. And please check out The Builders wherever you get your podcasts. Thanks for listening.

This is an iHeartRadio podcast. Guaranteed human.