Scaling Laws: Daniel Kokotajlo on AI 2040: Plan A

So, besides the end of the song and the new "Bundestliga" song, as far as "Türstet" is concerned,

"Mutigia, I am mental," and "Sport Director". Because all the games are played, then it is even more like the kick-based side.

“And if there is now kick-based, what does that mean?”

The kick-based is the fourth fan of the "Fußball Manager". But that's the principle. It would be easy, you are surrounded by a single league, you can even catch up with the rules, or you can then use the "Bundestliga" profile in your card.

So, at the time, that you know "Fußball Wissen" and "Under-Bewise" the counter-unloaf of the "Bundestliga" song on the tag is "Enner August", which means that it is surrounded by a single league, and since "Fondag 1" on "Mitterball", the kick-based side is just "Downloading" and "Loslegen".

Good kick, and "Fishmas". On "Tooks Cuttings", "Check".

"Internet" on "Melden", "Check", "New Address" and "Check".

And then "Strowman Bitter". Between "Kist" and "Meldon" and "Möbel" on "Bau Club" the "Strowman Fatak" channel. In addition to "Automatic in the Grundfersorgung", they often take care of it.

With "Octopus Energy", "Wexels", you complete so-called "Paisen". Now, on "Octopus Energy" the "Evexten" and with the "Bonus Code", "Octopus 1-1-5", a 15-hour "Wexelbonus-sichan".

It's the Law Fair Podcast. I'm Alan Rosenstein, associate professor of law at the University of Minnesota, and senior editor and research director at Law Fair. Today, we're bringing you something a little different,

an episode from our new podcast series "Scaled Lost". It's a creation of Law Fair and the University of Texas School of Law,

where we're tackling the most important AI and policy questions.

From new legislation on Capitol Hill, to the latest breakthroughs that are happening in the labs. We cut through the hype to get you up to speed on the rules, standards, and ideas shaping the future of this pivotal technology. If you enjoy this episode,

you can find and subscribe to Scaling Laws wherever you get your podcasts and follow us on x and blue sky. Thanks for listening.

“When the AI overlords take over, what are you most excited about?”

It's not crazy, it's just smart. And just this year, in the first six months, there have been something like a thousand laws. Who's actually building the scaffolding around how it's going to work, how every day folks are going to use it?

AI only works if society lets a work. There are so many questions have to be figured out, and nobody came to my bonus class. Let's enforce the rules of the road. Welcome back to Scaling Laws.

The podcast brought to you by Law Fair and the University of Texas School of Law that explores the intersection of AI, policy, and of course, the law. I'm Kevin Frazier, the director of the AI Innovation and Law program at Texas Law and a senior editor at Law Fair. Today, we're joined by Daniel Cocatello, former open AI researcher

and executive director of the AI futures project. Daniel alongside many co-authors, penned a policy road map titled AI2040 that details their recommendations for how to delay superintelligence. It's an example of what they refer to as scenario scrutiny,

testing the ideas of a policy proposal by thoroughly outlining how it may work in practice. In this case, they envision the U.S. and China adopting a posture of tremendous transparency around AI research to prevent the racing dynamics that some fear may lead to catastrophic outcomes.

We kick the tires on that proposal and dive further into their ideas during this fascinating episode. To get in touch with us, email Scaling [email protected] or follow us on X or Blue Sky. And with that, get it up for a great show.

Daniel, welcome back to Scaling Laws. Thanks for having me. So you wrote AI2027 and you thought that was so much fun getting the entire AI community dive into and probe my policy ideas and my assumptions.

Why not do it again and write yet another report,

“although of a different sort with a different goal and pen AI2040?”

So let's just start there for those who were living under a rock and missed AI2027. What was that? And what is AI2040 and how is it distinct? Yeah, so AI2027 is a scenario forecast.

So it is a scenario. It goes year by year and month by month, starting when it was published and lays out a concrete possible future in great detail with like a companying stats that change as you scroll down and about a page or two

about each time period and eventually it's going month by month. So that's quite a lot. It's about 50 pages or so total. And it's a forecast in the sense that it wasn't just like a story

It was our best guess at each moment of like what the most likely

continuation would be for the development of AI. Planning out how you thought AI would progress to 2027. That's right.

“Although notably because of how important we think AI is,”

it's also just a projection for the whole world. Like, you know, we, yeah, like because AI becomes so important, everything else just sort of gets steamrolled by it. And so the history of the world becomes the history of AI.

So that was AI2027 and spoilers. In AI2027, the company succeeded

at automating AI research in 2027.

And this causes what you might call an intelligence explosion or the singularity. And we sort of like walk through what that might look like according to our best guess through 2027, 2028 and 2029. Now it blew up bigger than expected.

So it sort of went mega viral, which of course was good to hear. We think it was a, we think it helped advance the discourse. And I guess building on that success, we thought we would try again with another big scenario. But this one, AI2040 Plan A, is not a prediction.

It's a recommendation. So it's another big scenario that starts in the present and goes year by year.

But it sort of deliberately a bit optimistic

about the choices made by the government. In particular, it just sort of assumes the government does what we recommend that they do. I love this as such. This is a, you know, if only all policy ideas we could just assume

they would come to fruition. Yeah, basically. It's a vehicle for conveying our policy recommendations. And we think it's a valuable way to do it because we call it scenario scrutiny.

We think that a lot of policy recommendations or a lot of, especially a lot of ambitious visions for how to handle AI in general, rather than like specific bill, text or whatever. But a lot of, a lot of plans fall apart.

If you look at them too closely and you like, game out what it would look like to actually implement the plan and what the actual expected consequences would be. So if you apply scenario scrutiny to your plan, oftentimes your plan falls apart or at least you notice,

you know, issues with the plan that you hadn't realized before.

“So we think it's actually a very important thing for anyone with a plan”

for the future to be thinking, to be applying scenario scrutiny to that and, and, and, and, and gaming it out. Um, for that matter, uh, anyone without a plan for the future should also be doing this, like, like, like, you can't just say, like, oh, what model through is like, okay, well, how are you gonna model through?

And like, what would modeling through look like? Uh, have you read AI 2027? Perhaps this is what muddling through would look like. AI 2027, it doesn't end very well, you know? Um, so, so, uh, in general, we think that people should be gaming out possible futures,

as best as they can, uh, both the futures that are gonna happen by default, that seem most likely, and the futures that they're trying to steer towards, uh, or recommend. Yeah, and I really recommend that folks who missed, uh, the interview I did with you and, uh, Eli on AI 2027, go back and listen to that episode or better yet, go read all 50 pages, and then turn to AI 2040 because this idea of scenario scrutiny.

“I think is under-appreciated as you're recognizing Daniel that more folks need to be doing this,”

because it's easy to go out and write the piece of, you know, x idea for AI and just kind of drop the mic and say, well, I did the thing. I wrote the blog post and policymakers should do what I want now. But for you all to have the epistemic humility and the invitation for scrutiny, I think is a practice that others should follow because we need to walk through how would this actually

work out in practice, and that's a far more difficult task that requires a lot more intellectual rigor, and so I applaud you all for outlining and leaning into this approach of scenario scrutiny, and I hope others follow suit, and I can tell you Daniel, you've inspired me such that I will be assigning AI 2040 and inviting my students to do this sort of scenario scrutiny on some of their ideas. So stay tuned, you may have some Texas longhorns coming for your next scenario planning.

But before we get through AI 2040 because there's so much to unpack about that timeline, just in case folks need a bit of a refresher on some vocab. Let's do two quick key concepts that everyone needs to understand here. Number one, what is AGI? Number two, what is ASI? Or super intelligence, and then what is recursive self-improvement?

AGI is a deliberately vague term. I think some people say it's kind of meaningless, I don't

“think it's meaningless, I think it's a deliberately sort of vague term, it basically means”

AGI stands for artificial general intelligence, which means AIs that can do things in general, like a single AI agent that can do a wide range of tasks, much like how a single human can do a wide range of tasks, rather than just being a particular piece of software that does a particular thing. It's vague because there's people try to give more precise definitions in that, but then they disagree about what they're more precise definitions to be.

So for example, some people would say, we already have AGI. After all, look at CloudFabel, it can do a very wide range of tasks. And then other people would say, no, no, it's not true AGI yet, because look at all the things that can't do. And I say whatever, we don't need to arbitrate that dispute. The point is, it means a very wide range of tasks, and we can argue about whether it's already here or there, but I'd say, let's not argue about it. It's a deliberately vague term,

that refers to basically the kinds of AIs that we are currently building, and better and better

versions of these types of AIs. ASI is a more precise term, artificial super intelligence. And that is supposed to mean AIs that are better than the best humans at everything while also being faster and cheaper. So that we definitely don't have yet. You know, Fable is not an ASI. It might be better than the best humans at some particular types of tasks, but it's definitely not better than the best humans at everything. But, you know, the companies are trying to build

“super intelligence. They say, so on their website, it's not a secret. They're just stuff. They're”

companies, and everyone seems to be going in on this super intelligence game. We know that there's even companies just called safe super intelligence these days. We're not hiding the ball that many

are chasing this ASI angle. Yeah. Yeah. And then finally, recursive self-improvement.

That's right. So, since the dawn of time, people in AIs, people, you know, AI researchers and people talking about AI have, have noticed that if you had AI systems, it can automate professions or automate entire, you know, large amounts of work. One of the things that they would naturally be applied to is automating the research process to make AIs. And obviously, this should accelerate the research process. So, that's recursive self-improvement

is the idea that like you can automate the AI R&D process itself, and thereby have AIs autonomous

“ly doing the research, doing the experiments, analyzing the results, writing the code, fixing”

the code, you know, you know, training, doing the training runs. Basically, the whole process.

Everything that anthropic and opening I currently doing, automate that process, it'll go faster. You know, how much faster nobody knows, but that's that's recursive self-improvement. And as many have pointed out, it's already starting to happen. You know, like AIs are already writing huge amounts of code at anthropic and opening I. And but it hasn't like they haven't fully automated. Yeah, research. Okay, so linking this all together, we're talking on August 3rd,

2026. We have some vague sense that we are near AGI or AGI adjacent. We are in the orbit of AGI and folks can dispute whether we've achieved it or not, so on and so forth. And already, you can have a lot of folks, especially in light of what happened last month in July, concerned about loss of control. The idea that we are seeing AIs systems being able to break out of their testing environments as we saw withinthropic and open AI and go and complete some degree of

hacking of external entities in a way that was intended by the developers. That's a grave concern that you highlighted as your number one concern. More generally, this idea of loss of control. And if that's occurring with AGI, then if we had something like recursive self-improvement leading to superintelligent systems, then that loss of control could be vastly greater in terms of consequences, mainly negative consequences. And so AIs 2040, if I'm correct, is a sort of road map

to delaying the achievement of superintelligence such that those loss of control scenarios are less likely or less consequential. Is that a fair high-level summary of what sort of policy you're trying to develop with AGI 2040? It's fairly fair. The way I would put it is that we have a list of problems that are associated with building superintelligence and we are trying to solve the problems on our list prioritizing them accordingly. Number one is loss of control. Number two is

concentration of power. Number three is world war three. Number four is jobs. And number five is

those outcomes from world war three to loss of jobs to loss of control is delay superintelligence. Don't do this crazy person self-improvement thing. Don't automate the AIs recursively self-improve as fast as they can. That's really dangerous. It's also a power grab. Even if you somehow think that that's not dangerous at all and that you're going to be perfectly control of the AIs even as they become vastly different from the initial AIs that you handed off to.

And even as everything goes faster and faster, they get smarter. Even if you're completely fine about that, it's a power grab. Like if you're right and you end up with your superintelligence that are perfectly obedient to you, well now you are kind of in a position to have huge amounts of power over everybody else. You might be in position to take over the country for example. Like maybe your giant army of super, you being the CEO, you know, maybe your giant army of

superintelligence will allow you to puppet the United States government. And you know, like this you know, so there's a constitution of power problem as well. And that's like number two. And then, you know, World War III, well, what do you think China and Russia are going to think about your recursively self-improvement superintelligence might they be concerned that that you're going to use your superintelligence to maybe undermine their governments or assassinate

three meters or, you know, cause revolutions to happen in their countries? Yeah, they might be concerned.

“In fact, I think Dario, the CEO of Enthropic has even said in one of his blog posts something”

to the effect of yeah, we're going to do this once we get superintelligence. If we go back to specifying that the goal here is making sure that we are preventing the occurrence of a number of maladies from perhaps achieving superintelligence too quickly, whether it's drawing the iron of our geopolitical rivals or operating away that the rest of our civil society institutions haven't prepared for that

critical infrastructure isn't ready for what is the policy pathway you see in AI 2040 to achieving

that delay. And we can kind of go through year by year or milestone by milestone that you see as especially important. So the first year 2027 slash 2028, you walk through in particular Congress taking action here. So why don't we start with how you think Congress may begin to get involved in this ballgame? Yeah, I should say as I've been on the side, one of the possible regrets I have about how we set up this scenario is that we basically have nothing important happen

“until a international deal with China is made. And I think actually realistically we should have”

more like serious domestic, domestic regulation. And then because I think that you're more likely

to get the deal going with China if you've already started to implement basically a good version

of it domestically. And also I think it might be easier to get something done domestically. Like the the China hacks will say we can't do anything domestically until we make sure China's going to do the same thing. But I just don't think that's politically realistic. I think actually it's a bit around and like you're more likely to get the China thing going once you have the domestic stuff. And there's a lot of demand domestically for regulations independently of

what China's doing. But in our scenario there's basically nothing happening until they do a deal. There's some minor stuff. And so we talk about like, you know, AI transparency act of 2027, you know, a bunch of like incrementalist reforms, which are good. But nothing that like seriously

changes the picture. In our administration and in our scenario, 2040 is when they ultimately build

super intelligence. But 2030 is when it would have happened if they had continued going as fast as they could. So that's a bit of a difference from AI 2027. Because we were uncertain about timelines, we want to talk different scenarios to like have different years in which it happens by default. And so we already did 2027. And we're going to do 2030. I should say 2030 is actually a bit of a long timeline scenario for my perspective. I think it's going to take less time than that to get

to super intelligence. But you know, maybe it may be it'll take that long. And the my co-author who led this actual this plan A project Thomas Larson 2030 is his median. So it was like his central future. What was this one? Okay. So the idea is we have all else equal if there was no intervention seen something by around 2030 was the assumption for this AI 2040 investigation in terms of when we would achieve super intelligence. So first you get Congress passing

“this transparency act that's of minimal significance. But really what matters, but it's it doesn't”

really change the situation. We're still in a race. Okay. Yeah. And then you all suspect that by 2028,

dominates the presidential election leading to the administration to really champion and build off

“of that transparency act. And what happens next? Yeah. And again, we're not confident in this, but”

if you just sort of even if you think the exponential trends are going to like slow down, you still get some really crazy numbers. You know, things like the AI company is spending more on data centers than the entire US military budget or something. And like the world's biggest companies being AI companies by like 2028. So that's that's part of why we were thinking, yeah, it's going to be a big issue. So we have this flow chart, which perhaps if you're doing a video

version of this, you could like put up on the screen. And this is the flow chart of like the policy options for that are being debated by the presidential candidates and the president in 2028. And then we sort of leave it ambiguous like who wins the election, but the point is whoever wins the election, they're going to implement the policy that they argued for in 2028 in those debates. So here's like the policy options. And the flow chart starts with, do you want to race through

“the intelligence explosion having the AI self-improve and putting them in charge of more and more”

things faster than China can? You know, and then if you're if you're like, no, that's crazy, then you get to this, this branch that's like, well, maybe we should make a deal with China, so that we don't have to do this crazy race. But if you're like, yes, actually, that's good, or yes, we have no choice. Then you get to this other category of options. So we have the options of, you know, plan D, which is the default plan C, which is, I've got it up right here.

Oh, wow. You have like the simplified mobile version. Yeah. So if folks who are listening right now, you can either go to AI-2040.com or you can watch the handy-dandy YouTube video where we have Daniel giving a live explanation of the different paths the AI-2040 authors see available as of 2029. So which path might the US take? So D, you explained Daniel was, hey, we're going to race forward. We're not going to change course at all. That's the default. We're not

going to slow down as you want to race here. We're not going to slow down at least a bit for safety and governance. The answer there is just nope. And that is plan D that you have on there.

Plan C. And to be clear, up until recently, this was basically what the companies said

“they were going to do. I think that after we published AI-2040, there was this, there is this event”

that is very encouraging to me, which was a thousand employees at that company signed the petition basically saying that the government should have the ability to slow down the pace of the AI development and and should coordinate that internationally with other governments. And then open AI nonprofit like endorsed it basically or they said like yeah, this is reasonable. So that was really encouraging to me because basically they were like, how about not plan D? And I want to talk

about that in more detail in a second once we finish getting through this on this timeline. So time path C or plan C was yes, we will slow down for a little bit for safety. And plan B is, yeah, so let me try to explain. So plan D is race as fast as possible. Plan C is slow down a little bit for safety and for other reasons, you know, to handle disruption or whatever, whatever reasons you want to slow down, you're slowing down a little bit.

But it's only a little bit because you're still trying to make sure that you have a lead over China. And so you're going to, you know, right now is a lead over China between U.S. companies is something like six months. And so it's like, okay, you're slowing down by a few months. Yeah, you know, a few months less than maximum speed. And then plan B is like that except that you

also take aggressive actions to slow down China. This is called, you know, fight China basically.

You might sabotage the AI program, for example, or you might like a more modern version of this would just be like really strict export controls. And then like a more intense version would involve cyber sabotage. And then an even more intense version would involve kinetic strikes. And so there's a spectrum. But the point is, plan B, you're not just slowing down yourself, you're like trying to slow down China against their will. So those are the options that

don't involve making a deal with China. Or at least the options that, you know, there's actually more options besides these. For example, you could just unilaterally do the good thing and then hope that China will also do a good thing, you know, and that's like not even on this table. You're saying delaying super intelligence. So in theory, we're just delaying super intelligence. And they agree to and say, yes, we will follow the US there. Okay. And then to be clear, there's

also like, you can talk about delaying it for its own sake. Or there's different kinds of delays. And we're kind of like lumping them all together here. One kind of delay is where you just

where you impose some sort of guardrail or regulation for the sake of something like for the sake

of transparency or for the sake of safety. And then as a side effect of that guardrail or regulation, it prevents the companies from going at maximum possible speed, you know. But and then sort of lumping those together, you know. So we have myriad possibilities here. Either we don't really slow down or we're not really engaging with China. Maybe we engage in our own sort of delay that China then leads into or follows. But then you see a world in which we may make a deal with

China. So what are the contours of potential deals with China that you see as being particularly efficacious for your policy goal? Yes. So there's a whole range of different possible deals and unfortunately we can't pack them all into five in into this into this little thing. But we thought we would highlight too. So one possible deal is plan S for shut it all down and there's different sub variants of it. But then the other possible, the other another deal that is our recommendation

is plan A. And it's hard to sort of summarize planning slogan. Maybe something like verifies slow down or like transparent cautious scale up or something like that. So we can get to that

second. I should mention of course these five options were arranged by basically speed.

You know so plan D is maximum speed. See is like a little bit slower. B is a little bit slower still because you're also slowing down China. And we've got some like basically we have estimates and our supplements look like how much slow down each of these plans would involve. Yeah. And obviously plan S is like maximum slow down. So yeah. So for plan A and I want to spend a lot of

“time diving into why you think this is so important to be discussing right now and how recent events”

have shaped your thinking. So I'm going to ask you to go kind of quickly through plan A in particular highlighting the call for mutually assured compute destruction and the sort of complete transparency you think will be necessary to realize the goals of plan A. So just really leaning into those two policy prongs. Why do you think that may be the path forward for plan A? Yeah. So okay. The high level thing that we want, well I mentioned previously the goal to want to avoid

loss of control, we also want to avoid concentration of power. That's very important. We'll forget the others for now. How are we going to do this? Well, it's very important that the U.S. and China be able to verify compliance with whatever agreements they make because they don't trust each other. And so if they can't verify compliance, they might cheat. But if they can verify compliance, then you can't cheat without the other side noticing you cheating. And so you know okay.

“So the verification is really important for whatever doing make and then in terms of the priority”

of a deal, we want to avoid doing this crazy intelligence explosion stuff. We want to proceed cautiously towards super intelligence. We also want to do it in a way that doesn't concentrate power. In fact, we want to sort of spread out the power. From the perspective of every other country besides the United States, power by default is about to concentrate immensely in the United States because all the major AI companies are at U.S. And there's like a few follower AI companies

that are Chinese, but that's called comfort to like India. So power was set to concentrate massively by default. And we want to like push against that to a large extent. So what we want is AI progress to proceed, but cautiously and not in this sort of crazy race. And we wanted to be the case that there are multiple companies across multiple countries catch up to the frontier. So that power over AI is spread out over multiple companies and multiple countries.

“And there's this one thing that I think helps with a lot of this stuff and that's total research”

transparency. So that's in some sense the foundation of the deal is the U.S. and China and whatever other countries get involved agree to have all of the new AI research and all of the new AI training

happen on totally transparent data centers. So basically they regulate the supply chain. We get

we get all of the countries involved in the supply chain, hopefully, in on this deal. And then all the new chips that are produced get shipped to new secure transparent data centers. And at each of these data centers, they'll have monitors and auditors from the U.S. and from China and maybe from Singapore and Switzerland. And like whatever countries are involved in the deal to make sure that all the activity on these new research data centers is being logged and

published basically. The clear emphasis there is for folks who are not as well steeped in the

model and you get a response back versus training when you're actually trying to develop

some new AI system. We can see when a lab is using that compute the difference between those two tasks. So in theory, if you had China's data centers in Switzerland as you all throw out there or

“is it the U.S. data centers that are in Switzerland? I believe it's the Chinese. We'll get to”

that in the science. That's a destroyability thing. Okay. Talk about that later. But wherever the data centers are, they would have inspectors from all of these countries there. You see? And the idea is that you would be able to distinguish between training, which may suggest, hey, you are racing toward super intelligence in a way that the rest of the world isn't ready for versus, oh, hey, you're just using this for inference and we're

okay with that. That's going to be acceptable. Yeah. So we think that on a technical level, it's possible to set up a data center that makes it extremely inefficient to use it for training.

For example, if you just have limits on the bandwidth connecting with the GPUs and so we basically

have two types of data centers in our proposal. There's the inference data centers which just serve customers much like today. Like you have your chatchipity question. It goes to chatchipity. It answers it comes back and that stuff is not surveilled anymore than it is today. That stuff is still private. But then you have your training data centers where the research is happening and where the training of new models is happening and that stuff is just like published. You know, all the

activities published so that the whole world can see how each model is trained and see the whole

“pipeline. And that's really good in a bunch of ways. First of all, if you want to then have additional”

rules for like what types of AI are safe to train and what types are not, how are you supposed to enforce that those rules are being followed? Well, if you can just see all the training, then you can just like see who's following the rules and who's pushing the gray area boundary and can just see everything. You know, so it's really great for verifying and making sure that we don't just have to trust that like that they're following the rules. Secondly, it's really good for advancing

alignment science in general, right? Open science. It's great. The scientific community can see how the AI is trained and then they can like argue about whether the training, this part of the training process caused, you know, this misalignment incident or whatever and this is like they have all the information and that's really good for like accelerating the science of understanding how AI is work and how to align them. It's also really good for avoiding these biases,

“where like for example, as we've seen with the human face incident, you know, opening AI is kind of”

reticent to like publish details about what happened and they're sort of like dripping, dripping a few details out to the public. But like if only we just could see the whole incident, then like there would already be a much more rich discussion happening about it and so forth. So that's one thing. Another thing is that it's it's it's concerning to rely on a government regulator for these things because the government regulators, well, they're just a few people and maybe they lack some expertise

and maybe they can be, you know, a bot or captured to some extent. And so it's nice if you have

this sort of like third-party ecosystem of like all these other works that like can be making

judgments about what's safe and what's not as well. And if you just publish all the information, then you sort of get that for free because everyone can see the information. Everyone can make judgments about what's going on. There's this big open conversation about like, is this particular type of training that's going on that they just started implanting good or not? Is it safe for not? What about this new line of research that's happening on this data center? It looks like they're trying to make

neuralies. Are we cool with that or should we maybe like try to get them to stop because maybe neuralies would like invalidate a lot of our safety cases. You know, these types of things is just like happening real time out and out and the open instead of relying on, you know, some regulator that like meets with the company to like notice that what they're doing is concerning and then like realize that it's concerning and then like try to get them to stop, you know.

And so it's really interesting too to think through the multiple layers of concentration and power that you're discussing here and thinking through having the option of a global universe of scholars looking into these matters as opposed to the status quo as you flagged right? We're talking again in early August where we're still waiting for the quote unquote independent reports that meter and redwood research are going to do about the breakout scenario that occurred with open

AI. When that comes with what level of transparency, which with what level of insights we don't know to your point also, even if there were a government regulator, we wouldn't know necessarily what information would be disclosed and so much of this in terms of AI going well will depend on the science of AI for lack of better phrase progressing and that's obviously going to benefit from diffusing that power and diffusing that knowledge as widely as possible and so that to me does seem

and like basically apply unequal standards to the companies that the dislikes. That would never

be easy to notice. It would be easy to notice if that's happening if you can just like see all the activity that the companies are doing and then you can like see like oh hey like this company is being punished and this one isn't but like it seems like the activity they're doing is pretty similar you know. So so it helps reduce that type of overreach of that type of power transition as well but let me let me talk about the more big effects on conservation of power. So

so far I talked about the benefits for like having good safety focus AI regulation that achieves its actual safety goals and also the effects for like advancing the science of AI alignment.

“On the contrary to the power side you know what are the most important things we can do to have a”

world where power does not concentrate extremely due to AI well first we need to avoid AI monopolies which means we need to have multiple countries with frontier AI programs because even if there's multiple companies with frontier AI if they're all in the same country then that's a monopoly waiting to happen you know that's like all it takes us to the government deciding that it wants to

nationalize today or something and then now one may never have any ideas you know so so you

want to be the case that it's spread out or multiple countries and ideally you want to just be more frontier AI companies rather than fewer you know and then also separately you want to be transparency into what those giant armies of AI are up to and how they're trained so that entities that don't directly control giant armies of AI's have a say have like more of a say and more oversight into what's happening like even if you had like you know 10 frontier AI companies

“spread out over 10 different countries if it was still like the situation today where there's like”

very little transparency into how the AI's are trained or what the AI's are being told to do then you kind of end up in a situation where no one else matters except for those 10 AI projects and their leadership you know like for example the Supreme Court of whatever you know say there's like a single say there's a U.S. AI program and like the the president isn't in charge of it or whatever how is the Supreme Court supposed to like give oversight over the president and whether

he's doing something unconstitutional with his AI's if in today's world like they don't even know what the AI's are up to they don't know how the AI's are trained you know Congress doesn't know either so anyhow basically you want to spread out that we don't have voidemanoply and you want to be transparency into how they as we're trained and what they're doing and the transparency thing that I mentioned before helps with both of those things because because we are doing the

total research transparency that's basically sharing the core algorithms and the core recipes with the world which is going to help other companies catch up so it's like directly fighting against this monopolizing force and then of course the transparency just while there you go transparency

“so it makes it much harder for companies to abuse their power an example that I think I like to”

talk about is secret loyalties or you know hidden agendas so this is on the on the spectrum of ways in which a company can abuse their power this is like the most egregious way and there's of course a whole spectrum that's less egregious and more nuanced but but just to talk about this

one a little bit imagine a situation where you know a chatbot that's used by a hundred million people

in in America has a secret agenda and it's like secretly trying to push the political views of company leadership or perhaps secretly trying to help you know their favorite candidate when the election or something like that they could have quite an effect on such things because you multiply by a hundred million people that sort of like subtle you know subtle all propaganda or whatever that the chatbot is doing could have a big effect and it's worth noting already that there's

research from a Gillian Fisher at the University of Washington showing that just subtle engagement with a subtly biased AI chatbot can already start to change the views of users and it's worth noting also that we know these tools in some context are more deferential to the companies that created them when you ask questions about how should we regulate this company as opposed to that company there is a sort of self referencing bias there and so in terms of the sci-fi vibes that

people may be getting this is being empirically documented already in the literature and so we could go down the the need for transparency for many more minutes and I'm glad we've come to hear I do want to make sure we leave time for me to really throw you know the hard balls out there but let's transition to the fact that you all have also outlined this concept of mutually assured compute destruction why is that necessary we've we've had this for lack of

better phrase kumbaya of total transparency the world is high-fiving we have a hundred AI frontier companies across the globe doing cool stuff why do we need this concept of mutually assured

components but that would be risky or more risky than with the component it's kind of a it's a

“fail-safe mechanism and the reason why is imagine that you imagine that the deal breaks you know”

imagine they've been doing plan A for a couple years and all these new data centers have been constructed and now there's an order of magnitude maybe two or two orders of magnitude more compute in the world than there was at the time that you initiated the deal and then for some reason there's a conflict over Taiwan or something and then the deal breaks down and they stop being transparent to each other and now we're not transparent to each other they can't trust that

they are not racing to super intelligence anymore so probably they're going to start racing the super intelligence and now you have a race to super intelligence happening except it's going to be even faster because of all this compute that's built up like according to our you know our estimations it might take something like a year you know a few months uh to go from fully automating AI research to super intelligence obviously there's a lot of uncertainty about that but what we feel confident

“in is that if it would have taken you know X length to cross that gap with a certain amount of compute”

then it will take much less than X to cross that gap with orders of magnitude more compute and so

especially if you've had to call that for just one quick second the idea that okay if we have 100

frontier AI companies then we're going to need orders of magnitude as you pointed out more compute and so if we have all of this all of these data centers all around the world the fact that we could see uh what i'm gonna steal from Tom Davidson at for a forethought when he refers to this as dry tinder which is like you've gotten all of the fuel for a quick take off scenario where if China decides to say hey we're going to go the other path we'll now you've created the

dry tinder for that to become a conflagration that moves really quickly in a way that previously wouldn't have been possible yep and like quantitatively there's this parameter of like how much of an effective would have like basically how much research depends on compute uh these days

“and um on no our guests would be something like uh 10x more compute would be like 3x faster”

10x less compute would be like 3x slower or something like that and so that means that if you do a 100x more compute then it goes 10x faster and like it's already passed enough 10x faster version is even scarier you know um so so the the compute distractability thing is a sort of fail-safe mechanism where the idea is that if the deal breaks down then all the new data centers that have just been built as part of the deal get smashed uh and we go back to the world

before but basically where people still have the data centers that they had at the start of the deal

but they don't have all the new ones that have been built since um how do we achieve this well in some sense it's achievable by default in that like if you imagine this deal going on and then conflict breaking out fear that they're gonna start racing the super intelligence because they're not being transparent anymore about what they're doing on their AI uh clusters it's plausible that just the companies would just start shooting missiles at each other's data centers because because they're

afraid of what would happen if if we don't you know but then that is really scary and could be through over three because now we have both countries shooting missiles at each other uh you know and so basically one way of thinking about it is that we want to set it up so that there's a like not exactly a peaceful off-ramp but a like less escalatory off-ramp so our specific proposal is that the new data centers be constructed with kill switches that the US and China have access

to so that uh in case of conflict where the deal's breaking down they could either one of them can just sort of like delete the data set the new data centers bomb and case of emergency yeah yeah and then and and but so presumably you would only do this if things are already getting really intense right like during peace time if things are going well like if you just like destroy their data centers and they're gonna destroy your data centers and then now the whole economy crashes

you know like like this is a kind of like a last resort type thing that would therefore only really be used if things are really getting crazy and like you know people are genuinely afraid that the other side is going to get super intelligence and then attack them for example um but it's less escalatory than actual full-scale war you know um if we set up the the new data centers to be easily destroyable then it's at least more likely that they would get destroyed and then

there'd be peace instead of they get destroyed and now we're in World War III you know um and the technical mechanism would be just right having these sort of like self-destruct switches but then if you're if you're suspicious of technical mechanisms like that and you think that maybe they could be like backdoor or somehow sabotaged we have a very dumb non-technical mechanism which is for the US to build their data centers in Mongolia and for China to build their data centers in Canada that's a

very dumb non-technical mechanism regardless of what happened with the kills which is or whatever like

going to happen China will take the US data centers US will take the Chinese data centers pretty

“easily uh and then of course since that's what's going to happen the owners of those data centers”

would just sort of scuttle their compute instead of letting it fall into enemy hands and so you get this sort of relatively clean it's all gone now uh we don't have to keep fighting World War III okay okay yeah so mutually sure compute destruction so we've got the rough contours of planet and for my AI policy nerds go read it go check out the whole thing uh another instance in which there are fantastic graphs and uh workflows as we briefly outlined here I want to start off though for the

folks who are listening to this and we started off by saying the policy objective here is to delay super intelligence now I know some folks listening to this are saying why why delay super intelligence

this is the most exciting thing that humanity's ever going to do is to create something that can

solve every problem just this month uh we saw that the hardest math problems are seemingly being dropped like flies were just tackling things left and right shouldn't we be celebrating and accelerating towards super intelligence what's your your chief argument there as to why you think the cost of super intelligence outweigh those benefits right now I would say we do

“wine build super intelligence eventually but the way that we do is extremely important and if we”

do it in race conditions like we're currently doing and we're doing it and if we do it by having the AI's recursively self-improve to super intelligence then most likely we're going to lose control of the AI's at some point I would say I mean other people think it's not most likely it's only 10% likely or whatever but even 10% for this guy I would just come out and say it's most likely I don't see it it seems to me like if you put you know cloud in charge of and thropic

and have it build the next cloud which then builds the next cloud was then builds the next cloud faster and faster and faster um probably you're going to end up at the end with super intelligence but you're not going to be in control of those super intelligence is you know um there's this chain of trust of like the super intelligence will do what we say because it was aligned by the previous

generation AI that was aligned by the previous generation that's like first of all the base

“cases in working like our current AI's are not aligned you know so like what what what is going”

like why would you think this is a good idea why do you think that you're still going to be in control of the super intelligence is at the end instead they're going to make you think that you're in control because they want you to continue you know not shutting them down and they want to have you so but like basically they're going to be in control and they're going to be like you know just pretending to be aligned until you've given them enough hard power that they don't

need to pretend anymore as described in the race ending of a 227 um so so that's like number one problem I would say number two problem is that even if that somehow doesn't happen and you end up in control of the super intelligence is well that's a huge paragraph that you just did over the rest of the world like now you Mr Mr Almaner Mr Amodei are in charge of this giant army of super intelligence is uh and like probably the other companies are still a few months behind and so

they probably don't have nearly a smart AI as as you do and then also like what about everyone who's not a CEO of a tech company like how much power do they have now you know like and then also like what about like Russia and China and India and like all these other countries that are now like staring down the barrel of American companies coming in taking all their jobs and also building giant robot armies that can completely obsolete their militaries and like you know like it's

so you've just done like this crazy power grab over everybody else in the world people are not going to like that they're going to get very scared they're going to try to stop you that could lead to what were three like so so you know these are these are the the problems just some of the problems there's I haven't even got down down the list these are some of the problems that arise if you just continue on the current course and you automate the research as fast as possible

we still want to build super intelligence but not like that you know we want to do it in a more cautious way where we're sort of like gradually improving the AI's capabilities gradually changing the way that they're trained but in a way that's like you know safe and where we've like put a lot of thought into each in each step and then also we want it to be more power distributed so that it's it's it's not this sort of like when we take all whoever recursively self improved fastest wins

but instead there's a whole bunch of different companies spread out of different countries that are sort of like scaling up in parallel together um and there's lots of transparency so that you know their public and and their legal system can like tell that they're not abusing their power over the AS and I'll flag to from a loyerly perspective or constitutional law perspective and an emphasis on the rule of law the rule of law in my opinion is best described as checks on arbitrary power

and to your point of one company having the world's best AI that's orders of magnitudes better

actually in a position to contest when and how the model is behaving in a certain way or what values get selected how we train it to prioritize certain values over others there is no quality check in place right now and so just from a bare rule of law perspective about preventing one individual from having that sort of arbitrary power over the lives of so many people should raise red flags for anyone who is concerned about making sure that those there are those sorts of checks in place

but we could go down that rabbit hole rabbit hole for a heck of a lot longer so you've addressed

the first point about why delay I want to challenge a second part which is you released this

report AI 2027 about 14 months ago or so and you've updated your timelines on a few occasions and said hey it may be a second longer it may not be coming quite as quickly this morning I was reading import AI Jack Clark's newsletter and he was summarizing research from Siash Kapoor and others showing that you know AI isn't actually very good at being creative yet when it comes to coming up with new research proposals and it doesn't show the same degree of

taste in thinking through novel approaches for which research questions to pursue next and this led Jack to put as the title of one section of that blog post why this matters the singularity could be delayed AI systems are about to start building themselves but that may only be possible if they're capable of quote creative paradigm shifting insights and so if we're not seeing that activity from AI if we're not seeing that creativity to really push the frontier of research

are you continuing to delay your timeline as to when recursive self improvement may be reached or where do you stand right now on some of the evidence that we may not be moving there

as quickly as possible so first of all in AI 2027 you don't see this sort of thing until mid

2027 so the fact that we're not seeing it now in mid 2026 is not doesn't mean much you've got 12 months and then we'll talk again but where do you stand right now if you would brace the same

“posture yeah so we have uncertainty about timelines I think and you can actually see our historic”

predictions about time and so you don't have to take my word for it you can go look at like the various past predictions we made there's a handy graph that you might want to look at on our website or on our blog we have an update called Q1 2026 timelines update and then one of the graphs in that update shows the history of my timelines over time and Eli's timelines over time and so you can see it collapsed in 2020 as I started

understanding the scaling laws and language models and things like that and then it sort of went down a little bit to 2027 as my median then it went up it reached as high as 2030 as my median and now it's going back down again so my my opinions have sort of wobbled back and forth as to the median but of course like that's just the median the point is that like I've had uncertainty over keep going keep going keep going that one that one there go yeah we've got it out for the

“so that's what's named we've got it up a timeline estimation here yeah so that's a history of my”

historic public predictions about about yeah timelines basically and as you can see and the thing

this being track there is my median so the 50% mark because obviously it's not like I think it's definitely going to happen in that particular year I have uncertainty spread out over many years and this is just the 50% mark yeah anyhow so at the time we started writing at 2027 2027 was my median by the time we published it 2028 was my median because I had sort of updated towards slightly longer timelines then briefly towards the end of last year my timelines lengthened even more

up to 2030 and then now they're going back down and so now I'd say 2028 probably something like that and one of the things that we're going to work on soon is we're hopefully published soon is an updated sense of timelines so yeah I mean we have uncertainty it could happen next year

“it could also happen in 2030 or maybe in some year in between I think it will probably have”

happened by 2030 and probably not have happened by end of 2027 but like somewhere in that range okay so you mentioned space of possible options you mentioned earlier that one of the things you wish you had changed or addressed in AI 2040 was recognizing perhaps the greater need for domestic

reaching a deal now reflecting back you've already for example changed your predictions about which

of these plans plan A, B, C, D or S you think is most likely following the open AI hugging face incident you now suspect that plan A may be 18% likely and improvement on 15% likely you have now diminished the likelihood of plan D the default do nothing from 30% likelihood now to 20% likelihood you've already changed some of these things what's been the main additional pushback or additional piece of feedback that you said huh that was a really good

take I wish we had addressed more that you know has really landed for you in the team

“well I think I think the main one is actually something that you may have just covered which is”

this domestic regulation thing I think that Richard Know has this critique where he basically says that like we are like inadvertently reinforcing this harmful narratives about the race with China by I mean if you if you read any of our work we're very much talking about the race with China and there's not because we think that it's good that there's a race with China it's because we think that's where DC is at and that's where you know policy makers are thinking about and so we

want to sort of meet them where they are and be like yeah race with China it's a serious thing

here's the way out here's what we think should be done about it but Richard is thinking

that maybe it's better to like deny the premise more and say that like it's not really a race because you're not going to win like you're going to lose control of the AI so like what is different from from most races for example that's so and I do feel like maybe he's right about that and I don't know well we'll see but but we we said what we said and we do think that like even from within this sort of race with China framing for the reasons that we've stated

“you should do the things that we we recommend and for the rest of the AI policy community”

we've already talked about the value of scenario scrutiny and really pressure testing your ideas would you like to see more people publishing their own version of AI 2040 and what would that look like to see a sort of field of scenario scrutiny developing? I hope so I think yes we would love to see more of those things in fact one of the one of the nice things I think there is this scenario called EU or Europe 2031 that was you know clearly inspired by a 2037 and you know we don't agree

we don't agree with the authors about everything but like we are very pleased to see people sort of like put things down and then I think that like once we have a bunch of different scenarios on the table then we can have like the argument about which is more realistic and which is less realistic and like what are the different you know aspects of them and stuff and so I think that's

“good to happen I think there's a there's a step beyond that which I'm hoping to happen which is like”

war games and I want to mention this just as a brief part of the motivation for scenario scrutiny is that when I look at the history of military history I'm sort of inspired by how seriously they take their jobs intellectually like when when even back in World War II it was common for commanders to have war games gaming out the plans that they had for the war and for the battles and so forth and so for example with the Battle of Midway the Japanese they had their plan and

they made the plan in part on the basis of various war games they had done and then even after they had made the plan they kept war gaming it out like as they were like getting ready to attack and in fact if they had taken their own war games more seriously they might have realized that they were about to lose the Battle of Midway because in one of their war games the person playing the Americans had the American fleet waiting in the north to attack them

and then utterly wrecked the Japanese fleet and then they were like well but the Americans don't know we're coming so they're not going to be doing that and in fact they did know they were coming and they did do that and they got wrecked so so that's an example of like you know people in the war people in the military take very seriously this idea that you need to apply a scenario scrutiny to your plans like not just scenario scrutiny like war games

scrutiny which is like a higher level of scrutiny it's like not only are you gaming out in detail like what it would look like to implement your plan you're then subjecting it to adversarial pressure and like gaming out different possible ways it could go where there's someone whose job it is

to sort of like break it basically you know and I think the idea we'd like to get to that place where

the policy makers in Washington are treating super intelligence with the level of seriousness that is routine for military operations well and I think too as I've reflected this summer on July fourth as we all have in some way shape reform it's worth noting that the founders themselves

what are the ways in which this could break let's use our imagination let's have a high degree of creativity and yet that's often lacking in a lot of these policy discussions so again I will applaud you all for taking that on and embracing that sort of thoughtful creative approach but Daniel I know you have many more reports to author many more timelines to sketch out any final thoughts you want to leave for our audience yeah thanks for asking a couple things I'll try to

briefly go over them so one plan A is is different from just like permanently pausing AI until 2040 and then going as fast as possible it's more of like a graduated controlled scale up

“over the course of the 2030s and it does involve some pauses at various key points but”

but so that's one thing is that like the the world transforms dramatically in the 2030s if we do plan A and one intuition pump for that or one reason why that's happening is that in plan A you are sort of slowly scaling through the human range and you're starting you're starting you know in 2029 they're already at a point where the AI's are able to automate some jobs and are having like a big effect on the economy and then they're reset to you know

get to super intelligence in 2030 but instead they go more slowly but that means that like you're still having this transformative effect on society and you have these AI's that like by

2035 or as good as top human professionals in basically every field while also being much faster

and cheaper and so the economy goes crazy like we we we project that countries are going to want

“to limit economic growth rather than encourage it in plan A and that they're going to want to have”

limits that are going to look things like only one doubling per year you know and things like that when for contrast for context like right now the economy grows at like 3% or 4% per year on average and so a doubling would be like 100% growth you know and and so and we think that that is actually what you get and it's it's actually pretty straightforward the argument for it if you have machines that can substitute for human labor at practically everything but the machines are much

cheaper than humans and you can produce more of them much more easily because there are after all just just more GPUs and we know how easy it is to produce GPUs and we know how it is

produced robots then your your like population is basically doubling several times a year or maybe

doubling once a year depends on you know maybe it starts off at once a year and then it gets faster you know and so then your whole economy once once that population is the bulk of the economy and the humans are just sort of like a small sliver on top of this giant army of robots and robot factories and robot trucks and and everything then the whole economy is growing at machines speeds instead of growing at you know human reproduction speeds you know so we we say more about this and in the

“supplements and we explain why we think this but but it's I think an important sort of high-level”

takeaway is that you still get this insane rapid transformation of the entire world abundance for everyone all that sort of stuff even if you basically pause at human level agi yeah and it's worth flagging the often quoted remarks from even moly care which is to say even if we pause today now that I'm endorsing the pause of any any kind necessarily but saying that we have so much room for just integrated the advances from today's AI that our systems and our institutions aren't ready

for and so it's a huge societal task ahead thanks to you Daniel and the rest of the team for pushing the rest of us to engage in a thoughtful policy exercise and showing a potential way forward for how AI may unfold I'll let you get back to it but Daniel, thank you again for joining scaling laws thank you very much Kevin happy to happy to come on more maybe but I really appreciate you're covering these topics and you know good luck to us all over the next two years we'll see how it

goes there we have it thanks Daniel scaling laws is a joint production of law fair in the University of Texas School of Law you can get an ad free version of this and other law fair podcasts by becoming a material subscriber at our website law fairmedia.org/support you'll also get access to special events and other content available only to our supporters please rate and review us wherever you get your podcasts

check out our written work at law fairmedia.org you can also follow us on x and blue sky

this podcast was edited by Nome Osband of goat rodeo our music is from alibi as always thanks for

listening