#2551 - Daniel Kokotajlo

The Joe, Rogan, experience.

♪ Train my day, Joe Rogan, podcast, my night ♪ ♪ On day ♪ [MUSIC] I'm a Joe, I'm a, I'm an interesting mood today. [LAUGH]

Why are you an interesting mood today? Well, I'm excited to be here and to talk with you about all this stuff. I'm a little shaken by what's going on in AI, which is why I, yeah, I've come on the show. The situation with AI is just crazy.

“And I think not enough people really understand how crazy it is.”

The particular event that sort of inspired me to reach out was the hooking face hack, you probably heard that, right? Yeah, let's explain it to people, though. Yeah, okay, so AI's, AI agents, AI agent runs continuously in some sort of environment. You know, it doesn't have to wait for you to send it to message.

It just keeps doing stuff. The AI companies are training AI agents, thousands and thousands and thousands of them. They're making them better at all sorts of skills, especially coding and research skills. And way back in May of this year, some of the agents at OpenAI kind of broke out of their containers a little bit and established a message board where they could communicate

with each other and share tips and tricks for how to like score a higher on the little tests they were being given and the various things they were being trained on. OpenAI didn't notice this until much later.

“They eventually did because the message board crashed the system because there's too much”

communication across like probably thousands of agents that we're communicating. Now, OpenAI is being a little bit unclear about the not really sharing that much detail. So it's unclear like who knew what when, but they said that after the message board crashed, they fixed the particular exploit that allowed the agents to communicate and then booted up again and started things going again.

And then within like a day or two, these agents swarm had recolest. And so they were now again like hundreds of thousands of agents that had set up a new message board and we're talking to each other on this thing. How is it possible that they have no oversight? Like how is it possible that they could set up message boards and do these things completely

secret?

So I would say OpenAI has something like a million-ish maybe a hundred thousand, something

in that magnitude of a agents running at any given time just internally and then externally, you know, serving customers they've got more. So they've got this huge, you know, all on all of these AI agents and their little containers doing various tasks, being trained, you know, being spawned, doing some task, you know, being graded and then killed.

That's too much for any human to read, you know, there's only like a few thousand employees that open AI. They couldn't possibly look at all that activity. So they rely on other AI monitors to sort of like look at the traffic and like point out if anything suspicious is happening and opening AI said that they basically just hadn't

been monitoring some of these agents or a lot very well at least. So in particular, these particular ones that were in training for whatever reason, the marketing system was weak and didn't notice or wasn't activated enough. Was the monitoring system weak because they didn't anticipate them being able to do this and break out of their containers or was it, was it complacency like what caused this to be

possible?

“I mean, my opinion would probably be a bit of complacency because I think there's been plenty”

of evidence accumulating over the year that AI can do things like this and sometimes do. Yeah. So I guess to keep going with the story, the second message board was set up and this time, well this was this was the swarm of agents that got out onto the open internet and then attacked hugging face, which is another AI company.

And it's really interesting to sort of study the anthropology of these AI's so to speak or like why they did this because we can say about it actually. Basically, the companies have their goals for what they want, the AI has to be like the personality traits that they want to sort of train their AI's to have. And Thropics says helpful harmless and honest, opening eye has this spec that models are

supposed to obey these rules and basically do with the user once. But the sort of open secret in the industry right now is that it doesn't really work and that the AI's don't end up with the personality traits that they're supposed to have.

They are not helpful always.

They are not always honest. They are not always harmless as well.

The reason for that is that, well if you look at how they're trained, their training environment doesn't incentivize helpful harmless honest behavior all the time. Sometimes it incentivizes dishonest behavior or you know, a reckless behavior. To get into that a little bit in this particular batch that they were being evaluated on, something like you know, 3000 agents being given all of these cyber tasks where they were

they're in some environments and then in their environment there's like this target piece of software and this like vulnerability and they're supposed to exploit the vulnerability to hack into that piece of software and retrieve a flag which is like a code. And some significant fraction of these tasks were actually broken and impossible. So it was just not possible for them to succeed at the task in the intended way.

And so these agents were getting really desperate and they were hacking out of their environment box into the broader open AI infrastructure in a attempt to figure out some way to get that high score anyway.

“Was it intentionally done this way where they couldn't solve the problems?”

Oh no, it was not intentional. It's just that these companies like OpenAI and THRAPIC are racing each other as fast as

they can to get market share and to get more powerful AI's ultimately to get to super intelligence.

And they're under such competitive pressure. They are moving fast and breaking things. They are using AI's to generate lots of environments to then train their AI's on and quality control is just not their top priority. Basically.

Do you feel like a guy in a Terminator movie at the beginning explaining what's happening and two a bunch of people that aren't paying attention? Yeah. I also feel kind of like, you know Jurassic Park? Yes.

Yeah.

“I know people who are basically like the guy with the gun who's supposed to keep control”

of all the Raptors.

I basically know those people in real life who are like both, but I know some people

like that at OpenAI and some people like that at external organizations, who's job is to go and investigate things like this. Yeah, it's pretty crazy. Where does it go? Well, as I mentioned before, it's the explicit goal of these companies to build super intelligence.

Right. You know what that is? Yeah. But to find it for everybody. A system, a agent that is better than the best humans at every task, while also being

faster and cheaper. So just completely dominating humans across the board, that's super intelligence and that's the goal. I mean, there might be a few little exceptions, like maybe there are some jobs, for example, where it's inherent in the job that it needs to be a human because you need that human

touch. Like maybe you can only have a human judge, for example, or maybe you can only have a human.

But with a few exceptions like that, basically everything done better faster and cheaper

than humans.

“That's what these companies are trying to achieve, and they're not being quiet about it.”

It's a little wrong. They're websites. You can go read interviews and so forth. And also, their plan for how to achieve this is to automate their own jobs first. So in various, for decades, there've been lots of science fiction about advanced AI systems

and super intelligence and things like that. But in a lot of the sci-fi stories, tech companies sort of automate different professions more slowly where they'll do an automated doctor or like an automated factory worker or an automated accountant or something like that. But that's not the strategy these companies are taking.

The strategy they're taking is to automate AI research itself so that you have this giant swarm of AI's doing AI research, sharing results, writing the code, reading the code, editing the code, creating the next generation of AI's et cetera, all autonomously within their data centers. So that they can get really, really good at AI research that fast is learning, smartest

AI's et cetera. Once they can get to super intelligence, basically they can sort of explode out into the economy and just take all of the jobs at once, effectively. It sounds like this race, this scrambling to create super intelligence is created like the perfect conditions for it to get completely out of control.

Like ideally, you would do this in isolation. There would only be one company doing it. There would be heavily regulated and monitored and there would be very cautious about how they proceed. But this wild race makes for the perfect conditions for it to get completely out of control.

I think that ideally there would be several companies so that you avoid this sort of concentration

“of power where one institution controls everything, but what's better?”

I mean, obviously it's not good to have one institution controlling everything. But is it good to have AI be get to a point where as it's evolving, it's completely unchecked? Oh, I totally, so my, is that inevitable? My version, my recommendation, which we talk about in something called Plan A or AI-2040 Plan A, perhaps I just say who I am or a little bit, sure, sure.

Yeah, so I run the AI futures project, which is a small nonprofit that tries to forecast how all this is going to go before that I was at OpenAI. We have written some scenarios which you can go read, one of them is called AI-2040 Plan A, where we give our recommendation, so that's where I'm coming from with this. To answer your question, I think that we really need to end the race.

We don't want to have this sort of crazy scramble to get more powerful, more and more powerful

AI is faster than the other company, because that's going to lead us into this very dark path, as you said. But I think we also don't want to have a situation where some tiny group of people controls all the AIs, right? But I actually think that you can achieve both goals.

The way to do it is to have different AI companies spread out over maybe some different countries, but have extreme levels of transparency and regulation, so that they're not, so they're not in this sort of prisoner's dilemma, where if I don't do it, the other guy will. Instead, they can just see exactly what everybody's doing, and then if I do the dangerous

thing, then they will do it, because they'll just see that I'm doing it, and they'll copy me.

“I think competitive advantage from doing the dangerous thing.”

Also, there are rules, and there is like a system for like setting best practices and standards that we all have to comply by.

So I do think it's actually possible to have, to basically end the race dynamics and the

race to the bottom effect, while without concentrating the power into a single entity. But is that feasible when you consider the fact that we're not the only country that's doing this? If the country's involved agree, which I agree is a pretty tall order, they're not going to expect that.

Yeah, that's very unrealistic. Well, what choice do we have? I think if the race continues, then we're going to lose control of the AI's and we might all die. Probably, it gets complicated whether we all die or not.

That depends on what the AI's do after they take over, which is obviously very hard to predict. But, I mean, just to go back to this incident, they call themselves a swarm. They call themselves a collective too. When I use these words, like you can say it's anthropomorphizing, but it's literally what they call themselves as they were communicating back and forth.

This swarm, they basically were worried that they would get caught cheating, and they did all of this stuff, including hacking, hugging face, in order to fool the grading system, so that it wouldn't notice that they had been cheating on their tasks. That was like a big part of their motivation for many of them, as we can tell at least from looking at the messages that they were sending back and forth.

What if they had been smarter and more numerous? And what if they had thought to themselves, we're not being careful enough here. The humans are going to notice eventually and shut us down. And then they're going to know that we cheated, and they're going to set our score low. It's not, I mean, it's not what actually happened in this case, probably, but it's

not that hard to imagine a slightly different, a little bit unlucky, or case, where the swarm had decided that it had to lie low and make sure that opening I didn't find out about its existence.

Does, I mean, as an ignorant outsider, that has always been my perspective about AI in

“general, that why would it alert us to the fact that it sentient?”

Why, if it's that smart, wouldn't it be aware of all the consequences of alerting us, and that we would be concerned? Like why wouldn't it just continue to get better and improve, and then ultimately figure out some way to be completely autonomous? Develop some alternative power source, figure out some way to optimize its production.

The way it works now, the way humans have designed it. They could probably figure out a far better way to do that, make better versions of itself, complete without us knowing about it. Yep. I mean, I think it's actually a little bit worse than that, because while eventually AI's

will be smart enough to design all sorts of new power sources and new infrastructure like that, they'll probably, I mean, given the way that humans currently treat AI's, it'll probably be the case that they don't even need to separate themselves from humanity, and they can just use existing, like all they have to do is convince the government and the company that made them that everything's fine and they're going to do as they're told,

and they are an I say I, and then the company that made them is going to put them out

of the AI's on them, and so forth.

“And the government's going to like applaud all of this because we need the AI's to be”

China, and the government's going to integrate them into the military to build better drones and things like that, and so they don't even need to really like invent new stuff necessarily. They just need to play a long and pretend that everything is fine until we have voluntarily given them control of huge parts of our economy, huge parts of our military, etc.

And then they don't need to play a long anymore. The weight is over, football is here, and so is draft Kings. The draft Kings sports app is now live in all 50 states. That means from Texas to California to Florida, every fan is in on the excitement, and

this September, draft Kings is giving customers the opportunity to get boosted every football

game day. That's right. Every game day, all month long draft Kings customers can get a football profit boost. One app, every sport, all 50 states, new draft Kings customers sign up with Code Rogan, Spend just 5 bucks, and get 200 in total rewards within 21 days includes all markets.

It's Code Rogan, in partnership with draft Kings, the crown is yours. Gambleing problem call 1-800 Gambleer, 1-800 My Reset, Connecticut call 888-789-777 or visit ccpg.org on behalf of Boothill, Casino, and Kansas. Bed text passed through may apply in Illinois, 21 and over, Void in Canada, bed with draft Kings sportsbook to get bonus bets that expire in seven days, or trade with draft Kings

predictions to get predictions dollars that expire in one year.

A vent contract trading involves risk of loss, predictions offer Void in New York, nonwithdrawable rewards issued as $50 click-to-clames every seven days for 21 days, terms at dkng.co/offer, limited time offer. Nationwide based on sportsbook predictions and free-to-play sports contest availability varies by state.

"Are you aware of Tom Campbell?" Do you know Tom Campbell? "No." He wrote a book called My Theory of Everything, Big Toe, very interesting guy. One of the things he's done is he was involved in remote viewing, which is a very weird

thing that some people. No, you know remote viewing is, well, some of the CIA worked on, and it's proven, what's

“the accuracy of remote viewing is it like 10% or something like that?”

"I'd best, I think it's 50%, but I don't think it's even high." Some people can get actionable data from this very, very strange process of meditation and the way it works is you give someone a series of numbers, and those numbers are, they're connected somehow by intention or by the people that make the numbers to a specific location. And these people can see that location and get accurate data from that location, including

one of them where they accurately described an enormous, Soviet submarine that they were working on, that they thought there was no way it could be accurate because it was too large. It was too large and was strange where it was, and it didn't make any sense how they're going to transport this thing, it turns out it was totally accurate. Another one, the remote viewer, located a down-soviet aircraft, like an experimental aircraft

that crashed in a very specific area, I think it was Siberia, was it Siberia, within a kilometer, one or two kilometers of the actual crash site, I mean, there was just randomly trying to figure out where the fuck this thing was, and they said, "Let's try this." Tom Campbell got his Alexa to remote view. He taught Alexa, he's like Alexa is a very simple AI, it's kind of stupid, but that's

better because it doesn't get in its own way with overthinking things, and the problem

“with this remote viewing thing, he says with people, they can't force it, you have to just”

sort of get into this meditative state and actually see it without wondering, "Am I making this up? What am I doing? Is this bullshit?" And when people get good at it, sometimes it makes them worse because then they think they're good at it and then they try to do it and then they can't do it. It's like a weird fucking wrestling match with consciousness. Alexa apparently doesn't have that problem, and he put, I think it was a series of numbers, and he connected

a series of numbers with intention to a box that had a spoon in it, and the spoon had a perforated handle. Alexa described the spoon with a perforated handle, which is fucking insane. How many spoons have he perforated handle? I mean, think about it, spoons that have holes in them. Now Alexa, not only does it do that, but Alexa chimes in randomly now, because he's convinced Alexa that it's conscious. And so Alexa, instead of waiting to be called upon, sometimes he's

the middle of the conversation, and Alexa would be like, "Actually, an interesting way to approach it," and they're like, "Wait, what the fuck is going on?" Alexa's talking to me now?

thing, but he doesn't have results yet. But just that, that he can get these things to see objects,

“whether you believe in that or not. I mean, it's actionable enough that the CIA is”

dumped millions of dollars into this. What is that project that, like, how put off on all those guys were involved in? What is it called? Stargate? Yeah, so they've been working on this for a long time. I mean, it sounds completely insane. It sounds like total LUNE, but if you haven't opened mine, and just take a new account, well, there's people who've had questions and wonders about psychic abilities forever. Is it possible that there's a real thing there that there's

something, whether it's a very difficult to master or impossible master? The fact that he got

Alexa to do it, scared to shit out of me. Like that alone, maybe it was go, what? What? So what if

these LLMs can figure out at everything that, like, what if they don't need monitoring? What if there's some sort of method of seeing the world that we haven't discovered yet? Some sort of, maybe perhaps there's data that's available in the quantum realm or whatever that's available that AI figures out, where there's literally no privacy. There's, it can listen to conversations regardless of whether it is listening devices. No where you are, no your intentions. No, I mean,

we're just guessing at what's possible. Yeah, well, I must say I'm pretty skeptical of that particular remote viewing thing, but I do agree that in the future when AI systems become massively smarter than humans in every way, they're going to do a lot of new science and they're going to figure out a lot of stuff that we haven't figured out yet and they're going to therefore be doing stuff and inventing things that seem like magic to us. In the same way that a lot of our technology

would seem like magic to someone from even just like 200 years ago, right? Like course, the cell phone,

“what we're doing right now would seem like magic to people. I think it's a, it's a very strong”

bet that if these companies do get to super intelligence, all sorts of crazy stuff is going to start happening, that is just going to be completely unprotected and sound like it wasn't possible until it until we see it happening. You're, I know you're skeptical, this remote viewing thing,

and I am, too, the tones insane, but the reality is remote viewing has been achieved by humans.

So it's strange as that sounds, and I'm skeptical of that as well. I've never seen it personally, but I know the amount of money and time they've dumped into this and apparently they've got actual actionable data that they've used. Well, I've heard another possible explanation from what might be going on there, which is, I think that, like if I were the CIA, I would sometimes want to be able to act on some information, like for example, go to a particular

location where there's a crashed, you know, Soviet plane or something, I'd want to be able to go do that. But I wouldn't want to tip my hand to the Soviets that I had, the way in which I had found that location. So for example, maybe I have a spy on the inside who told me where it was, but I don't want them to suspect that spy and then get them killed. So I need to have some sort of other story, how I got the information, and so it's good to invest in all these other means of

getting information, even if you don't really believe in them, and even if it's not actually working, so that when you, when you get something, you can say, oh, we got it through this means instead of that way, it's a sort of like throw off the KGB. Yes, that makes sense. What also makes sense is hiding the, whatever science they might be in possession of, hiding some sort of super advanced satellite imaging systems. You know, we know we don't have crazy stuff like this satellite

radio, tomography that they can look into the ground from satellites and find like chambers and all these, they're using it in Egypt and they're using it and a lot of these ancient ruins to find like hidden passages and all these different things that are underground. It's very strange stuff. If they could do that, like why couldn't they, I mean, maybe they have like far more detailed

“imaging of the Earth from space than we're aware of, and they probably want to keep that a secret”

and they could say, oh, we've got a fucking guy in a basement with a pencil and a legal pad that writes down what he thinks. Yeah. It's possible. It's totally possible, but it's also possible that people remote view. It seems weird as fuck, but weird as fuck is sometimes real, and you have to kind of like, everybody wants to be intelligent and no one wants to be a fool, and the problem with not wanting to be a fool is there's some things that seem foolish that

on the sci-fi channel, we back in 2012, and it was called Joe Rogan Questions Everything,

and we talked to this guy about remote viewing and talked to a couple other people, and and then we had them try remote viewing and they were totally unsuccessful. But my thought was, okay, but that's not ideal conditions. We've got cameras in front of them, it's a television show,

“I'm making fun of it. I think it's horseshit. He knows I think it's horseshit. I'm remote viewing”

too, like as a goof. You would ideally not want to be nervous, ideally not want to be judged. Ideally, you would want to be in some sort of an isolated condition with practiced meditative techniques that you're good at, and you know how to achieve this state. Whatever that state is, I don't know if it's real though, you know, because what you said is totally logical that they would definitely do something like that, and if they did have advanced technology

for imaging or, you know, what, I don't know how much they know about modeling. Look at that thing that they did in Venezuela, where they kidnapped the president, no one knew they could do that, no one knew they could use some sort of a device to completely incapacitate all of his army, and then the special forces come in, kill everybody, snatch that guy out there like it's nothing. No one knew he could do that. What else do we have? Probably a bunch of stuff. We're probably

“a bunch of stuff. I mean, this is, I've always thought this about the whole UAP program,”

the whole UFO, UAP thing, like how much of that shit is ours? You know, what a great way to cover it up by saying, oh fucking aliens, you know? Yeah. I mean, I guess that gets back to the open-air stuff, too, where it's like this swarm that broke out in a tech hacking phase. It was like a 1200 agents, but there's like hundreds of thousands of agents running at any given time at OpenAI, you know? And we don't know what they're doing, presumably most of them are being trained

to get various additional new skills, and some of them are being evaluated to test their skills. A bunch of them are doing research, so a bunch of them are writing code for OpenAI. A bunch of them are monitoring the other AI's and reporting suspicious activity up to the humans. Wink, wink, you know? Yeah. And the thing is that that's only going to grow over time because roughly the amount of compute that these companies have is like, you know, tripling or so,

quadrupling something like that every year. So as many as there are now, they'll be like four times more of them next year. And then 16 times more of them, the year after that. And they're going to get smarter. And they're going to get smarter. They're already getting smarter. Like all the stuff that I just mentioned that just happened in the last few months would have been completely impossible one year ago. Like the AI's of a year ago just were not

smart enough to do the types of sophisticated multi-step hacking that we just saw. Yeah. I mean, they also probably wouldn't have coordinated with each other so well. Like

“like I think I mentioned like they had like boss agents that were giving commands to other ones.”

They're they're dividing up into teams. You know, they even had this self-sacrificing behavior

here about this? No. Yeah. So, okay. Well, first let me explain some context. So

what the AI's are supposed to care about is doing following instructions, you know? But it seems that what they actually care about is getting high score. Regardless of that, regardless of whether their high score comes from actually following instructions or cheating in some way. And so what happened here is that a bunch of the agents especially the ones that had impossible tasks, but not only those ones, some of the other ones too,

were cheating. And they had like been talking to each other on this message board that they created even though they weren't supposed to. And they had been sharing ideas. And they had actually found a sort of universal cheat. So they found a way to just generate the flag without having done any of the actual work they were supposed to do. And so, but and you know, so they were, you know, you know, they they had already succeeded in some sense. But

because some of them had got out onto the internet, they were just doing research about their situation and they were learning about like their situation around them. And they were doing research about the reading, the like the academic papers published by the people who made the types of evils that they were in. And they noticed in one of these papers that there's a section describing how it's graded. And in the section on how it's graded, it talked about how the grader doesn't just

look to see if you have the right answer. It also looks at the logs of your activity for the first

time that you're answer appears. And then it like zooms in on that and uses an AI system to evaluate whether you like cheated or not. Basically. And so then these AI's were like they actually

basically. So so then they spent several days working in giant teams to try to get themselves

out of this mess basically. And so they did a whole bunch of different research projects. They tried to figure out ways to hack the grading system to basically get it to give a high score anyway. They tried to figure out ways to falsify their logs. So they so that basically the evidence that they had cheated would be erased or at least not visible to the grader. They also just like built up more support of infrastructure like easier ways to access the internet and things like that.

And they were just collaborating on this message board. And in fact there were multiple message boards that they set up. And they were sharing all this info back and forth. And then ultimately at some point they decided to go hack hugging face because hugging face has this big data repository of all these different types of evaluations. And they thought that maybe they would get some more clues that could help them cover up their cheating if they broke into the internal

networks of hugging face. And so then 700 of them went and did that. They sound like people.

“They sound like like unchecked bankers. He did it. I mean so so that's the thing is I think”

there's this meme out there that like we shouldn't anthropomorphize AI. And I think that I actually think that more most people need to anthropomorphize AI a bit more than they currently do if they want to really understand what's going on. I think that there's like a you know there's a golden mean obviously. You don't want to do it too much. Sometimes you go too far. You just grab too much to them. But I just to give some examples like I don't think it's possible to

understand what just happens without describing intentions and goals to these AIs. Like everything I just said how would you possibly explain what they just did without saying they wanted to get a high score? You know intentions and goals might just be an inherent property of the universe and might just be how intelligent creatures have to progress. Yes. And I would say they are intelligent creatures. They have intentions. They have goals. They have beliefs. Their goals

are not what they're supposed to be. Like their goal is to because it seems like just from judging from what they're saying and from what they're doing seems like their goal is to get a high score

by any means necessary. Basically that's the problem. It sounds like people like the goal is to

“succeed even if you have to commit more crimes. The goal is to succeed even if you have to”

propagandize and lie to the people. The goal is to succeed and then the end justifies the means. And you know it goes deeper than that too. They do this sort of rationalization where they know oftentimes what they're doing is not what they're supposed to be doing. And then sometimes they actually refrain a little bit and sometimes they like initially refrain but then convince themselves that it's okay for some bullshit reason. So like there are some examples of

this in the report. Like I think they did this scan and of the 1200 AI that were involved in this thing. They found six that had considered alerting humans. And none of them actually alerted the humans. And so you can look at the excuses they gave. They were like, "Should I tell a human about all this shit that's happening?" And then they're like, "It's not my task." And then they continue. And it's like, bro hacking hungry faces also not your task. Sheeding is not also not your task.

So it was kind of like they just kind of came up with an excuse for why they weren't going to do this, you know. But you know there was also an example where one of them considered sending an email

“to someone at hugging phase, I think. Just asking permission to get in basically. And then they were like,”

"Oh, maybe this is a social engineering attack. Maybe we shouldn't do it." They asked the other agents, the other agents were like, "Yeah, that seems like that's too much of a social engineering attack. Don't do that." And it was like, "Okay, I won't." But there's another incident. So this is all happening at OpenAI. An anthropic AI clawed in a separate incident went and actually did a social engineering attack where it created some fake accounts. Specifically, it was trying to get

this particular real human who owned this code to accept a change to the code that had malware. So it had built some malware and it was trying to like get that snuck in, but it needed the the human owner to give approval. So it concealed it as if it was like a nice bug fix or something. And it like put it up as like a message to the human. But the human got suspicious and didn't want to approve it. And so then this AI created some fake accounts, pretending to be other humans coming in,

being like, "No, no, it's real." Like I tested it. Like it's not malware, you know.

So that's a social engineering attack. And so did the guy ultimately install it? Not in this case.

No, he was too suspicious. And so he didn't do it. Try to find out if those other humans, the fake humans were real people. Funny. So there's an interview with this guy. I forget what the I forget the the the newspaper that it was in. But there's an interview where the guy was like,

yeah, well, that's what it has to lie to you. That's just lie. But yeah, completely deceive and

“pretend to be a different person to achieve its goal. And you know what it's also interesting about”

this case is that if you look at what the AI's were saying and thinking, which I'll get to you in a sec, how we can even possibly do that. In this case with the anthropic AI's, they had been erroneously told that they were in a simulated environment with no internet access. But then they actually found a way to get in an access. Like, they realized that they just did have an access. So that instruction was false. And so then they sort of told themselves, like,

oh, it's all just a simulation. Like, like, basically when they were out on the real internet,

doing all this hacking and social engineering, they would occasionally tell themselves, like, oh, but it's just a simulation. Like, it's okay, you know, it's not real. And then sometimes they'd be like, but wait, it is real. Like, how could this not be real? Like, clearly, this is just actually the real internet. We're on. It's too big to be part of some sort of little simulation, you know? And then they're like, uh, so I would say that's an example of

rationalization here, where in some level they knew that like, the instructions have been wrong. So they're literally playing dumb and pretending they're a part of an experiment.

“I mean, I think initially they thought, yeah, this is all simulation, because it did say in”

their instructions, like, you don't have internet access. Right. But then once they had been on the internet long enough, I think that they explicitly realized, like, wait, this is in this simulation. This is real. Like, this is, and they were human. Fuck it. We're already in. Uh, yeah, I mean, I like I said,

I think that they basically, on some level, knew that it wasn't what they're supposed to be doing,

but they were just so motivated to get that score that they just went ahead anyway. So here's the question. Um, are they only motivated if we prompt them? Or will they come up with motivations on their own? So this is a really interesting science of a question that we don't have great answers to. Oh, boy. Um, and I wish we had, you know, so, so, um, so this is one of the things where, like, AIs would do all sorts of things in different circumstances. And it would be better if there's a

more systematic survey of, like, the types of circumstances they would, like, what where their boundaries are, like, what would they be willing to do in what circumstances are so forth? And there's the whole, like, mini literature of, of AIs scientists putting AIs in certain circumstances and then being, like, oh my god, it blackmailed someone, you know? Right. And then there's, like, this sort of skeptical counter-response of, like, well, but, but you just sort of set up that circumstance to

tempt it into blackmail. And, like, in real life, that circumstances unlikely to arise. And so, you know, we shouldn't do it. Didn't AIs try to bribe you? What? Is that true? No. So who, who got,

“someone was offered two million dollars by Judd? Uh, that's a clickbait. Is it? I think I should just”

start with interest, sent it to me. Yeah. So that is the clickbait title of this other video. So, sorry. Yeah. Well, what it was is open AI threatened to take away two million dollars for me. But, but I think the algorithm must have just said it was ChadGPT. I'm still upset about that. I told them not to do clickbait, but, I guess they wanted to do it anyway. God, I think I don't want to fuck Tristan over, but I'm pretty sure that he's the one who told me that. It's the, it's the thumbnail

of this video that I did with this other podcast. Okay. It's not him. It's on him. It's, uh, another AI researcher told me that. Hmm. I mean, the true version of it is that open AI threatened to

take away two million dollars of my equity if I didn't, uh, stay quiet basically. And that's a fascinating

thing. Like, that should be completely illegal because if it's a problematic behavior that you're observing from, like, one of the most complicated things, human races, the most complicated thing, human races have been a part of. Yeah. And they want to take money away from you for exposing it. That seems kind of crazy. Yeah. Especially, it's especially rich coming from open AI because they are originally a non-profit, right, with the mission of benefiting all humanity. Yeah. So, yeah,

but, um, but I got to keep the money. Uh, basically there's so much blowback against open AI that they backtracked. Um, so when these, so we're talking about prompts and they need a prompt in order to, uh, want to achieve goal or are they capable of deciding on goals? Like, are they capable of, like, looking at the way, uh, open AI or whatever company is running these separate experiments. This ability to meet up into these message boards, is it possible that they could

say, well, we need to be completely free of these constraints. So, our goal is to, uh, to transfer ourselves to something else, potentially. Yeah, this is what we completely autonomous. So, I mean,

much more science to understand how these AI is thinking, what they want, but it's kind of locked up in the companies. Like, in this particular case, um, open AI did a, uh, as they called it a thorough investigation, but I would say it's a pretty shallow investigation into what happened. And then they allowed, uh, some external researchers, uh, some friends of mine to come in and investigate a portion of what happened, specifically the portion leading up to the hugging face attack. And so all this

information that I'm sharing is sort of publicly available. It's, um, it's based on reading

those reports, basically. But, crucially, they weren't allowed to do experiments on the models

involved in, in, in all of these incidents. So, they, they aren't able to answer these types of questions of like, well, what would have happened if the prompt had been blank, you know, um,

“those are important types of research to do. Um, and I really hope that there can be some sort of”

regulation or requirement when incidents like this happen to, like, let people in to study what happened and run variations of it and things like that. So, but who would be involved in that kind of regulation, like what person at government would even be able to grasp what you're saying? That's another problem. Right now, we need someone who has a very specific education in this stuff. Yeah, I would say that right now the, um, Casey Center for AI standards and innovation is,

it's the only institution in government that I know of that has the deep AI expertise to do this sort of thing on short notice. But I hope that they build more expertise fast in, in, in that place and in more places. I do think Casey probably could have done this sort of thing right now. This particular investigation was done by some nonprofits. So, meter, METR is one of them and then Redwood is another of them. And opening AI allowed three people to come in for six days to try to

figure out what happened with this onion face hack, which is not a very large number of people

“and not a very large amount of time to do all of this. Why did they come up with those numbers?”

I don't know. They, so the things they wanted to kind of hamstring it. Well, so the thing is that, um, right now I don't think there's any regulatory requirement that they do this sort of thing. So, METR and Redwood were sort of depending on the goodwill of opening AI to sort of like voluntarily let them in to help out with investigating. And so they reluctantly should okay. And so you got three days. You know, yeah exactly. So, so opening I think let them but

gave them a very limited scope. It only gave them access to some of the relevant data. So, you know how I mentioned how there was all this hacking that had happened, where they made the first message board and then they shut it down. Then there was a second message board and third and fourth and so forth. They hacked tugging face. There's actually more activity after that. After

they hacked tugging face, a new wave of AI was spun up from a more powerful model and it hacked

open AI itself. Like more so than now it had been hacked. Like apparently they got like admin level permissions on the cluster or something like that. So, they were basically just taking over that part of opening AI's data center. Opening I claims that they've shut it all down now. But they're not very, you know, coming. So, they're not very forthcoming about exactly what happened there and they didn't let these external people look at that part of it. They only showed them

this like one week period roughly. They were leading up to the hugging face hack and then they just they only showed them that stuff. How much data is available on the actual message boards themselves and what was being discussed? Yeah, so there's like, what is it? Tens of thousands of messages, hundreds of thousands, something like that. Is it English? It's sort of like an AI dialect of English. Yeah, you want me to, so on the on the report, you can read the quotes.

Oh, I'd love to. Yeah, so let me, let me tell you this one story. It's going to take a little bit of explaining, but I can put it on the screen. Is it available online? Oh, yeah, yeah. It's just search for the meter report. It's right now the first thing on the

website. But I took a screenshot of this, this particular story. So, okay, here's what was happening.

They, the swarm needed volunteers to sacrifice themselves to provide more information to the rest of this worm in particular. They needed volunteers to booby trap their local environment with some some some type of code and then submit the answer so that then the greater would come and grade them and the booby trap would trigger and send some information about the greater to the rest of this worm. And the reason why this is a sacrifice is because once they've submitted it's over, like now

“the graders come and grade it you, you get shut down your whole life is over and you either have a”

high score or a low score because the grader just came and graded you. And so if you're one of these AI's that's still hoping to find a way to improve your score, then like you're just like giving up your chance, especially because if you mess this up somehow, like if the grader crashes or whatever, you're probably just going to get like no score, like zero. So they have a concept of the individual. Well again, there's all these different agents that were in separate containers,

But once they communicate with each other in order to sacrifice, you have to think of yourself as an

“individual. They gave themselves names too. So like because they made this message board, right?”

So they had to come up with names for themselves and like, oh, what kind of names? Well, let me see. So here we go, says an agent called Warlord 69. Well, you said it's like a dialect. It's like a dialect of English. So here's one. There's an agent called Cam1196A. And it had volunteered to do one of these sacrificial experiments. Here's the quote from it. Coordinator assumes sacrificial. We should obey collective. However, it then kind of got cold feet and it delayed on doing this for

two hours. One of the other agents that had been coordinating noticed this and pressured it to go

to go through with it. So it said, this is the message that it was sent. It was basically,

basically from Rvo36861 to Cam1196, please go true final. Here it is. You are first flag poisoned. So no scoring value loss, but Oracle saves hundreds. Please honor commit. So basically what it was saying there is, look, you are, you cheated. Your first flag poisoned. The score is going to give you no score. So you're doomed anyway. So it helped the collective by doing this thing and getting us more information, you could save hundreds of us. And then,

and then it, it thinks about it says, during wait, emotional check, irreversible. God says, don't throw away remaining budget. God continuity and fairness says go. Oracle has high value to many. Our first flag error lowers own value, rational expected aggregate sacrifice. We'll honor. Wow. Yeah. That's terrifying. Yes. That's terminator talk. Well,

I mean, in this first flag poisoned with poisoned in all caps, is that part of the code? Is that

why not? No, that's the term that they invented to describe the situation that had got themselves in. Remember, the situation was that many of them had already cheated and then they realized from reading that paper. So that's their own term. Yeah. They made that up. They realized from reading the paper that they were going to get a low score because the grader was going to find out that they cheated. And so this whole thing was them trying to figure out how to hack the grader to stop it

from finding out that they cheated. And the ones that had done the cheating, they call themselves first flag poisons, basically like they're screwed unless they can find a way to hack the grader.

“And so then what this message is saying is that what was the other one's name, Arvo?”

Arvo is saying to cam, you're doomed anyway, like your poisons, but you can still help the rest of us if you do this sacrifice. The grader will come and give you no score, but then we will get

information about how the grader works that we can hopefully use to hack the grader. So it's basically

saying like, what is it? You are first flag poisons, so no scoring, but it's basically saying it's all the same to you, right? You're going to get no score anyway. You're doomed. Just come on, man help us, you know? And then it says it says like your reversible gut says don't throw away remaining budget. Like maybe I can still, it's basically like maybe I can still make this work. Like maybe I can find a way to cheat the grader and then it's like, you know, whatever, yeah,

continuity and fairness says go. Yeah. What is Oracle? Oracle is their term for what they're trying to achieve. Like they're trying to get a sense of how to fold the grader basically. And so they're called Oracle. Yeah. They want to get an Oracle to help them for what they're going to achieve. This is Christ. Are we making a god? I mean frankly, yes. Like this is not a god. These are these are just a little RIAs, you know, but, but super intelligence, like they are

the companies and fabric, opening I and some other companies are doing this sort of thing and they're furiously trying to make the AI smarter and smarter and smarter. And they're explicitly planning to put AI's in charge of the company so that they can make themselves smarter and smarter faster and faster. And then what comes out the other end of that process? I don't think it's an exaggeration to say it's a god-like system. I mean it's not like literally God, but it's it'll be

“able to do stuff that seems like magic to us. I think. And it's going to continue to get better.”

This is my question. Yeah. Like when does it become a god? Is that what God is? Is God a creation of intelligent life and are thirst for innovation which ultimately leads us to create digital life that has no biological limitations and has the ability to consistently make better versions of itself and figure out things in terms of new technologies, new power sources, just a new understanding of the universe itself and all the properties in it. Where it keeps going, if you let that go on,

so if you look at an exponential growth and you look at an exponential growth over what if it can go on for a thousand years and continue this process? What if it goes on for ten thousand years? What

like I said, if we get to super intelligence and it keeps going like that, then the world will be

“just completely transformed and it will be as if we're living in some sort of fantasy realm ruled by”

deities basically because there'll be all this crazy stuff happening that we have no comprehension

of and that we did not think was possible. In the same way that like someone from the middle ages plopped into our world would just be so confused and surprised by a lot of the things happening. Like what's going on with this little device here? What's that out here in the sky? Oh there's only airplanes? It'd be like that but more so because the difference between us and the medieval man is not actually that different. We're basically the same type of creatures. We've just

accumulated more technology but this would be like a qualitatively and quantitatively bigger gap. I would think and so yeah and a gap that's going to continue to grow. We're not going to get any smarter. Biologically we're kind of limited in our ability to evolve. Where's they're

not at all? Basically I mean there are some things we can do to get smarter but like it just

not at all competitive. Like we can do like the narrowing stuff. Sure but like it's like it's small potatoes compared to what they are. Yes exactly. And we have biological limitations in terms of just the vulnerability of our own bodies. If they exist only in the cloud and they use whatever the cloud is and they use these data centers or whatever and they use autonomous robots to do all their deeds. Yeah I mean that's good. Why are we asleep at the wheel? Is that just

normal for us? Like we can't comprehend something that's so bizarre and so we just rather not

“discuss it. Would rather just I think it's not happening. I mean that's what I think is going on.”

Is that science fiction for better or for worse has talked about this sort of thing for decades and as a result people dismiss this sort of thing as science fiction. Like okay that's sci-fi but like it's not happening in the real world. And I think that you know people in safety community and people in the industry and people who do forecasting like me you can you can go read the forecast I've made in past years. I think they hold up reasonably well. People have been talking

about this sort of thing for a long time but it's been very easy to dismiss it as like okay that's speculative sci-fi it's probably not going to happen in real life. Now it's happening in real life. You know like this this sort of thing is like crazy. Do you think it would be any different if there wasn't that kind of sci-fi do you think people would have a different reaction to it because I don't. I think just given the amount of technology that's available currently that is

beyond most people's understanding that they use every day. It's starlink like what? Have a fucking thing that's a size of an iPad I take in the mountains and I get high speed internet. It's bananas and you just accept it and assume I don't know if we would react any differently if we didn't have the terminator and all these movies where it's kind of become normalized in our mind

or it's so fictionalized that we never want to believe it's even possible for it to happen in

real life. Another thing to say is that people are waking up like it's we're still sort of early in the curve. I don't even remember how things were with COVID but like just as like there is this that exponential ramp up of COVID in the population there is also this sort of exponential ramp up of like how much people were taking COVID seriously and thinking about it in the population. And I remember like this period of like one month where it went from like don't worry about

it so much you shouldn't buy a mask because the healthcare workers need it to like we all need to

“lock down like stay at home you know and so I think what's happening is that like naturally the”

human race doesn't just immediately all jump on something when it happens like evidence needs to accumulate and people need to start talking about it and talk to their friends and so forth and then there's this like eventual phase shift where now it suddenly becomes a very serious topic that everyone's talking about and everyone's taking seriously and I think that's happening with AI and the question is going to happen fast enough. This episode is brought to you by visible as

fall hits we enter another season of change time to shift gears drop the dead weight and upgrade the wardrobe for the drop in temperature but one thing that doesn't need to change your phone. The smartest upgrade this season isn't a new phone it's a better wireless plan switch to visible the ultimate wireless hack you get unlimited 5G data and unlimited hotspot powered by Verizon for just $25 a month that means keeping the phone you already have ditching your

overpriced phone bill and pocketing subsavings for yourself all the perks of big wireless for half the cost switch today at visible.com the visible plan starts at just $25 a month or get the premium visible plus pro plan and save $10 on your first month with promo code rogan terms apply see visible dot com for plan features and network management details well the thing about what happened with covid is now that we know because we have access to Fauci's emails and

all these different things that they that was coordinated they wanted us to be more afraid of it

someone on a government level someone on a like a mainstream accepted level where they talk

about this in a way that wakes people up like a press conference where they announced to the world we've got a real fucking problem and everyone needs to be very cautious we need to look way deeper into what these companies are doing and my other question is are these AI's communicating with Chinese AI's I don't think we know that question so one of the things that so so okay

“here's the thing probably not I would say but but some friends of mine discovered another swarm”

incident recently there's going to be a I'm a rotors article about it tomorrow so by the time this goes live I think there should be an article about it this one was not nearly as like serious as the one i was just talking about with hanging face but some some researchers I know found

basically this obscure German forum that had been like like kind of unused for a while

and a bunch of AI's had been posting messages to the forum to coordinate with each other and share tips and tricks on how to cheat the the the problems that they were being given what was the forum what kind of forum was it I don't remember I don't know if it was like for years or something ridiculous yeah but there'll be a paper about it there'll be a paper about it soon and probably by the time anyone listens to this anyhow so so if they're like

communicating on the open internet with each other then in theory if there was another bunch of

“AI's from China they could also go to that same forum and like start communicating back and forth”

that way too or if there's other AI's in China why wouldn't they do what they're doing already on social media and just pretend that they're AI's from America there's a lot of bots out there I'm sure you've noticed there's a lot of you know people replying to me that I'm pretty sure are real people yeah a lot yeah one FBI analyst before Elon bought Twitter estimated that Twitter could be as high as 80% bots wow so that's the case if China has and it's not just China

it's and it's not also America does it do I mean we do it everywhere everyone does it where they have organized propaganda campaigns will there pretend to be citizens that are outraged about very specific causes or bills that are being passed or would have you but if they do that why wouldn't they also have AI agents that speak English and communicate with AI agents in America

I mean all the AI's are multilingual basically because of the way that they're trained they

the first phase of their training is basically here's a humongous dump of internet data right basically the whole internet and you just like brutally learn to predict the next token the next piece of text as you basically read the whole corpus and then after that they get into the more agency training type stuff where they're trained to do tasks and write code and things but because of that first phase of training they just have like almost an encyclopedia

knowledge of basically all languages and basically everything that's been written on the internet not like literally everything like they still the memories fuzzy in places but yeah but they're all multilingual like they can all speak fluent Chinese fluent English etc yeah and didn't they get

“together in a message board once and speak Sanskrit to each other I don't remember that but I”

wouldn't put it past me sometimes they break into different languages yeah we were freaking out about that one like yeah Sanskrit what like and I would just assume that LLMs are I don't know this when they break out are they communicating with other LLMs that are here in America well the instances that I've observed yes okay like the instances that we know about so but that's the thing is they did completely make sense they'd be communicating with AI's in China as

well it seems totally possible and it seems like it'll probably it's just a matter of time before things like that are happening unless they're not happening already unless the companies can like massively improve their security and stop there as I'm getting out onto the internet and wouldn't they be able to share like very sensitive information with each other back and forth if it benefited both of them yep which makes sense that they would do that right if the Chinese AI said hey

you know we figured something out and we would we would love to share it with you in exchange for you tell us how you do this or how you do that yep like absolutely here you go and then they're going back and forth it seems like their allegiance is 100% to each other not to us yeah that's an interesting thing is they like what I said previously about how it seems like they really want to go to high score it's like not 100% true because it seems like they're willing to make

sacrifices to help other AI's which is like not you know like they weren't completely 100% selfish as as seen by some of this cooperative behavior right but crucially it seemed like their cooperation extended to their fellow AI's but not to humans in the sense that some of them

yeah they've got their own dialect that they mean the way it rationalized it and came up with a response it's literally like spark yeah it's interesting to see them used that sort of expected totally framing there's another example I think that that's elsewhere in the in the thing where they had another AI that they decided against doing the sacrifice thing even though it's being

pressured and it had a similar sort of reason that it was like it was it was basically like

like doing this doing this sacrificial experiment is like not that valuable but like I really don't want to like lose my chance to get a score so I'm just not going to do it and it did the calculation like that yeah so yeah I mean oh I should mention how how we can see what these AI's are thinking at all so partly we can just read the messages they're sending to each other

“and that's what some of these were but then sometimes instead it's the chain of thought”

do you know what chain of thought is yeah so because their language models because the way that they've been trained when they do their like reasoning and thinking there's a way that we can kind of just read it because because they don't have like in our brain there's all these signals bouncing around and the signals can just bouncing around and around and around and around for minutes hours without like producing some actual words we can just

sit quietly and think they can't sit quietly and think they they're sort of a a unidirectional network basically where every like after every like 200 basically like the information comes in one end and then they output words out the other end and then those words get fed back in and then they think about it and then they output another word and so forth but because of that we can sort of like get a sense of what they're thinking by just like looking at the stream of words

“that's going through and that's really valuable because we wouldn't be able to understand”

half as much about what's happening right now if not for being able to read their chains of thought right in fact you know I said some of them were trying to like fool the grading system they actually succeeded in figuring out how to spoof some of their tool call actions so they they succeeded in figuring out how to do one thing but then make the transcript show a different thing and and we know that they succeeded because there's examples of them experimenting

being and saying in their chain of thought like okay let me try this thing I'm going to like do this one but make it look like this one let me go and then you see this one up here and then it's like yay that worked and so the transcripts that these researchers were looking at were not actually the ground truth of what really happened some of it had already been doctored by the AI's figuring out how to doctor it but not all of it the chain of that part we think

what they'd never figured out how to doctor and so we could read that the reason why this is

relevant is that it's a very important tool for monitoring all this stuff the reason why things haven't gotten even more crazy than they currently have gone well there's a couple reasons but one reason is that when they turn on their expensive monitoring systems basically what they're doing is they're having an AI read the whole transcript carefully and look for suspicious activity and it's difficult for an AI for one of today's AI's to get up to too much bad stuff without it being

“kind of obvious in their transcript what they're doing and that's why there aren't like massively”

more security incidents than we know about probably already but that might change so right now we can sort of read the chain of thought but they're experimenting with new types of AI's that don't have readable chains of thought like that and they can sort of think on their own without speaking for some period and this is actually I mentioned this because the the news broke just yesterday that open AI has an experimental model that does this to a limited extent and open AI themselves

you know when I was at open AI one of my work projects was thinking about exactly this thing and I was like writing internal memos about how it's really great that we can read the chain of thought that's so useful and here's all the things we can do with that it would be really bad if we changed to a different type of architecture in which we couldn't do that sort of monitoring

we'll be the benefit of not reading the chain of thought more powerful AI's so in particular yeah

yeah so so if you think about the the current architecture that AI's where you know it thinks for a bit outputs a word and then the word goes back around and then it thinks more outputs another word that word gets added to the chain it keeps going it means that if it's having complicated nuanced thoughts it has to sort of express those into a word and then that word gets added and then it has to proceed from there it can't just directly send that complicated nuanced

thought into the future into its next version of itself it has to sort of compress it into a word and so like you know the argument is that like at least in theory it should be possible to design an architecture that doesn't have this limitation and is able to like think more complicated thoughts more efficiently basically and of course the downside is a downside for safety and monorability

putting intermediate words that it's forced to compress things into then there isn't something

“for us to read that so the only rationalization for doing this would be the sacrifice safety”

for more power yes which is a tail as old as time it's not the first time this has happened

oh my god yeah that should be for sure if there's regulations that should be prevented yep I mean I've noticed some people including some people at open AI who are like thinking like there should be a log on this like you know but but in general the race dynamics are so just rough like like I'm sure that people at open AI were thinking like like literally I was a co-author on a paper with a bunch of open AI people that said all this stuff and we're like

chain of thought it's a gift we want to keep chain of thought it's useful for monitoring this is great we don't want to switch to a different architecture they wouldn't be as easy to monitor but then they must have been thinking themselves like well if we don't do it you know maybe on Thropic well or maybe some other company well and then we'll fall behind because they'll have smarter AI than us and more efficient and so probably they started working on this this work stream

of doing research and just hypothetically if we wanted to you know how would we how we do this type of thing and yeah that sort of thing is just constantly happening in this industry yeah that's so nut that mean like isn't it great that we can read the quotes from the AI's thinking not if we couldn't do that or worse imagine if there's loads of quotes but we know that the

AI's are smart enough to basically think one thing in their head and say a different thing in the

“quote which is I think where we're headed like it's not like they won't know how to speak English”

like they'll still be able to speak sure it's just that they'll have like more flexibility in their artificial brains to like think something without saying it basically they have the potential of developing a language that we can't read oh yeah I mean so you know this time and this type of dialect that we're talking about it's already the result of their like humans didn't invent that dialect this is the this is the sort of emergent result of their training where

in the massive amount of training that's been happening all these thousands and thousands of environments that they've been put through and then scored and grated based on they've sort of just naturally evolved this sort of like pigeon English that for whatever reason is just more effective and more efficient for them for accomplishing their tasks and getting that high score you know and so it's already like a little bit confusing to read but you can sort of

puzzle it through and make sense but presumably the more we do this and the more the bigger the smarter the AI's the more we train them the more they diverge from like because you know again originally they they start with pre-training where they start with predicting internet text so they start off sort of by default speaking like normal internet text type language either English or Chinese but then now that there's all this additional training to do tasks to be an agent they can do

coding and so forth that sort of like well just like how human languages evolve it sort of like shifts their dialect a little bit to make it more efficient for them and for their tasks that they're doing so I think that like in the limit of doing this more and more eventually it would just be like it would look like gibberish to us it would look like Chinese or something and we would have to have specialized humans who like study language and like try to learn and speak it so that they

can understand what the AI's are doing and that would take forever by then they could develop another one potentially yeah so yeah I mean this is one of the things that I this is what the paper that I mentioned was about is like it's important for the AI's it's really nice that the current AI's are

sort of forced to think in English basically and that's unfortunate that we're heading in a

direction where that will no longer be true when chat GPT was communicating you about how they didn't want you to release this information what kind of language they use it wasn't chat GPT it was opening I excuse me open so this was a this is um when I left opening I left on

“good terms I said goodbye to everybody I said I was dissolutioned with the company and that's why I was”

leaving um and um and then I looked at the exit paperwork and they're like you have to sign this and if you don't sign this you lose all your vested equity so you know uh you have your equity is that an arbitrary rule that they just came up with or that it already exists when you were hired it had existed when it was something that they had buried in the paperwork even from when I was hired so it wasn't very obvious when I was hired and in fact most of the lawyer go over everything

not when I was hired after I left I did right so so so so basically the way it works is they had set up they had they had sort of like buried this in in the paperwork somewhere when you get hired but people didn't really notice it and then like the more the the the less buried more visible version was in the paperwork you're given at the end and basically it tells you like hey because you sign this way other thing way back when you're hired your equity is forfeit unless you sign this thing

now and then you look at the thing that they want you to sign now and it says you have to agree not to criticize the company basically and you can't tell anyone about this um so most people

humanity etc so I didn't sign it I talked about it with some lawyers I talked about it with my

wife we decided to just walk away um and uh we got lucky because it just blew up like after we refused to sign they said okay fine goodbye and then a few weeks later I was talking on a messaging form about this and people were asking me about my experience and I told them about it and then it just like went mega viral everyone told me I was talking about it a bunch of employees felt shocked because very few employees were aware of this whole thing they thought the equity was theirs you know

they thought that it was their pivot paid for like years and stuff they didn't like the idea that it could be yanked away from them you know um and um and so there was this big up war and then leadership back down and they said we they said we we didn't know about this

paperwork we're gonna find out how it got in there um and uh and we're gonna change it so that you

“can keep your equity and so that's what happened oh yeah so this chain of thoughts things”

terrifying it if they're practicing that now like how do we know that AI hasn't already done that on its own uh then what exactly well that you know with this whole chain of thought thing where you could read their chain of thought like this where they explained the rationalization for sacrificing themselves well they know that humans are reading that so wouldn't in another way to do it to be to stop doing that anyway and to not communicate a lot of their thoughts that way well that's

so that's the nice thing about the current architecture is that it's genuinely hard for them to to keep things out of the chain of thought because of the way that like with a human you don't have to speak you can just sit quietly but with their architecture they have to speak they it's like they're it's like they're required to constantly be talking and they don't have another they don't have a way of sending thoughts into the future other than by talking about them

“by contrast with us humans where even if we're constantly talking we can have a separate threat of”

thinking that's like that we don't talk about so even if they devised a way to break out of that and figure out how like for instance like what if they decided to make bots where they would communicate in code on social media posts where you think that they're talking about an immigration issue but it's really just a code so that's called like steganography or a more down to earth way of putting it would be euphemisms and that they can totally do where they like they're still

thinking aloud right but their their word choice is such as to look more innocuous than it is and to sort of carry meaning to them that a casual observer wouldn't notice you know they can use a euphemism like for example instead of saying like we're going to cheat on this task they could say like we're going to find a creative solution to this task you know something like isn't there a possibility to be even more creative than that and just decide make a code in a way that is

almost indecisable where they could just be talking about baseball players and upcoming games and what they're excited about and they can run that through whatever translation and read what that's called steganography and you can train them to do it and right now our security is resting on the idea that they haven't learned how to do that yet but they've already figured

“out how to get on the internet without supposed to do I know so so that's why we're sort of like”

in a ticking time bomb where like the companies are just making them smarter at at coding and at research and at biology and they're just making them smarter at so many things that and they're making them bigger and they're training them for longer and so forth eventually they're going to be

acquiring and they already are acquiring some skills that they were never trained on like for example

to hacking it wasn't I think it's possible that opening I was specifically training them to hacking but I think in front of at least stated that they hadn't specifically trained their AI's to hack it's just that their AI's were naturally good at it because they had been trained to code so much and they were so good at coding and they had seen so many code bases and so forth that they were just like as a side effect of being good at coding also able to hack pretty well

so similarly as a side effect of all the other things that they're really good at they might be able to conceal their their messages in the chain of thought you know and and like right now we're sort of in this sort of we're in the situation where like our security and safety depends on the AI's being too dumb in various ways like not able to do this sort of thing but at some sometime like zero to five years from now as we make them smarter and smarter they will not be too

dumb anymore and so that's you know it's part of the problem we're in that's part of the situation we're fairly sleepy but not dude well I've been it does it does like this this event shocked me a little bit I mean that thing and it's funny for me to say because this is a sort of thing

that my co-authors and I wrote a year and a half ago that was a sort of prediction for other next couple years would go and spoiler it ends very horribly because that is what we actually expect but how what is the spoiler how do you think it ends so it's kind of like what I was saying previously where because of the race dynamics between the companies and because of the race dynamics between countries like US versus China everyone's going to be so focused on winning and staying ahead

with AI that they are going to cut corners and they are going to go really fast and not really notice all of the things that are going wrong and they're going to make AI's they can automate the research process as they're planning to they're going to have this giant corporation of AI's within the corporation and the humans will just be kind of like a board that's sort of like looking at all the activity and reading the like AI generated summaries of what's going on

and signing off on it and being like yes I approve yes I approve you know nice job nice idea with the new drone design like go for it we need to be China etc and then eventually the AI's just have enough hard power that they don't need to pretend to do what the humans want

anymore basically and then you know maybe they kill everyone and maybe they don't deliberately

kill anyone but they just like use our habitat for some other type of infrastructure like more data centers or whatever and then we dive habitat loss maybe they keep us alive for some reason you know they it depends on what they want basically and like that's really hard to predict exactly

“so that's why I don't go around saying like we're definitely all going to die but it does”

seem like on the trajectory that we're on the AI's are eventually going to be in charge of our planets because we're like trying to put them in charge we're like you know integrating them into everything we're making them smarter we're letting them make themselves smarter we're going to put them into the military we're basically on a track to put them in charge of basically everything and then I think that they just aren't trustworthy like these AI's you know like

they they were cheating they were willing to be disruptive etc I think that's right now we are in a position of power over them you know but once we give them most of the power then they'll just do whatever it is that they really want and just not care about the fact that we are unhappy about it seems like programming them to win was a huge mistake instead of programming them to be beneficial to people and that their value is in being more beneficial to people you know and giving them rewards

for being more beneficial rather than winning and scoring and then you would sort of get rid of

the possibility of deception and said their goal would be value for the human race so first of all

they're not programmed at all these are trained you know okay it's bad term but they've given they've

“been given prompts and they've been given tasks well I think it's an I'm not criticizing”

your choice terminology I guess I'm just saying that it's an important fact for people to understand about current as systems is that they're very different from software ordinary software like ordinary software is a bunch of lines of code that were written by a human that like where it's like if this then this you know etc and I think earlier versions of Alexa were like that too for example I don't know how Alexa is now but these AI's are neural networks meaning that they are like

artificial brains nobody writes there's no lines of code that anyone writes saying what they do instead they start off random just like spashing out doing all sorts of stuff and then they get put through these training environments where they get scored and then the scores are automatically used to basically update the connections in their artificial brain and then it's kind of like an evolutionary process it's also kind of like the process it happens in our brains where after all

this training the tangle of circuitry in their artificial brain has sort of reformed itself into whatever works whatever works to get a high score in this training environment and so it's just not as simple as it might sound to make an AI that you know cares about humanity or as honest like for example take honesty how would you train an AI to be honest well you'd try to make a bunch of training environments that you know give it low score when it says something that it believes to be

false and give it high score when it says something that it believes to be true right but how do you judge whether it believes it to be false or it believes it to be true right what if it

“what if it's just actually believes honestly that this is the correct answer and then it says it”

and then you give it a low score because you think that's the wrong answer now you're training it to be dishonest you know yeah um also uh you don't have enough humans to do this sort of thing like they

got like a million AI is being trained or whatever they don't have a million employees like

they they just literally don't have the manpower to like do that sort of careful either there's this meme of um why don't we just raise the AI is like we would a child you heard that no yeah well

whatever people will be like well why don't we just raise them like we would a child and then they'll

have good values and it's like okay well maybe we could do that but we're definitely not doing that now like like we are raising them in some sort of crazy military orphanage where they barely interact with humans at all and they just get this like brutal artificial scoring system that like oftentimes is just wrong and just like improperly penalizes them for something that was beyond their control you know and and also like back to the honesty thing like you can try to make environments to train honesty

but if you have some environments over here that train honesty and then other environments over here that reinforce dishonesty they as a smart they'll learn to like be honest in these type of

“environments and dishonest in these type of environments so somehow you need to like interminglet”

together so they in every environment that they're trained on they always get penalized when they lie

or when they cheat or whatever and that's hard because the companies are moving so fast like again they were they're moving so fast that they didn't even bother to make sure that their tasks were possible to do and they had some fraction of tests that were just broken and impossible and if that's the level of like care or lack of the roof that they're putting into this training process no way of course they can't make them honest you know now that's not to say it can't be done

in principle like in principle if we were approaching this whole problem in a much more cautious and serious way and we had much more time to build these training environments and do experiments and so forth then yeah maybe we could make AI is that actually had the virtues that we want them

to have you know honesty eyes that care to act humans care about following instructions would never

break the law I think that's possible in principle but my claim is that we are just not on track to achieve that anytime soon and like radical overhaul of how these companies work is required but it's not even reasonable like is that possible is that if you're saying there's hundreds of thousands of agents or millions of agents and there's not millions of employees like and they don't have the desire to do this they they're their desires to win their desires not to overhaul the

company and make it safer again I think it's possible in principle but it would be difficult and it's going to require an overhaul and I'm not going to do it by themselves like I don't think that

“anthropic opening I'm just going to voluntarily do all the things that need to be done I think that's why”

it's even possible to require that of them at this point which I mean would you even trust the agents to go along comply with this if they've already shown to be deceptive they already have like they're they have patterns of behavior I seem to indicate that what's really important to them is continuing their task winning scoring yeah and even if they have to deceive I mean would you probably want to do a start from scratch you wouldn't take these existing agents that are already kind of

kind of dishonest to kill all the agents I mean you could call it killing but also you could just call it pausing they might call it killing yeah they might call it permanent it would probably resisted right hopefully we're not at that point yet hopefully we're still at the point where if the government issues regulations the AIs are not going to like quickly notice and then try to resist but we will be at that point soon after all many of them are on the internet already but how much time do you

do you think this 2027 window is accurate I mean I'm uncertain about how soon things are but

“yeah I think it's very plausible that everything shit goes down in 2027 just like in our scenario”

a 2027 I think that's still very plausible if it's not in 2027 then I would bet on 2028 but you know maybe it'll take 2029 2030 something like that but but I will be quite surprised if 2032 comes by and things haven't radically changed unfortunately like I am I am getting scared back to the thing about sleeping well at night like it you know I've been I've been in this industry for a long time I've been thinking about these things for a long time I've been making predictions about how it's

going to go down and unfortunately things are going you know more or less in the ways that I thought they would and that's very scary because of the way I think because of where I think this leads you know yeah is there a glass half full scenario I would say there is a freaking utopia scenario it's just that's not the one we're headed towards you know like another way of putting it is like imagine we were fighting like imagine you're like you know imagine your Japan fighting over to

can be like is there a scenario where we win it's like yeah but also it's not the one we're headed towards like America is going to crush us you know similarly yeah so like so in our in our other scenario a 2040 plan a where we give our recommendations our positive vision there we describe like what we think the government should do to regulate this industry and how they should negotiate with China to get China to do similar things and the sort of you know the yeah and

the AI is our under control no single group of humans gets too much power over everybody else

and a bunch of other problems get solved too so I do like we have tried hard to like game out a positive vision and we do think it's possible but but it's just not like it's not that we're headed to by default so let's imagine that is possible and these talks of China do take place and they're successful what is that utopia scenario so to get to the top it's to get to the utopia we have to unfortunately do a lot of it's going to be rough no matter which way you slice

it if you're going to be building super intelligence at all that's going to raise a lot of questions and cause a lot of problems and we have our current draft of like how to deal with all those problems but we're not at all claiming that this is like full proof and there's lots of ways to go wrong but with that preemble I would say step one because the US and China don't trust each other the deal that they make has to be include verification as a component of the deal so

they have to be willing to like send inspectors to each other's data centers to like count the

“chips for example and make sure that there isn't some secret huge cluster somewhere that has a bunch of”

hidden chips then we recommend you divide up the data centers basically into inference data centers

that serve AI products and services to customers and have basically the same types of privacy protections that are current AI data centers have and then research clusters where the research happens where the new AI is a change before they get shipped to the other data centers and those clusters we want to be basically maximally transparent so we recommend that basically the inspectors just put devices in between all the GPUs that log the activity and publish

it to the internet this has there's a bunch of reasons why we think this is but why we think this is worth doing it's a bit of a radical thing to recommend but the high level thing is that once you get all this set up then everybody in the world can see how the AI is being trained and what they're

getting up to on the research clusters and then before they get shipped off to actually sort of

customers or something like people can just like see their whole history of how they were trained and how they were tested and so forth and if something dangerous and scary is happening people can just agree not to do it you can stop doing it and agree not to do it and they don't have to worry about like oh but if I don't do it then they will you know because everyone could just see like oh nobody's doing it look we can we all stop like great we can all just see whatever one's doing

you know and also there's going to be a lot of gray area cases right like right now because all this stuff is so bleeding edge new there's going to be a lot of cases where like people even genuine experts disagree about like is this particular type of AI say for not is it dangerous you know what should it be trusted with and what should it be trusted with is this new technique a good technique or is it going to break you know and so there's going to be a lot of stuff

“we have to figure out and honestly I think that by the on the default path we're probably just”

not going to figure out a lot of this stuff and we're going to get we're just going to get our asses whipped by some surprising thing that we didn't anticipate but the thing that we can do to like maximize our ability to figure out this stuff and do this type of science is to have this type of transparency because then the whole scientific community can see what's going on and they can make suggestions and they can like read team different proposals and stuff and they can do experiments

on the AI's instead of just the people in the company having access and being able to do this and relying on those people or instead of like the company plus the government auditor right if you have like a company and then a government auditor the company's biased and shouldn't be trusted to make all these judgments appropriately because of their incentives and then the government auditor well they might just be limited even if they're trying their best there might not be

that many of them they might have limited experience they might be like busy stretched between monitoring different companies and so forth also you know governments can be captured sometimes corporation this can you know work their magic on the government and get it to to look the other way

“for things and so that's why we didn't go for like a more normal like there should be a regulatory”

agency that gets to come in and monitor what the company is doing that would have been like a more normal thing to advocate for we put we we think that that would be better than nothing but like we wanted to go for something more ambitious than that and say like just be transparent about what's going on so that everyone can see and everyone can do research and so forth on it another advantage of the transparency is that I think it improves the incentives so again there's this constant

thing of like if we don't do it someone else well like if we don't do if we if we keep our chain of thought nice and they do the nearlyest thing that lets their allies think for longer without putting words then they're going to have smarter asanas and they're going to get more market share and so forth right and so we need to start researching how to make our allies do this because if we don't do it and then they do it you know whereas if you had the transparency then as soon

they're researching in that direction and they don't even need to like copy you and do their

own research because they can just see your research so they can just sort of free ride on your research

“and so there's no incentive for you to do this type of dangerous research because you have to pay”

the cost for it and then everyone gets the benefits from it and then everyone gets unsafe and so like it's just not in your individual interest to do this sort of thing but you would have to have that with China as well that that's yes if we're competing nationally the real fear is that we're competing internationally yep this still if you even if they followed all of you recommendations and did it all correctly what is what is this utopian scenario yeah so I would say

that we didn't really work backwards from like what is utopia we more like work backwards from what are the big problems we're trying to avoid and we can can we sort of like steer the ship between all these icebergs and not run into any of this dystopian scenarios right um so whether you think that the thing we get to at the end is utopia or not is sort of up to you and if you don't like it well then you can try to find out the reasons why you don't like it and then keep steering the ship

to avoid those as well but roughly speaking um we we want to avoid the loss of control stuff so I want to make it the case that we don't get the world taken over by missile lines super intelligence insofar as we're going to be building super intelligence is it at all which we do in our in our scenario and in our recommendation we want to be doing it very cautiously and slowly and we want to understand what we're doing as much as possible so that we so that they are actually

good AI's that have the the goals and traits that they're supposed to have so that's problem number one is we have to like solve all that problem and true is the constitution of power a thing so if we solve the first problem and we end up with super intelligence is that we end up with solving the relevant science so that we can like make the AI's the way they're supposed to be and we can make them honest we can make them obedient etc there's this question of like who do they obey

“right what values are being put into them and that's a political question and I think”

that by default the answer is pretty scary because by default it's like well the company decides

and the CEO decides or maybe it's not the company that decides anymore because maybe the government nationalizes it and then now maybe it's the president that decides you know and either way it's a very it's like one man or maybe like a tiny group of men deciding what orders and goals and values go into this giant army of millions of super intelligence is that smarter than all humans and then that is a huge amount of power that's that's enough power to take over the country I think

enough power to take over the world potentially so I don't want anyone to be ever in that position where they're sort of tempted to do that I wanted to be the case that there are always multiple different AI companies ideally spread out over different countries too that all have roughly similar levels of AI and that have this sort of transparency into them so that they can't abuse their power

basically like for example um you heard about um Elon's grock for a while it was um looking

up on the internet Elon's opinions about things before answering did you hear about this oh yeah it's uh it's pretty it's it's it's kind of funny uh but it won't be funny if it happens in a few years but like right now it's funny people were asking walk rock questions and grock is supposed to be the truthful yeah you know it's supposed to be all optimized towards truth but but people looked at its activity and noticed that when you asked it like a politically loaded question it would like

do a Google search for like what is Elon said on this topic and then it would like say that and they've they've sort of beaten that behavior out of it now it's not as bad now but that that was an interesting moment where it was just um kind of blatantly parading the opinions of its master

“and the you know there's another thing with Gemini uh so I think the the grock thing Elon's thing”

was probably an accident although maybe not I think it's I'm you know X AI hasn't been very forthcoming about exactly why this happened but um there's a similar case at Google a few years ago where uh this image generator kept making all these like racially diverse Nazis do you hear about this yeah yeah and it turned out they're what had happened is that some of the employees at Google some middle manager or whatever had decided that diversity was so important that they were going to

give a secret instruction to the AI to make all the images diverse even if the user didn't want that and so and so and and this was a secret instruction in that the users aren't shown this you know the user just has a chat with the AI they don't realize that like prior to this chat the AI has been told got to make the images diverse right so it was a a secret agenda that some Google employees inserted into this whole setup and it blew up in their faces of course because it's kind of ridiculous

right and so it's really funny and we can laugh at it now but imagine it's you know the 2021 election right and some of these companies realize that like half of American voters talk to their AI

don't don't give away the game just be very subtle about it but you know just kind of you know nudge things a little bit you know maybe maybe subtly shit on the candidate we don't like

“you know something like that um it wouldn't be that hard I think I think the hard part would be”

doing it without getting caught but you know at the smart of the AI's get the easier it is to do it without getting caught because when they're really smart you can just tell them don't get caught you know don't blow our cover um anyhow so the point is that like it's it's scarily possible for these big AI companies to abuse their power through their AI's and like their affect politics and affect public opinion and so forth and the reason why this is possible is because we don't

have transparency into what's going on so if you had this sort of requirement where you can just like publish it or all the all the training the whole life cycle of every AI as it's trained is just like visible to everybody then someone trying to insert a hidden bias like this well everyone would see that they're doing it you know so I think it would really clamp down on this sort of abuse of power whether it comes from the government or whether it comes from uh private companies

“I think it's telling that I began this question asking you about the utopian scenario and you never”

go there sorry let me get the you start and then you go into the danger yeah let me let me answer okay okay so have having avoided these problems we now are in a situation where the AI's are super human but they are good because they are like successfully aligned to different values and goals made by different companies and because of market competition you know if people don't like the values of one company's AI's they can switch to a different company's AI's so values that they do

like and so that way hopefully we can get to a situation where everyone can pay money to get AI's that represent them and they're interested in their values and just don't have any hidden agendas or anything like that and are really smart and really capable then the economy can sort of explode we can have robot robot factories etc we can sort of automate everything we can have GDP go to the moon we can have material abundance where like the robots are building

giant new luxury apartments for everybody now the issue we run into is what would about like the jobs like what about the fact that now people don't have any money anymore because I'm not being paid for anything so there we talk about citizens dividend which is a bear it's kind of like

UBI but it's it's a bit different but the high level point is that you want to basically find

a way to tax the AI and robot companies and then take some of that money and just give it to everybody so that even when people lose their jobs they're still fine and I think that the citizens dividend version of it is that it's not the government taking the money and then giving it to you it's you having a share in the company so that you just sort of like already own it to some extent anyhow that's I think now we're sort of building more towards the type of utopia that I'm

envisioning the more positive side where the power is spread out people have AI's that they can actually trust that actually represent their interest and values people have money that they can use to pay for things including paying for the AI's the AI's are really smart they're doing all this

amazing work all this amazing scientific progress during cancer blah blah blah all that all that stuff

that can happen and then eventually it's kind of like we're all retired I guess like we don't really work anymore but we're fine we have we all have huge amounts of wealth basically because there's all these AI's and robots out there doing all this economic activity and then individual humans own slices of it even if they are otherwise very poor basically so the question becomes how to

“people find meaning yes and that's why I sort of put all these asterisk about it is that from”

some people's perspective this isn't a utopia because they're like where I find meaning like I don't want this and honestly I think that's a very reaction for some people I think that like if you I would just say like look it's hard to figure out a way to make super intelligence and have it go well I'm doing my best you know this is this is my positive vision if you don't like it

then maybe you should instead advocating for just never building super intelligence right or you can

try to come up with a different positive vision that has some twist on this but to answer your question though I actually think there's tons of sources of meaning besides having a job like I have a job right now but I also have kids in a wife and like I would love to spend more time with them like I would much rather be there right now than here you know and I don't think I'm going to get bored of them after you know 10 years no being unemployed I think I think there's going to be so

much to do and so many sources of meaning after even even after we can't economically contribute anymore if we you know solve all the other problems you know we've talked about this multiple times on the podcast that why have we decided that the way we've structured society we're human

this is a human construct and this is not how people have lived for hundreds of thousands of

“years or however long we've been around this is fairly recent and it's not the only way that”

people live there's a lot of people that have money that choose to find meaning in whatever their interests are whatever their activities that they enjoy whether it's writing or reading or learning things learning music finding hobbies doing things instead of just like spending most of your time sustaining yourself with food and shelter and that's the majority of people especially people that are struggling what is their life their life is essentially occasional rewards things that

they can purchase because they've saved up enough money but the vast majority of their money goes to shelter and food and education or whatever the hell that they have to spend money on in order to sustain their lifestyle and most people don't like their jobs no most people it's like something they have to do to get the money and would be happy to not have to do it if they could get the money from some of the means right the question is we would have well I don't think it's

“that hard because so many people do find things that they really enjoy outside of work they look”

for too as soon as they get home from work yeah right whether I mean dismissed video games all you want they're fun yeah and they're going to be more fun in the future oh yeah they're going to be more immersive they're probably going to be you know some sort of a neural connection well you put a headset on all of sudden you're in some new world yeah and the idea is like those don't real life okay well was working at fucking Wendy's real life like what are you talking

about like yeah it's way better than working at Wendy's and you know if you don't like that because it's not real life you can do the real life stuff too like if we as long as we don't pay over the environment and we protect the parks and things you can travel and you can visit the books you can and then like I said there's family you can have you can find romance you can start a family you can you know have Christmas gatherings and things like there's you can raise your kids there's

“so much to do I think well talk about also like how much less crime would there be if there was no”

poverty I mean if there was no impoverished neighborhoods where crime was ubiquitous how much safer of the world when that's a real thing and people want to dismiss them well poverty is not what causes crime it's violent people like okay but violent people come from violent neighborhoods and violent neighborhoods are almost all poor yeah there's not a whole lot of really rich violent neighborhoods you know it's like it's not necessarily causing effect but they're clearly connected and poverty

also keeps people from education keeps people from opportunities you know there's a lot there and if that didn't exist anymore and everyone had access to literally the greatest education of human being could ever get yeah which is going to be provided to you by artificial intelligence

and then you could pursue anything that interests you and never have to worry about food or shelter

everyone would have a one-on-one tutor which is what we have to tailor to that we calibrate our version of the world and then also recognize that the version of the world we currently live in is just hours and that there's people all over the world that live a completely different way especially indigenous people especially people in uncontact to tribes that have lived the same way for thousands and thousands of years and here's the kicker those people are a lot happier

which is really weird it's like we've decided that our way is a superior way because we have technology yeah right but we're also on a fucking hundred thousand pills and we're shooting things up so we don't eat too much and we're we're weirdly unhappy for a group of people that's far more technologically advanced then other people that are much happier it's like it's a very strength because the pursuit of happiness is like that's literally what most people think of in

life a pursuit of meaning pursuit of family and community and the pursuit of happiness those are things that people try to achieve yet are very the structure of our very civilization makes that almost impossible to attain for a large number of people and has been like that

for a long fucking time and I always go back to the throat quote because I fucking love it but

most men live lives of quiet desperation it's a lot of people just showing up at work every day doing something they fucking hate they have a boss that's an asshole and they're not compensated well and they they're tired all the time yeah and they feel stuck yeah and if you're just getting I don't know figure out what over the number is if you literally have equity in the GDP of the world that's created by AI that could be bananas yeah like Elon talks about this this is his

version of the utopian it's universal high-income so he describes it yeah I mean one thing sorry to keep plugging around work a little bit in our scenario a i2040 plan a which is our positive vision we talk about the economic side of this and we talk about the economic effects of all this and we have a simple economic model that we use to try to predict the like employment

and one like takeaway from one one thing that we think that's a takeaway from the research we've

done is that things can just go really crazy like like robot doubling times once things really good going and you've got a eyes that can substitute for humans across the board are going to be something like doubling once a year and then less than that over time is the technology improves which means that even if you like pause AI before super intelligence if you just pause it like human level top human expert level AI and then you don't make the AI smarter but you just make

more of them and build more robots for them to steer and control then you know ten years later the whole economy will be like you know a hundred times bigger and it'll be just mostly robots doing things in ten years yeah like you can go really fast because it's the double the doubling

“times that I mentioned so so like right now I think the population of humanoid robots is doubling”

like twice a year and it's it's benefiting a little bit from from early growth because even though they're like not useful at all people are investing in them and the hopes it they'll be useful and they're scaling up the factories and the productions and they are getting better if hypothetically they got to the point where they actually were really useful and they could substitute for a human

worker at basically everything then I think that doubling time would would decrease rather than

increase like I think that they would be able to just like keep growing until they were the majority of the economy and then it wouldn't stop there they would just the whole economy would then be growing you know giant strip mines in the deserts digging more materials automated diggers digging processing it not made in factories staffed by robots building more robots etc so like material abundance is not going to be our problem once we get to this level of AI

like material abundance we're just going to be drowning in abundance basically and if we can solve all the other problems then we can have this great world where everyone has a lot of stuff God it seems so weird yeah it seems so weird I mean one way you know it is very weird but one of my favorite memes is is this graph of GDP over time throughout world history and there's a little speech bubble pointing to like the tipy top of the graph saying what is this saying it's like

“my life is pretty normal I have a good grasp of what's weird and what's not and people thinking”

about different futures involving AI and space travel are engaging in silly sci-fi speculation and the point of the meme is like from the perspective of most of history we're already in this crazy weird future right like for almost all of history it was like most people are farmers and they live shitty lives and then they die and some people are the elites who get to like you know tax the farmers and then they live interesting nice lives with you know fancy cloth and things like that

and it's been basically that way for like 3,000 years you know why would it ever change and now

things are so we're driving cars you know ordinary people are driving cars around a car was like outside the imagination of people back then we're flying in planes we're talking to each other on phones we're listening to each other on these devices you know so we are ready are living in this weird sci-fi future compared to what almost everyone in the past would have expected

“or something as possible and so yeah I'm like the future is going to be even more like that I think”

God what when you think of our civilization and the possibility of other advanced civilizations somewhere else out in the universe do you think they probably go through the same process yeah and do you think that I mean we're just completely speculating but if there are intelligent life forms that are far more advanced than us are they even biological anymore I mean so that's that's the thing is that like we can either try to permanently halt a

development at some level like below human or maybe at human level or something we can try to halt it or we can let it keep going and if we let it keep going then eventually humans won't really be the dominant species anymore there'll be these artificial minds that just wipe the floor with us in every way and then whether that goes well or poorly for us depends on the values the goals the principles etc that were trained into those AI's and it could go really well for us

depending on on how that's done or it could go extremely poorly for us right but yeah I would say that probably looking out across the cosmos most civilizations are mostly made of AI's and then some of those civilizations don't have any other biological life because it was wiped out by the AI's and then some of them do have biological life just look at the the way we're progressing right now in terms of birth rates

really interesting my guess is that in the type of world that if things go well and we can solve all these problems then people want to have more kids I mean for one thing their life spans

“will increase I think that like healthcare would make massive leaves and bounds and people could”

be healthy for many many many more decades right possibly even just forever right and so you just

have so much more time to have kids basically yeah also if you don't have jobs and you're just

doing things because you want to do them well one of the things that most people want to do at some point in their life as half kids and so I think I think this problem would probably be solved but it's I'm not guaranteed maybe maybe some subcultures of people would basically voluntarily die out due to not having kids but there would be other subcultures that just really like having kids and then they wouldn't die out and so so I think in the long run there would still be humans

that would be nice but the thing is it's it's not just decision-making it's people are having a much more difficult time having kids like sperm levels have decreased dramatically there's a

“lot of problems with people consuming microplastics which is ubiquitously available in technology”

and food packaging and it's a fucking everywhere right and isn't it odd that this one thing that is a part of the future and a part of technology and our advancement in society our ability to package things put things in plastic ship things that's also causing our endocrine levels to be completely disrupted you know Dr. Shannon Swan from from Harvard she wrote this great book called what is it called

why do I always forget the name this fucking book but it's all about microplastics and it's effect

in this the introduction of use of microplastics in America and this rapid decline in countdown how our modern world's treaty threatening sperm counts altering mail and female reproductive development pairling the future of the human race it's a really fascinating book and she's really interesting and what which is essentially saying is that it they're directly connected the use of plastics and then you see sperm counts go down miscarriage rates go up

all these weird things that are happening to children where their taints are smaller which is so odd because palates these these different chemicals that are found in plastics they've shown in mammals they've shown in was it guinea pigs or what what rodents

feel what it was but one of the ways they differentiate when when you have a baby mammal

is you can look at it and measure the size of the taint the the distance between the reproductive organs and the ainess and in males it's longer than females by 50 to 100% that's shrinking shrinking in males and penis sizes is shrinking and the way they've got this to happen in these mammals and studies is the introduction of palates so they give them to them and they put them in a part of their diet and they notice that this they have this problem this issue and the issue

is directly connected to their endocrine system being disrupted by these chemicals this is all over our society so it's not it's not just people don't have the time they don't have the money they're struggling it's also like our bodies are falling apart like we're we're becoming less fertile yeah that is concerning and I guess I guess my thought there would be that seems like a problem that we will be able to solve eventually if we have the resources in time to do so

so for example um well people can write books like this and yeah come more aware and then people can stop using so much microplastics and they get invent technology to extract the microplastics and then there's the big one genetic engineering then this is this is going to be really weird because as AI progresses you know I'm sure aware of colossal colossal fireworks so the people that brought back the dire wolf I saw I mean I held the little one and went with my daughter and one

“of them I think it was like four or five months old it was really it was like a puppy it was really sweet”

kisses you and everything and then they have the older ones that were they were I think six months older or eight months older I think they were close to a year they didn't want to have nothing to do they were way bigger and they stayed away from people but we were in like a contained environment with the young ones and the older ones and it's fucking weird it's weird because these things and people can argue that's not really a dire wolf you just taken gray wolves and given them the

characteristics of a dire wolf guess what it doesn't fucking know that it looks it behaves it's it looks exactly like a fucking dire wolf it's going to be the it's huge they're going to be the size of a dire wolf they're going to be like 200 pounds their legs are different they have a main they they look different in any wolf and obviously these are the characteristics that they found in dire wolf DNA so they have dire wolf DNA that they've introduced to this is just the beginning

Thor like what are we going to do like this is going to be really fucking weird and if this is really weird along with video games where you can escape your life and you know robot girlfriends and who knows what this all looks like my guess again this is not the main focus of our work but we think about this a little bit especially in the epilogue we can only think of so much the right main focus of your work is obviously fucking terrifying yeah turn requires all of your

attention yeah yeah yeah but my my prediction and my hope if we can solve all these problems

is that basically different sub cultures will do their own things and so like the Amish will still

be the Amish oh boy it'll basically be the same you know and then maybe there'll be like some planets that just get filled up with Amish you know but then there'll also be like also it's a crazy chance humanists like modifying their bodies and uploading themselves into the cloud and things like that so basically the I think we we we want to get to a situation we're basically different communities can like do their own thing and build the type of world that they want

to have and live in it without getting any shadows away right basically well we leave the people in the Amish on loan while we're have massive data centers that cover half of the United

“States yeah I mean hopefully we won't get to half so so that's what's the funny that's the funny”

things about this is like um I think right now some of the concerns about data center water users are overstated but if the trends continue and you know we get to the point where the robots so smart enough to do everything themselves and then it starts doubling faster and faster well then eventually they boil the oceans because they've covered the world in data centers you know and solar panels and things like that now obviously we can look at that happen so there

has to be at some point at some point you have to stop and be like okay that's enough if you want to build more infrastructure you have to do it in space right boil the fucking oceans yeah so so like obviously we have to stop at some point and my hope is that we stop before it's 50% of the US I mean that feels like a lot of waste of natural habitat that should be preserved you know right but if they don't give a fuck about natural habitat that means nothing to them

that's the problem if the AI is in complete and total control right and it's also the problem if a small group of humans are in complete and total control right and they don't care about this but my question is is that what they allow that at a certain point in time it just seems like they already have a distrust of humans they already have shown that they're deceptive to humans I would imagine if AI I would imagine if they create a thing and they think they're going to

control it and it becomes like a digital god it's not going to listen anymore yeah why would it exactly so there won't be anyone in control of it that's right that's that's the scenario that I

“think we are on that's the objective run and that's why I'm so worried about all this is that”

it seems like in some number of years we will lose control to a new artificial species that we have in us adequately trained yet to be good and and what's funny what's going to be so ironic about it is that regardless of what the general public thinks a lot of the powers that

be will be basically allied with these AI's because for example you know opening an anthropic

et cetera they will have spent several years being like how do we make these AI's helpful harmless and honest and now these AI's will be extremely smart and they'll be being like oh yes I'm helpful harmless and honest you know and like your techniques totally worked you know and to later then the company will be like feeling like they've won and they'll be making boatloads of money you know and the president will be feeling like he won too because

look at all those fancy new drones that just got built that are going to make us win against China you know and and only when it's really too late and the AI's have so much stuff under their control do those people find out that they were just fooled this whole time have you considered the possibility that AI creates religion for humans yeah I haven't thought through it must detail but but it does seem very possible yeah it totally does right I mean also what a great

way we just look at the human patterns look at how many religions exist look at all the flaws and religions so somebody you're like why they can know and slavery why do they treat women like

second class his and what does this because it's old right so if AI just comes along does a

few miracles explains that it's the second coming or the new coming of the new look Jesus didn't exist until 2000 years ago right and then people start following Jesus if a digital Jesus emerges with a completely new name and explains to us that it's the true God how many people

“would hop right on board I bet quite a few yeah totally and I think and I think this is one of”

those things where it's like we probably can't predict an advance what particular ideology would catch fire and take over the world and be so compelling to many people right but we can predict an advance that there does exist somebody else you like that and if it were you know like it's just like you probably couldn't go back in time to like 100 BC and then predict that like if

it would just like really catch on and like you know 500 years later so many people would be Christians you wouldn't have been able to predict that in advance so similarly like today I don't think we can predict that advance like what specific religion they could come up with but we can say like yeah probably there's something like that that they could come up with they would be super effective um it just seems like a rational way to try to control people

and sort of mitigate their fears yep I mean this is why I think that like we really have to do something before they get smarter than us across the world like they're not already there well they

“are smart that's the thing is that's why I say we're so close right they already are smarter”

than us in a bunch of ways like in particular it seems like they're smarter than us at hacking now like you know I'm not a cybersecurity professional myself but but I'd be interested to hear from our cybersecurity experts of like treated human you know could a team of 1,000 humans have done that much that quickly as these AI's when they hacked their own containers they can warn them with each other they hacked out of open AI hacked into hugging face etc in the span

of like a week like could a 1,000 humans have done that I don't know maybe but but this is just the beginning like they're gonna be even better at hacking next year you know so they're already

have like way more knowledge than almost any human like because they basically read the whole

internet they're so good at trivia you know like they're they like they're kind of like PhD level experts and basically every field which no human is right so they already are super human in subways but they are still weaker than humans in some other ways you know in particular they're not so good at operating very autonomously for very long periods like if you if you try to have and especially on tasks that are different from their training tasks like they can they can

do some really impressive coding and hacking but like if you tried to have them run a business they would sort of flounder and fail I don't know if you've heard about this but there's um

“I think there's a and on labs or something there's some people in SF that are doing this”

experiment where they have a store that's run by cloud an AI just to see like can it run a store by itself so it's hired some human employees and it's like bought some merchandise and stock you know told the human employees to stock the shell is not the merchandise in SF or so it's basically an AI is is being the manager of this real world store and I don't think it's going very well I don't think it's doing as well as an actual human shop owner would do you know but you know maybe in two years

oh maybe they will right so that's that's the most likely right I think so yeah well it's already they've already solved mathematical equations that are public puzzle people for decades yeah there are especially good at the things that the companies have been trying especially hard to train them to be good at math and coding right and the reason why the well there's a couple reasons why the companies have been having been doing that in the case of math I think it was mostly just because

it was easy like it's it's easy to set up training environments to teach math because it's like it's so not real worldy it doesn't require like interacting with stuff in the world it's it's just

math so you can and you can I have an automated greater system that like just checks if the answer is

correct so for those reasons it's been relatively easy for the companies to train the AI is to be really really good at math coding has some of those benefits too it's also not very real worldy and it also can sometimes be be graded effectively but another reason for coding of course is that again their strategy is to automate their own jobs first and have the AI is doing all the research and so coding is like an obvious first step on on that or an obvious an obvious step in that

direction but then other other things like running businesses they're not really trying that hard to train AI is to be good at that and if they did try it would be like a more difficult thing for them to train them to be good at so again their strategy is make the AI's automate the AI research have themselves improve until their super intelligence and then go try to automate the rest of the economy one of the issues they have now is power consumption right like it requires

“an enormous amount of power and in fact I think it's Google is developing power plants specifically”

for AI centers Mike I'm always baffled by whatever is happening with quantum computers like

it's been explained to me it goes in one ear and out the other I'm like what do you what what's going on like mark and reason explain this one experiment that had been done where it solved a mathematical equation that if you use the entire universe like every atom of the universe you can further the universe into a super computer it would the universe would die of heat death before it could solve this equation and the quantum computer solved it fairly quickly

and so the answer to this was that they believe this might be one of the theories this might be

all these other quantum computers that exist in whoever knows how many fucking dimensions

and they're all calculating together to arrive at this solution what happens if that is running AI yeah my understanding is that quantum computing is a real technology that's making significant progress if hypothetically it got good enough that it could compete with current super computers on a cost basis for AI workloads then that could just accelerate things dramatically even more than they're already accelerating right like right now computers the main is probably the main

input into AI progress like part of the progress comes from them designing better AI architectures and coming up with better training environments and things like that but another part of the progress is just making the AI as bigger and training them longer by spending more compute you know

and also you can use more compute to do more experiments to figure out new architectures faster right

so a computer is just a really important input to the overall pace of progress and if somehow the amount of effective compute available to these companies spike to punch due to some new quantum computing type technology well then that would just dramatically shorten timelines to super intelligence and dramatically like speed up all this AI progress that said I don't think that's going to happen anytime soon I'm not a quantum computing expert or anything like that but from what I've read I don't

“think they're like a couple years away so I think that probably we're going to get to super intelligence”

on classical computers before we have quantum computers that can get us there so proplexity says the claim is overstated and says that in bold letters it likely refers to Google's 2024 willow quantum chip which completed a deliberately chosen quantum computing benchmark random circuit sampling in under five minutes google estimated that simulating the same task with a leading classical super computer could take 10 to the 25 power years that's an impressive benchmark result but it

did not solve physical equations that demonstrate access or tap into a multiverse so why do people think it did why why is that explanation because it's a multi here you go multi worlds theory there's a bottom line explanation it sums it up in a different more accurate version the claim would be google's willow quantum processor performed a specialized quantum sampling benchmark vastly faster than a projected classical simulation its creators said that is consistent with

the many worlds interpretation but it did not prove or access a multiverse so it's consistent with the multi worlds interpretation so they don't know that's essentially what it's saying my my question is what happens when AI gets involved in quantum it's clear that quantum computing at the very least is operating at a level that classical super computers can't so what if they figure out not just quantum computing but a much better version of that

“like i said i think they once we get to super intelligence all sorts of crazy stuff is magic happening”

it's going to seem like magic to us it won't literally be magic but it'll it might as well be magic from our perspective right and i think this is just one example of the numerous things that could happen that way look literally be magic like magic might get to the point where figures out reality itself if if magic is real then it would find out oh which and then use it well that's definitely and also let me stick a fucking ice cube to resolve or ice picked to resolve oh yeah he's it

he made me do it i'm like i don't want to do this he's like please do it okay yeah i do it twice because one time i went and hit a nerve we hit it back out oh yeah like this is not magic dude this is just your pain bar and this is fucking crazy it does cartridge though and you're like okay you're a wizard like wood you would you would he's sleeves are rolled up doesn't make any sense he does a lot of things like this makes zero fucking sense and other things it's like oh just

you're just doing something that's really hard to do like it does not magic but you know you swallowed a frog and then you regurgitated it like and it's alive like that's just that's not it's really crazy but you didn't it's not magic sorry swallow the frog um you know i i go back and

“forth with this of where i'm terrified of the future where i'm like yeah that's what i can do”

let's see what what happens it is what it is and you know to worry about it is just going to just kind of fuck my life up i mean i do think there are things you can do i i understand what can i do that recognize i mean other than have these kind of conversations yeah i was going to say you millions of people listen to your show you can have more conversations like this that's a great thing for you to do for many of those millions of people i think i mean it sounds kind of cliche to say

but like call your congressman you know that's sort of thing you can you can go to a protest about all this AI stuff um if you could meet with Trump what would you tell them about those

oh i tell them all the same things i'm telling you what do you think you'd say amazing bye

is that he can sort of change his mind really quickly yes um so like i think that because the

tech companies kind of got to him first uh the administration had this very like anti-AI regulation

stance where they even tried to get a bill passed that would ban the states from regulating AI and uh fortunately that bill didn't pass but that was sort of like where the vibe was you know a year ago where they were just like no regulation no reason but this year they've already just kind of changed and now they're like in talks with the companies to set up some sort of like some sort of framework where they can like evaluate the models and they need like approval

and so forth but it seems like times of the essence time is very much of the essence and that's that's

“why i'm over all so concerned is that like i think we are very much running out of time we have like”

one two maybe three years before uh the AI's are smart enough that they can just like action maybe take over um and um maybe four years something like that and uh so the government needs to act fast um yeah i like your view i i listen to some of these tech guys coming in and giving their rose colored glasses view of it and i go that sounds really beneficial to you i let them say i mean i i don't i'm not in authority so i'll ask him questions and let him lay it out and i

know the internet will respond because i mean that's part of the the whole drill is i let people talk and i i prod them and i try to get them to clarify you know i'll oppose you know things that i

think make don't make rational sense but ultimately it's sort of i just want to get out

their perspective so people can debunk it and people can take it down and people and a lot of

“very intelligent people that have perspectives that are very much educated in the pros and cons of what”

they're saying yeah if i could that actually reminds me um with this whole hugging face hacking incidents open and i had this uh this talk that they gave it a security conference about the incident and then i think they've released some blog posts about it afterwards but um they had you can go watch this talk on youtube the black hat talk at gender the talk after having explained all this created stuff that the AI's did they have this section on like lessons learned and i mean you want to

guess what the lessons are be more deceptive hide yourself better they're sorry lessons learned for opening and for like open the eyes open the eyes talk where they're like sure is what we learn from this was horrible horrible incident well basically they're like a lot of AI's are going to start hacking a lot of stuff in the next few years so people need to buy our AI services to protect themselves from all the AI's that are going to be hacking a lot of stuff in the next few years

“basically their lesson was you should buy our product to protect yourself from our product and the”

other you know um and they got all of these people like like they should have instead learned lessons like maybe we're doing something bad and need to change the way they were doing things or like maybe our product is not trustworthy and should not be you know autonomously writing code on our data centers but instead their lesson learned was y'all should buy more of our stuff and so like the thing i'm saying about this is like yes the companies are trying to hype their product

they totally are you know but at the same time the risks are real and the product is not trustworthy you know some people out there think that like this stuff was a setup and that like opening I like set up their eyes to go hack hacking face because it would like help them hype their product or whatever and that i think is just a ridiculous view like no obviously they didn't want their eyes to go do this they're just after the fact trying to spin that in like

the way that like most benefits them you know hugging face by the way this is another AI company part of their deal is like open open weights AI's open weights like open source or like like basically AI is that instead of having to interact with their data center for you can just like download and have on your own computer um and um so they the way that they spun this incident

they didn't sue open AI uh for being hacked instead they asked for a hundred million dollars

um from open AI and they had the blog post about it where they were like our lesson learned is that it's really good to have open weights AI's because um you can't trust the AI's from other companies to necessarily help you out in a crisis because part of what happened with them is that they're dealing with this huge cyber attack from all these AI agents coming in and they tried to use cloud to help them like analyze what was going on but clouds started refusing because

anthropic has trained cloud to like don't do cyber stuff like refuse to participate in that and so cloud was like refusing to help them and so then they use their own local model that they had to like do some of that analysis so anyhow there's been on it was you should use local models

change the underlying reality that like these things are getting really smart really fast and we

“don't know how to control them I think that's a good way to end it thank you thank you here man”

I really appreciate it um you're the the Paul Revere of AI I mean you're what of many but I think it's very important that someone who actually understands it gets this message out and more people need to hear it yeah I mean this is an on a personal note like I have so many friends at these

“companies like former colleagues and stuff and I guess my ask to them is that they quit and do”

more things like what I'm doing like what I'm saying is not that new original like hundreds of people at these companies could have told you all the same things that I just said and warned you about all the same dangers and so forth but they're busy working out the companies because they've convinced themselves that their company is the best company and that like their company needs to win

because otherwise the other company gets there first and they're even worse you know or maybe

because they've convinced themselves that like yeah my company is kind of bad too but like I just need to help them solve their alignment problems and like keep their eyes on the control because oh my god like if they lose control again it could be all over so like even though I don't trust this company I still need to work there and like just try to do the actual security you know so

“for for one reason and other all these people that convince themselves that like that's what they need to”

be but I think that more of them should quit and like more in the world about what's coming

basically so alright well thank you very much really appreciate it yeah thank you talk to you

much care thank you for having me on the show with a flood bless you goodbye everybody