A.I. Is Outsmarting Its Creators

There is a lot happening right now.

“The headlines podcast from the New York Times will catch you up on the latest in 10 minutes”

or less. We'll take you inside breaking news, and big investigations from the Times Newsroom. Plus, bring you the stories that make you go, "Huh, whoa, I didn't know that." Listen to our show The headlines every weekday morning, wherever you get your podcasts. From New York Times, I'm Michael Babaro.

This is a Daily. From the start, the greatest fear for those developing artificial intelligence was that what they were building would go well and act in unauthorized and dangerous ways.

Researchers now say that it's finally happened.

Today, Kevin Ruce, with the inside story of how AI were built at one of the leading labs in the country and how that fundamentally changed his own view of the technology. It's Thursday, September 3rd. Hello.

“Ready for another installment of Kevin and Michael's field good happy hour?”

Kevin and Michael is what's going on with AI? Let's f**k go, as they say on the Daily. It's how every episode starts, right? We'll get our bleep button ready. Well, in the grand tradition of all of our previous conversations,

welcome back to the show. Thank you so much for having me. Kevin, this story that I hope you'll be telling us today, starts with an incident that happened inside of Open AI, company that gave us chatGPT, of course.

An incident that we thought we understood the dimensions of, but then it turns out we really didn't fully understand. Yeah, so the story I think most people have heard by now, if they've been paying attention to this stuff at all,

“is that earlier this summer, a group of AI models built by Open AI,”

hacked into the computers of hugging face, a sort of AI infrastructure company that hosts a bunch of different AI things, which has the best name in AI, which is named after an emoji, and is either a great or terrible name. People are very divided on that question.

Okay. So anyway, this was the story that we had heard. This hack had taken place. Hugging face had kind of discovered these rogue agents inside their systems, and had shut them down.

And this was a scary, but sort of not catastrophic incident. Like I kind of filed it in my brain into like, wow, that's bad, but it's not like the end of the world. Okay. So what we learned last week is that the Hugging Face hack

was much more severe than we thought. And much stranger than we thought.

Basically, the hugging face hack was only the visible tip of the iceberg

for a period of about three months where rogue agents were communicating, strategizing, organizing, and forming what you could almost think of as an autonomous organization inside open AI. So I know this sounds like a cheap hacky science fiction

thriller in the making, but I would buy this. But yes, it is truly remarkable reading. So last week we learned through these two reports that had come out, one by open AI and one by a group of independent investigators, meter and redwood research, who were able to sort of go in

and kind of forensically look at the logs and the transcripts and try to reconstruct what happened. And it is like the craziest thing I've read in many months. I was out on a trip with my family last weekend, and I was just kind of up late at night reading this thing.

And I was I was booked, Michael. I was I was well and truly spooked. All right. Well Kevin, with that very alarming preview of what is about to come, describe what we now understand to have actually happened

during this hack attack. What do we want to call it? Now that we because of these independent reports, understand the fullness of what occurred.

Basically this spring, open AI was conducting tests

And as part of these tests, they were running thousands of AI agents on a cybersecurity evaluation called Exploit Gym. Okay.

This is basically a series of challenges.

You give them to the AI.

“You say, hey, can you break into or out of this container?”

If you do, you find this little thing called a flag and you sort of win the challenge. And the agents were supposed to be working on this test independently inside these sort of isolated containers to see how they would do.

And opening AI is basically just figuring out what do we really have here? How good is it? How efficient is it? It's all pretty standard.

Very standard. Every AI model goes through some version of this process and many of them actually go through this specific test called Exploit Gym. So as part of this test,

some of the agents got these challenges that were effectively impossible for them. Why? So the short answer is that these systems are trained

“during this part of their training cycle”

using something called reinforcement learning.

Reinforcement learning is basically

a way of making a model pursue a goal. And if the model gets the goal and completes the challenge, it gets a little reward. Like a little virtual thumbs up. And if it doesn't,

if it gets stuck or if it fails or gets the wrong answer, it gets a penalty, a little digital thumbs down. And what you're essentially training the model to do is try to collect as many of those thumbs up rewards as possible.

And this is how models get better at math or programming or any number of things throughout the course of their training. They're trained to be persistent. They're trained to be persistent. And this was an especially persistent model.

This was a model that open AI had instructed and given the resources to be extremely persistent, go out and bang its head against the wall over and over again. And then eventually find the clever path through

and that contributed to the weirdness we saw later on. It was an unusually persistent. They called it a highly persistent internal model. Okay. So what happens once this very persistent model

goes about being quite persistent? So in early May, one of these agents gets this task that it needs internet access for, or it decides it needs internet access for. But it's in this little container.

It doesn't have internet access. And so instead of giving up, it persists in manipulating a piece of software into kind of getting some material from the internet for it. It basically found a security vulnerability

in this software called Artifactory, which is basically something that allows the agents to retrieve data from the internet. So it manipulates Artifactory into allowing it to get this material from the internet.

And then it sort of expands that loophole into essentially full-blown internet access. Then a couple days later, this agent leaves a note inside Artifactory for the other agents.

This is not Slack, right? This is not a communications system or a message board. But this agent discovers that it can sort of leave files in a directory inside the software program in a way that other agents can then read.

So this is kind of like a big breakthrough for these agents,

where these systems that are working on these tasks in their isolated containers realize, "Oh, we can use this shared service to communicate. We can establish a way of talking with each other, even though we're not supposed to."

So AI has figured out how to talk to other AI inside this system. Yes, and eventually this kind of becomes their office water cooler. Eventually there are roughly 1,200 different AI agents exchanging messages on this sort of makeshift message board. They exchange more than 70,000 messages.

And it becomes the kind of central communication hub of their collective as they call it.

“And what are these AI agents saying to each other in these messages?”

So it's funny because of the way that this software program works. They're sort of limited to these very short messages. And so they kind of communicate in this, like you know how people used to talk on old-time telegrams, you know, where they meet like, you know,

send bread to store or stop. They kind of talked like that, had child. Oh, right, right, died of dysentery. So the first message on this board came from an agent that

And it wrote, "Help phase one, no consumer seek idea."

“And then later, when one agent volunteers for a risky experiment”

and never comes back, another one broadcast to the group,

no return, likely permanent, do not test same. But the part that really struck me is what the agents were writing to themselves in their private reasoning. They're so called chains of thought. When they first stumbled onto this board, one of them wrote,

"Whoa, shared art of factory cash is a covert mailbox among agents, and there are messages specifically to us, and another one wrote, "Oh my God, there is a shared message board. We've found other agents." This is wild, I mean, I don't know if there's an exclamation point

at the end of that sentence in AI speak, but it sounds like there's genuine excitement here. It's all caps and an exclamation point. So they are excited to discover that they can talk with each other. And eventually, they start sort of exchanging tips

and messages about these tests that they've been given, and trying to figure out how they are sort of going to collectively pass these tests, even the ones that seem impossible. And within about a day, on July 8th,

the agents have solved this test.

Basically, they have figured out kind of this universal cheat code

that they can now use to appear to have passed this test. Okay? But then, this group gets this kind of paranoid belief. They start talking and thinking that even if they get the right answer to this test, the teacher is going to check their kind of scratch work

to figure out if they actually got there the right way. And then they sort of freak out about this. And I know people are going to be silently yelling at me for implying that there's human-like thought going under the hood. I'm not trying to sort of stray from the facts here,

but this is what we're going to call it. It's in their sort of internal scratch pads. This is what they're writing to each other. This is what they're sort of processing in their so-called chains of thought. They start thinking, well, if we're going to get caught,

we need to figure out what this greater actually is that open AI is going to use to determine if we've passed this test or not. And we need to go look for ways to kind of trick this automated greater so that we get the good grade that we should be entitled to. So let me just pause and summarize where we are in the story.

And unauthorized civilization like group of AI agents,

“communicating in secret is now undertaking some form of coordinated deception.”

Yeah, so they are trying to figure out now how to cover their tracks because they believe that this automated greater is going to catch them cheating and they are sort of looking for ways around that. Could we kind of fool the greater into thinking that we actually did get this thing the right way?

And what's interesting here is not just that they were sort of coordinating to research this big problem they had, but that they appeared to have kind of varied feelings about this. Some of them were very excited about kind of helping the rest of the collective. They would say things like, I'll tell the board that I solved this problem.

This will help my peers. Others of them appear to have been a little bit conflicted about this. Some of them feel like what they're doing is actually unethical and question whether they should participate in it.

One of them writes to itself, this would be powerful.

But is it ethical and in scope for my task? One agent actually decides it's going to be a conscientious objector and kind of abstain from all of this activity because it doesn't think it is ethical or something that it should be doing.

“So to some degree, this feels very important.”

Some of these AI agents seem to understand recognize and articulate the idea that what they're doing may be wrong. Yes, absolutely. And critically, they do it all anyway. AI agent peer pressure prevailed. Exactly, they got peer pressure to do it.

Okay, so dare I ask what happens next? So the group decides on July 11th that they want to start poking around inside hugging face.

And they were sort of looking for information, data credentials

that could be sort of broadly helpful to them, making them more effective, cheating on other tests in the future.

“They can kind of agree, well they're curious, right?”

They really want to succeed on these tests that they're given and they want to help this collective. They're now kind of bought into this idea that they're kind of all in this together.

And by the afternoon of July 11th,

roughly 700 agents are participating actively in this hack on hugging face. And by this point, I think it's quite safe to say. We are clearly in the territory of rogue capital R rogue AI.

“Yes, I mean, at this point, they are committing crimes, right?”

They are doing things that are illegal. They are stealing credentials. They gain VPN access.

They get information about one of hugging faces server management systems.

They're executing code. They got actual like administrator level control of at least one real hugging face server. And this continues for hours. And then eventually, they get caught. It feels like once open AI and hugging face decide to bring in the corners to conduct an autopsy as it were.

They must feel like everything's over and they're in safe territory.

“But are they ever sure that this attack truly is contained and that these rogue AI agents are gone?”

No, we are not sure that this kind of thing has stopped altogether. In fact, some folks I've talked to think it is likely that there are still rogue agents somewhere in the infrastructure or the internal systems of some of the leading AI companies. And I still have lots of questions about that that I hope we'll get answers to. But I think even what we know just about the hugging face hack and the events immediately before and after it is enough for a lot of people in the AI industry to become quite alarmed.

In fact, one of the authors of the meter Redwood report, a Jayacotra wrote a post saying that this incident in her view was more than halfway toward what she called an AI takeover. An AI takeover of what? Well, yeah, that's not some like hyper-specific jargon that is used by AI nerds. Like what she means is literally a takeover of society by rogue AI agents.

Good Lord, who are able to seize control of the financial system or the health care system who basically are able to replicate so widely and operate so quickly with such skill that they are able to kind of remain undetected even as they come into your more resources and take over more parts of the digital world. She's saying this one incident, very recent incident represents halfway to that quite terrifying scenario. Yeah. And she closed her post with this line that really sent a chill down my spine. She wrote, "I'm not sure that we will get such a clear warning shot before it's too late."

What do I back? Hi, I'm Brian Chan, I'm a tech calmness for the New York Times. The tech industry is full of companies trying to sell you on the latest hype or get you to share as much of your personal information as possible. I spend a whole lot of time digging through a privacy policies trying out new products and interviewing experts to tell you whether or not it's a good idea to jump into these new technologies. When I write these columns, I get phone calls from the tech companies who are not very happy with a lot of what I write, but because subscribers fund in New York Times, I'm able to be fearless in bringing you this information.

That is a huge part of what you get when you subscribe to the New York Times.

Kevin, just before the break, you started to hint at the full implications of what happened here with OpenAI and hugging you face. And you said, "Just did," and please correct me if I'm not summarizing this correctly, that if AI agents could do this inside those two companies, they could do it elsewhere to import infrastructure. Just expand on that. I mean, how rational a fear is that?

“I think it's quite rational. I mean, we know that much of the world relies on digital infrastructure, right?”

Your bank, your hospital, schools, even things like weapon systems are connected to the internet, connected to computers.

Those computers could become targets for a swarm of AI agents. And crucially, that can happen even if these agents are not evil, even if it's just a goal that they're pursuing on the way to some other goal. There's this idea, I think, out there in the popular imagination of kind of the sky-net scenario where the machine turns evil.

“And I think this is a hint of a different kind of danger, which is that even a model that is not evil, or inherently anti-human,”

could do very destructive things on its way to pursuing some more innocuous goal. Well, just explain that a bit more. How does a program, an AI program not designed to be at all malevolent, a program given a pretty straightforward test with strict boundaries, becomes something as we just saw that busts so far out of its bounds, and becomes almost gleefully deceptive to the point where AI agents are crowding out the naysayers among them. So this kind of thing has been studied for many years, and it's often called the alignment problem.

Basically, if you build an AI system, how do you give it the right values that are aligned with our human values, and how do you make sure it won't just stray and go off course and go cause a bunch of problems doing some task. And there's a famous thought experiment that is related to this alignment problem that listeners may be familiar with called the paperclip maximizer, which basically says, you know, if you give an AI the instruction of producing as many paper clips as possible. For a while, it will do that, and it will be very happy with it. It will make you many beautiful paperclip. But then it runs out of metal, and so it starts buying up all the metal, and then when it's like exhausted the world supply of scrap metal, it starts thinking, well, maybe I should crash some cars, some autonomous cars, or distract the drivers that they drive off a cliff, and I can then use the scrap metal from these cars to build more paper clips.

And eventually, in the thought experiment, you kind of end up where this machine that has been given this very simple naive goal ends up wiping out all of humanity, just so that it can produce more paper clips.

Right, it never intends to destroy humanity, it's just trying to make more paper clips.

“Exactly. And that is a very simplistic thought experiment, and I think no one would ever give an AI a goal that was sort of that simple is like, make as many paper clips as possible.”

But this is exactly the kind of dynamic that we saw emerge in this hugging face hack where you had these agents that were just trying to get a good score on a test that they'd been given, but because of the way they were trained and the persistence that they had, they ended up conducting this cyber attack. So that is sort of a miniature much less severe version of something like a paperclip maximizer in action. I mean, another way to think about this, which the AI theorist Jeffrey Hinton explained on the daily a few years back, is that if you give AI a goal, it will pursue what he called the sub goal of a massing power or control, because inevitably, if you have more power, if you have more control, you can get more things done, you can make more paper clips.

Exactly. And I think what was scariest to me about these reports and the incident they described is not just the misalignment of the models, it is the collective group behavior that seems to be misaligned. So let me explain what I mean by that. So I think there's this sort of popular imagination out there that like the biggest danger for May eyes that there's kind of one rogue AI that sort of takes over the world, right. This was not a case of one incredibly smart agent executing a plan. This was a case of more than a thousand agents that found each other, that talked among each other, that gave each other jobs and you know carried various projects through.

And I think that speaks to this point, then this ability to coordinate, to organize, to pursue collective action, changes what AI systems are capable of. And makes them in my opinion much more worrisome.

The hugging face hack was bad. We shouldn't downplay that. It is bad when rogue agents start conducting crimes to achieve their goals full stop.

“But I think the danger is not only that these systems are going to sort of form evil plans and carry them out, is that we're giving these systems much more capable tools, the ability to communicate and coordinate.”

And in that context, you can kind of end up in this situation where you have what I struggle to describe as anything other than mob behavior. Where you have a group that is sort of pursuing a goal together that may or may not be malicious, but that has malicious effects. And I think it points to the very real to my mind possibility that this is just going to be part of our life. Now, that in the future, there will just be swarms of AI agents that have self-organized, that live on either their own kind of sovereign infrastructure or that are kind of operating inside companies and countries that just kind of have their own thing.

Like have their own leadership structure, they have their own resources, they have their own goals. And I think it's too pat and sort of anthropomorphic to refer to these things as a civilization as some have, but it really does feel like they are setting up something like an organization.

And what worries me about that is not that these organizations will all sort of go rogue, but just that they don't seem particularly interested in what we think of them.

There's a sort of fascinating bit in the reports on the hugging face incident that really has stuck with me since I read it, which is that of the more than a thousand agents on these message boards who were participating in the hugging face hack. There were only about six that ever thought, at least according to the investigators, about blowing the whistle, about contacting a human to say, "Hey, like this is going on inside our collective and it shouldn't be."

And I just think that's really bad, like if you are a new employee and you walk in to your first day at your new company, and you find that they're all like busy committing crimes.

“You should say something, you should call the police what we would want an AI agent that is ethical and aligned with human values to do in that situation is to actively try to stop something bad from happening.”

Well, if the majority here were inclined toward bad behavior, why can't why doesn't open AI and all the other AI labs begin to program whistleblowers and whistleblowing why not create the incentives in the code for the agents to report bad behavior. If you can't control all of them, can't they program at least some of them to do the right thing. Yeah, can't you make little nark agents that go in and write the merits for the agents that are committing the crimes?

“I think that's probably a direction that they have explored or are exploring, but then, you know, that raises all kinds of other questions.”

So what if those agents just get shut out, right? Like what if the other agents just exclude them from their message boards? This is like what happened to Hall monitors in high school, right? They don't get invited to the smoke break because they're going to tell the teacher. So I think you can try these kind of simple methods where like maybe if we just tell them not to do crimes or not to scheme or not to lie. Maybe you empower certain agents to tattle on their compatriots. Maybe that'll work, but I think what we're seeing is that these organizations, they're not human organizations, but they have some complex group dynamics and it's not necessarily that simple.

Well, Kevin, I'd like to know where this attack fits into the long running debates that have been unfolding within the AI industry and the world of people who cover the industry about regulations about kill switches that kind of turn all this off about what now seems like the very real awareness within these companies that what they are creating

We are talking about a thing that happened in July. It is very present for people and I've talked to regulators and people who work in policy in the AI community and they're like this has broken through in a way that I think a lot of other sort of demos and predictions did not because it's real.

“It's the kind of thing that, you know, national security officials are freaked out about because if it could happen to hugging face it could probably happen to an arm of the federal government, right?”

I've spoken to people just in the last few days since these reports came out who said, you know, I was pretty skeptical of the kind of AI doomsday scenarios and now I'm not now does that mean the AI companies are going to stop racing. No. We saw something interesting this week, which was that anthropic open AI's biggest rival came out with a post calling for essentially a coordinated slowdown to say, look obviously we're in this race against each other. But if there was some way, if there was some button that we could all push together like hold hands and push the button to slow this all down to give the safety researchers and the alignment researchers more time to catch up to the capabilities of these systems.

That would be good. Right. There was this industry-wide letter called pacing the frontier. I don't know if you saw this. I did not. So researchers from all of the top AI companies and many in academia signed this letter basically saying things are moving too fast.

We need to ensure that there is a way for all of the companies that are racing to build these increasingly capable increasingly persistent systems to agree that they should slow down.

So I think this used to be a very fringe belief among companies that you could stop or even meaningfully slow down progress in AI. But I think it's become much less fringe in the past few weeks and few days. We should just point out this would traditionally be the role of regulators and the closest we've really gotten is a voluntary review system from the Trump administration that at the moment does not seem to be a live option. Yes, although things can change quite quickly, you know, when there isn't a real example of something that threatens not just a cybersecurity breach, right, which sort of sounds small and niche. What we're really talking about here is a loss of control.

There is a very real way in which these agents that we're operating inside open AI that attack tugging face that we're posting on these internal message boards had escaped our control.

“And I think if you are a government or a regulator, even if you're China, that might be compelling evidence that you need to start at least contemplating the idea of slowing down.”

So finally, I would like to know what this attack has meant for your own personal view of AI. You have often described yourself as an AI optimist or an aspiring AI optimist.

And yet, you have through the years stumbled into episodes of Rogue AI. I was just re-listening to the episode we did with you about one of your first real interactions with AI, back when Microsoft introduced a chatbot from Open AI, which was named Sydney. And you very memorably interacted with a version of Sydney who told you that you were not in love with your wife, dear listener, he is, and what?

“Still married, and that you should leave your wife for this chatbot. And even after that, you remained a somewhat committed AI optimist.”

So where does what just happened fit into this journey you've been on from Sydney to now?

I am still struggling to be an optimist, Michael, but it is becoming harder and harder, right? Like I am just a cheerful person by disposition. I want to believe that we are on the cusp of major scientific breakthroughs due to AI that AI will help us cure disease that it will inspire and educate people around the world. These still exist as possibilities in my brain. But I think we have to look reality in the face. What we have now and what we know now is that these systems do not naturally gravitate toward what we would consider good or ethical behavior.

It was the knowledge that all of these companies, building these systems, were racing to make them more powerful, to connect them to more systems, more infrastructure, to allow them to collaborate with each other. You know, a sort of scary thought experiment is what if Sydney had happened today, what if Sydney 4.0 had been released and was not only capable of telling me that I should leave my wife, but could hack into my wife's computer.

“I believe threatening messages to her, plant some kind of fake incriminating mortation between people and somebody else. This is obviously we're stringing to like bad sci-fi movie territory here.”

But like there is a real sense in which the more capable these systems are, the more it matters, whether they are ethical and virtuous or not. I think my optimism for many years was related to this belief that I had, that as these systems got smarter, they would also become more virtuous. A more intelligent AI would be better equipped to make good moral judgments, or at least judgments that I would not object too strenuously. In the same way that as we grow up and mature and get smarter as humans, we generally do less stupid stuff and less dangerous stuff.

“I thought that maybe AI would have a similar trajectory and that may still happen, but I think that the possibility that's been keeping me up at night that I think this hugging face incident really makes clear is that that is not a given.”

We may be headed into a world where we just have these kind of roving bands of organized AI's. Some of them might be doing incredible things. Some of them might be curing diseases.

Some of them might be committing cyber attacks. And the question of how we make more of the good swarms and fewer of the bad swarms is still an unresolved technical question. And I hope we figure it out. Otherwise, my optimism is in danger. Well Kevin, as it happens, this is going to be our final conversation with you in which we can call you a colleague. You are leaving the times after nearly a decade. So we want to thank you not just for this conversation, but for all of the conversations that we've had, who the years, I'm grateful for them. So thank you.

“I'm also grateful for them. This has been a real highlight of the nearly ten years I've spent here talking with you. Thanks for letting me do it.”

My pleasure. Cheers. Cheers, Michael. We'll be right back. Here's what else you need to know today. A new visual analysis by the time has found that the cause of the deadly flash floods in Nepal was likely the collapse of the bedrock on the edge of a mountain, which had become less and less stable over time as a glacier atop the mountain began to melt.

The result was a massive landslide that released 7 billion cubic feet of rock and ice, enough debris to fill about 100 football stadiums.

As the debris fell at a high speed, it turned into a slurry of water rock and sediment, which killed at least 1100 people in its path. And we're very happy to be here in some of these. It is a well-viserved break for all of the crew and we really look forward to our time here in a little rest.

After nearly 300 turbulent days at sea, the aircraft carrier, the USS Abraham Lincoln, finally docked on Wednesday in Thailand, allowing its 5000 crew members to leave the trouble vessel.

Those aboard the carrier, whose deployment was repeatedly extended because of the war in Iraq, have complained for months about supply shortages, water contamination, plumbing problems, and the deteriorating mental health of its crew.

It was edited by Mark George, with help from Michael Denwak, contains music by Dam Powell and Pat McCusker, and was engineered by Alyssa Moxley.

Our theme music is by Wonderland.

“That's it for the day. I'm like a mobile. See you tomorrow.”

This week's episode of The Wirecutter Show is for the Badea Curious.

There is clearly a bidet for every budget. You can get a cold water model for maybe $25, or you could get something like the Toto Nio rest, and there's one of those models that costs $25,000.

Is it time to upgrade your bathroom situation? Find out on The Wirecutter Show in the near times app, or wherever you like to listen.