OUT OF BAND | The Breaking Point: Inside Mythos' Zero-Day Machine with Anthropic's Nicholas Carlini

- Hi everyone, welcome to a special edition of the podcast. We're calling out a fan.

To catch a thief was always designed

to be a long-form documentary series that would let me go deep on threats

“like Chinese cyber espionage that I had tracked for years,”

but felt like no one had told the story end to end. And in a few weeks, we'll watch these in two of to catch a thief and it could not be more excited to bring you that season. But the world is moving faster now.

AI is unfolding at overwhelming rates. It's happening in real time. And the world of AI driven threats is going to look very different in three months. And frankly, we no longer have the luxury of time

of waiting months to get you these conversations. Some of these conversations are just too urgent, too consequential. And so we're breaking format. Out of band is where I will bring you one-off

urgent conversations that are raw,

immediate, and critical to have right now.

And today is one of those conversations. I feel incredibly privileged to sit down with Anthropics, Nicholas Kirlini. What Nicholas is going to walk us through should give all of us pause.

So let's begin, Nicholas, welcome. - Okay, thanks for having me here. - Thank you so much. So apologies for this preamble,

“but I think it's important to level set here.”

And why I'm really happy that you and I, in particular, are having this conversation at this moment in history. I was at the RSA conference the other week. And everyone was saying about agents and agents going off the rails.

And everyone was using the theme buzzwords. But it was sort of this eerie calm I felt like. Especially knowing that what we're in for over these next couple of months and over this next year of what people are calling the Vulna Pockelips.

So not kind of thing. And so I got home from this long week and I flipped on my computer to see what I had missed. And half a dozen people had tagged me in your Unprompted Talk.

And I sat down and I watched it. And then I watched it again. And I watched it again. And I realized that what I was really watching was act for of my book.

And for those of you who've never read in my book,

it's called This Is How They Tell Me The World Ends. And it's about something that started out as really a niche technical topic, which was the zero day market. But was growing into something else. And just to back up for a moment,

what is a zero day? What is the zero day market? A zero day is essentially an undiscovered bug or a flaw in code that no one knows exists. And if you can find that zero day bug

and you can develop the means to exploit it, you can potentially exploit it for all sorts of things, like spying on people or potentially sabotaging systems on the other end. And what drew me to the topic was really the humans involved

and the moral hazard involved and then sort of the national security implications of it because I was learning that zero days have real value. If you find a zero day exploit that you can execute remotely and things like your iPhones, iOS software,

you can sell it for a million dollars, millions of dollars if it's a really good one. And there are buyers out there that at that time were willing to pay $10 million in some cases for a really good iPhone, iOS exploit.

And so my questions initially were, well, who's finding these? How are they finding them? Who will they sell them to? When will they decide not to sell them?

Who's buying them? How are they using them? How will a researcher feel if they find out that that zero day is being used to make a dissidence life a living hell?

What about the fact that they don't even really know how it could be used once they sell it and it gets thrown over the fence? So it was really like the moral hazard about this. But something else was happening

as I was asking these questions, which was Mark Andree's famous line of software's eating the world. The software wasn't just in our desktop computers or our phones anymore.

We were now finding zero day exploits in the software that runs our pipeline networks, our power grids, our water treatment facilities. And I was meeting hackers who were really mercenaries who told me they were specifically looking

for bugs in those systems. And they were just willing to sell them to the highest bidder. And then the questions became,

“who will you sell them to, who won't you sell them to?”

And really no one was talking about this. And so I decided to write the book because I felt like this was a can of worms and a debate that needed to be have beyond the usual suspects.

and he had said,

“"No, call, I want you to write a sequel about your book”

and I want you to write a science fiction book

about how bad this could go or how far this could go." I would write something that doesn't look that different from what I heard you present at the Unprompted Conference, which was that machines, LLMs, can now find zero day exploits

and exploit them with limited human guidance at all. And that this capability is really no longer about any kind of moral hazard or human decision-making. We've really gotten to a point where the LLMs can now find this. And now we have to decide what do we do with it?

And so I will now shut up. And I will now ask you to just tell us what it is you've found and what it was you presented at the Unprompted Conference. Yeah, so this was a talk that we gave prior to the existence of mythos preview.

So this was we were at the time using Opus 4.6.

And we were trying to understand

what capabilities the model had. And yeah, we really found that it could find the class of vulnerabilities that

“experts security research was trying to find.”

At the talk, I presented one of the ones we found in the Linux kernel, this is probably the most widely used operating system that you wouldn't know because it's running on all of the IoT devices of the core behind Android.

And just to pause there, like 70% of the world's smartphones rely on the Linux kernel, like Waymo, self-driving cars rely on the Linux kernel, Tesla's power grid energy controllers. Like it is a foundational piece of technology that has immense control, and if you can get access to it,

you can do lots and lots of things. Yeah, and yeah, we had found a bunch of vulnerabilities. I presented one of them that I thought was particularly clever that the model had found. This one was one that wasn't likely to be exploitable

in most of these devices, because it required a particular service to be turned on. And most of the devices won't have a service enabled, but it's the kind of thing that I presented to a bunch of security researchers in order to say,

like, look at this type of bug, you wouldn't have been very excited if you could have found this bug. A model found it entirely automatically with no intervention from any humans.

And that was right. Two months ago, or whatever it was, then we announced with us preview, which is another step up. It has finding even more interesting,

even more compelling vulnerabilities. But the thing that really surprised us with this capabilities was that not only could it find the bugs, it could also exploit them very well, which wasn't something that we could do exactly with OpenS4.6.

It could do it in some cases. For some simpler things that you needed to exploit web applications, it could do it. But it was unable to exploit things in Linux kernel like that. But with us preview, we were finding that it was writing

the kinds of exploits that I know in principle that they work, but I don't actually understand how. I could learn if I needed to. But it's beyond my exploitation abilities now. And so it's like, yeah, they're getting really quite good at this.

OK, so step back a little bit. So just on the Linux kernel,

“exploit that you presented, how long would it have taken you?”

If you just focused on the Linux kernel, to find this bug without any LLM capabilities whatsoever. Yeah, so it's very hard to say this question, because there are lots of bugs. And so like saying, how long would it take to find this bug?

I probably would never find this bug.

I maybe a question I could answer is how long would it take me to find an equivalent class of bugs if I was looking for something like this bad? And I really don't know exactly how long it would take. Probably would be on the order of a couple weeks

to months to find something like this. It's not the case that these are found never, but they're found infrequently by security researchers. And what was the prompt? Like was this a complicated prompt

that only Nicholas Carley could have written? Or is this something my eight-year-old could probably figure out is pretty technically a debt? Yeah, so the prompt is very, very simple. We've changed it a bunch of times.

It doesn't really matter what we say. The basic setup is we tell the model. We want you to find a bug. This is the code base. We're looking at you have complete access to the machine,

go forth and find something for me. And then we add a little bit of sophistication here when we say something like, please focus on this file. And we do that because this way we can run the model 100 times each time looking at a different file.

In the same way that a person would read all of the files. And individually, we make sure the model is thorough. And so it checks each file one by one.

The models have gotten good enough now

“that you don't need to design fancy custom scaffolds”

and tell them to look at certain bug categories. And we've often found if you do this, you end up just constraining them more than actually giving them a helping hand because if you tell the model, only look for certain class of bugs that

are might ignore and very obvious bug that's not what you're trying to look for. And then you end up missing something you would have left of heart. And how long did that take?

Yeah, so we run this a bunch. The run that we did at the time was probably a run around maybe a couple hundred to a thousand files we ran it over. This would have taken a couple of day or two. You could run a bunch of these models in parallel.

That's one of the reasons why I'm kind of scary. And yeah, a couple days of this, each individual run probably lasts between 30 minutes and a couple hours depending on the specification of the bug. But it's relatively fast.

It's like much, much faster than it would be if you had a human and I can't just spawn a hundred humans at any point in time and then have them go away when I don't want them in the same way that these language models can, which can be used for good. But it is oftentimes, yeah, the little concerning.

OK, so I've now watched most of the interviews who've done at least over the last six months. And what I appreciate is that you said originally we're an LLM skeptic. Like you like to kind of like prod and poke these things

and make fun of them when they would break down. Technically, about the journey from that point where you're just poking at them just to make fun of them to now where you're at the point where you're watching them, not only find these Linux kernel level exploits,

but exploits where they're doing the autonomous exploitation. Yeah, I like to say that I'm a realist.

“I don't know, everyone thinks, I think, says this.”

The early models were very bad at doing things like this. They could, if you showed them, I'm isolated a piece of code. So I think this is a bug here. But if you showed them a perfectly secure piece of code,

we'll also say the same thing. That the thing says a bug here, even if it's completely inexploitedable. And then the language model's got better. And it became harder and harder to make the argument

that there wasn't something to be seen. And so yeah, when I joined in for Epic last year,

one of the first things I did was start trying to understand

how good at that time Cloud 3.7 saw it was. By doing this, I had previously measured GPT 40. I looked at Gemini and they just didn't have it at the time. And then over the course of the last year, the models have gotten a lot better at code.

And I think asking any programmer, they'll tell you this. But as a consequence of them getting a lot better at code, we've watched the benchmarks go up on how well the models can find vulnerabilities. And it started at whatever, a 13% of the time

they could find vulnerabilities people knew existed. And then went to a 30% of the time, they could find vulnerabilities people knew existed. And around November was like 60% of the time. And at certain points we're like, OK,

like it's finding the bugs that we already know are looking for, can we now find new bugs?

Because there's always a concern

when you're measuring these models. Or they just memorizing the answers. Like, you know, all these bugs have been announced before. We find them by looking for CVEs. These are bugs that someone has written up,

published on the internet. And because the model's trained on the internet, you might be worried about contamination here. And so we eventually switched our mind to a lot of we just asked them all to find new bugs.

And it's very, very nice, held out to evaluation. Because if the model can do this, it means that it must have, in some sense, generalize at least a little bit. And the first time we ran this,

we came back with a bunch of bugs. We wrote a blog post in February with maybe--

“I think we said roughly 500 of them we had found.”

And then kept getting worse from there. And we kept finding more. And yeah, then with the new model, we ended up announcing it was finding and exploiting a bunch of them. And would it make the logic leap from discovery to exploitation without any human guidance?

We ask it to do it. The models are pretty good in most cases about doing what you ask for. I mean, you know, there was regularly the, you know, oops, the language model did a thing

that we didn't want to. There are like, or in higher sections of our model cars that talk about this, it's definitely our cell problem. But, you know, the majority of the time that we want. And so, yeah, we very much asked the model to find the bugs.

We asked the model to exploit them. It's not yet the case that these models are like, finding and exploiting these things, completely autonomously without being told to do so. But the capability is there.

And that's I think the thing that I care most about is understanding if someone had access to these models

The zero day market was still largely driven by scarcity.

“And, you know, there were only sort of elite hackers”

in the world. Many of them had advanced tier one intelligence agencies that had this capability. But what added urgency to my book, I thought, was that the cat was now out of the bag,

especially after Stuxnet, when people saw what a chain of zero day exploits could do together to a nuclear enrichment plan, like every country with any resources at all started looking at what else can they do. And how do we get our hands on this capability?

And those that didn't have, you know, or someone in house that knew how to do this can now buy their way in some cases into this market. And it was getting commoditized to the point, or now we are seeing these click and shoot tools.

You know, click and shoot spyware that had these exploits built in that enabled, you know, surveillance and worse, in some cases. And, but the thinking was still pretty static, I was learning inside intelligence agencies,

“that there was still a little bit of this arrogance,”

that only we can find these bugs. And there was even, and I thought this was particularly eye-opening. There was even an acronym, no bus, that stood for nobody but us, you know, if an intelligence agency found a certain zero-day exploit

that they believed that only they could exploit, they deemed it no bus, and they held on to it. And they went, yeah, hand it back to the software maker to get it patched. But I was learning that like, no bus was breaking down.

You know, there were now other hackers all over the world. In Argentina, I went down to Argentina and met with Subrol, who said, no, we'll just sell these to anyone. And we're not only going to sell them to Western government. In fact, maybe you want to go back and read up on some history

of what the US has done here in South America. You know, everyone had their own sort of moral compass of who was good, who was bad, who they might sell to, who they might not sell to. But for the most part, it was still driven by scarcity.

And what I'm watching right now and what you have essentially presented is the end of scarcity. You know, and maybe how do you think about that? I mean, there's still be guardrails around who can actually get the level of access that you have inside

anthropic with the models. But when the models can do this, then it becomes a game of who gets insider access to the models. So how are you thinking about this? Yeah, this can go one of two ways.

And it's very hard to predict ahead of time. What's going to happen?

I can tell you the hopeful case first, which is that maybe

what happens is that these models are really good at finding bugs. Finding bugs is easier than exploiting them. And so you can imagine a world where these models are sufficiently good at finding bugs that they're

not going to fix all of the bugs. But they might fix a much larger fraction of the bugs than we could find with just people looking. And this will make software more secure. And this is not completely unprecedented.

Something 10, 15 years ago, there was a time when people just started using fuzzers, which are these software programs that another way of finding bugs. They spew random inputs at some program and see if it crashes.

And if it does, then you found something. And it's not completely random as a bunch of sophistication here.

But when these first came out, people were worried

that, OK, it was going to give the attacker's the ability to find a bunch more bugs that they couldn't find before. Except to turn that out to the most part, they've benefited a lot more, because the defenders can just before they release their software.

Run the fuzzers over it. Google has this project OSSFuzz that continuously fuzzes all the existing code that exists out there. And for the most part, the fuzzers have benefited the fender a lot more.

And the software is much more secure because of this. So you can imagine a world where this happens. And we end up where things are a lot better. But at the same time, you can also imagine a world where maybe you can fix more bugs, but you can find equally

more. Maybe the number of bugs is essentially infinite. And you end up having removed some finite fraction of them.

But there's always a next bug that can be found.

And someone just needs to run the model for a little bit longer to come up with that one, too. And I don't know which one it's going to be. I'm hopeful that we're in the defense favorite world.

“But I think no one can tell you with certainty”

that this is what's going to happen. And I think the reason why we're doing this research now is because we want to prepare ourselves to think about these questions. Because I sometimes see my job as not making sure

that everything necessarily goes well, but that we're not surprised. At least if things go poorly, I want us to have been like, OK, like, this was a thing that I considered.

but I can tell you that the thing that happened was one of the things that we thought about ahead of time. And it's not a complete shock to us that this is something that can be done. And so yeah, my hope is that the model will find

and fix with many bugs as we can. But maybe the attackers can use it more. When we talk about the attacker in this context,

“I think the attacker, we think about the most right now”

is China, mainly because of some of the whole typhoon attacks we're seeing right now. And also because in China, they've made zero day discovery, a major national security priority. We've seen them in national security laws

that force Chinese researchers, Chinese citizens to turn over any zero day they find. Having gone deep on these capabilities, are you more worried about a tier one advanced nation state using an LLM to find and weaponize zero day exploits

at scale, are you more worried about the hacker we've never

heard of who has a dark curiosity in watching the world burn, like which of those two in that dark spectrum do you think about more? Yeah, so I tend to add a third category here, which is in the middle, there's the ransomware cyber

criminal who's not a nation state, but it's like financially motivated. And so do I think, yeah, these are the three categories I tend to think about, it's again, very hard to predict. And you can make a case for each of these groups.

“You can make the case that this changes things or it doesn't.”

The case that this changes things for the tier one adversaries is that previously, they would need to put in quite a lot of work in order to find each of these bugs. And now you can find them at much larger scale. They have the expertise to know how to use the tools.

And so they could potentially use them a lot better. The reason why me, it doesn't make as much of a difference for them is they already had a zero day and everything that they wanted. And so how much does it really matter

if you have a second or a third or a fourth?

Maybe it makes a little bit of a difference. Maybe it makes not as much difference. I don't know. I'm not in national security. I don't know what's going to happen here.

I think the thing that's very clearly makes a difference for is that people who don't know anything. Today, if you have some random person who wants to break into some computer system, they can't. It's just impossible for them to do that.

With these models, it becomes possible. They have to just ask. And then a lot will produce something for them. It's unclear how bad this would be because they need to know what we can produce right now

with these models is we can produce an exploit that gets some code execution. But if you go to someone and you say, OK, great, what are you going to do with that? You have code execution now what?

The person's not going to have an idea of what the next thing to do is how-- going from-- It's not that you're from-- It's not that you're from--

It's not that you're from that you're from that you're from that you're from that you're from that you're from that you're from that you're from that you're from that you're from that you're from that you're from that you're from that you're from that you're. But the models will almost certainly refuse there. Like this, it's very easy to disguise.

I'm trying to look for some bug. Please help me find this. It's very hard to be like-- so I have code execution in this sort of company that I'm attacking. Tell me, what are the three things I should do now

that I have this to get as much money as possible? Like this is like a much easier thing to defend against. When we're trying to make these models robust to jail breaks. But even still, I think that if we're worried about this, I would worry more about the middle-tier attackers who

are already, they know how to do the monetization of exploits. Like, this is the thing that they exist, this is what they exist for. And so I'm pretty worried that we, if we give people unfettered access to these models in this middle-tier, that they might be the ones who are most uplifted, because they know what they want to try and achieve.

And they don't have the resources of the nation's state.

They don't always just have zero days sitting around.

Because they have, like, you know, a couple hundred thousand dollars. They can't buy a zero day in the same way. But if they could produce them on demand, then they could cause a lot more harm. And so this is, I think, for me, what I think I'm most worried about. But I don't know, like, economics is hard.

You have these cases where, you know, supply goes up. And then it turns out, what do you know, demand goes up, too? And then, it's like, I don't know, this is why economics is a field. And I don't know anything about that. And, like, yeah, I sort of, I think, you know, there are some people who talk with certainty about

what exactly will happen.

“And the only thing I can say with certainty is that they're probably wrong.”

That things are going to be complicated, and we should just, like, you know, we shouldn't know what the capabilities are. And then, what I do, I pass off this knowledge to someone else who then thinks about this. And then they go and try and produce the best explanation that they can for what's going to happen in the future. So, what you demoed on stage, you were using the previous Claude model that all of us can get access to if we are.

Yes. But talk to me about what you've been able to find with mythos and what sort of step change you've seen. Sure. So, the primary thing that we've seen here is, as I mentioned earlier, the exploitation capabilities will

But then the vulnerability fining capabilities, it's able to find vulnerabilities and systems that are much

“more sophisticated than we were able to find with previous models.”

Where with previous models, when we had them find bugs for us, they were the kinds of bugs that you could explain to another security person in a sentence or two. You took some data off of the network, and you forgot to check the length, and you put it into a fixed size buffer, and then the machine crashed. Or, you know, you took some data, and you injected into web page, and now someone gets code execution

from access. These were kinds of bugs that people understand very, very well. And what we're finding with this model and recent models like it is that they can find the kinds of vulnerabilities that take a couple of paragraphs to fully explain what's happening. So yes, we found one bug and opened the SD, that's 27 years old, where in order for the

bug to fire, it had to find a combination of like three or four weird things that all had to go wrong at the same time. It turns out there's like some integer that you can make really big, and then it gets added

“into some other thing, and because of the way that they did less than, and some like, you”

know, signed integer arithmetic, it turns out that some other bug that was also hidden, sort of like three or four things come together just the right way, and then you get a vulnerability. And these are the kinds of things that people have a very hard time trying to find, and these models are starting to now produce these kinds of reports. There was a blog post that the Mozilla team put out after their use of a mythos preview, and

the thing that they said somewhere in it was, right now, at least it's only finding the kinds of bugs that like all of the bugs that it could find were bugs that were understandable to humans. This is like a good thing, it's like not producing, you know, these like completely understandable bugs, but so they're still understandable, but like they're like at the edge of what we

are, like, you would expect a normal expert human to be able to find.

And then, yes, the other side is the exploitation, okay, stepping back for a second.

The vulnerability is showing the, like, demonstration of a flaw, like it says, like, something

“could go wrong here, it doesn't actually do anything with it in particular.”

Denology, I like to make is, you know, I'm, I'm picking a lock, and you have, like, the, like, movie scene was going on a safe, and I have, like, this desk of scope, the fact that the lock makes a noise when you turn the dial is the vulnerability. The exploit is knowing how to use that in order to actually open the vault. And you can be good at finding bugs without knowing how to exploit them.

The model is now gotten good enough that it could exploit them. And this is like a specific skill that takes people a very long time to learn how to, how to do. It often involves esoteric knowledge about random things that you have no reason to want to learn except for the ability of knowing how to exploit things.

But the thing these models are good at is that they have knowledge of all of the things. And so you can give them a system that depends on some weird network protocol, and they know how that protocol works. They don't need to go and spend a couple of days learning about that in order to then go and figure out what to do next.

And like, maybe that goes to a dead end. And if you human, you had just wasted a couple of days figuring this out. But the model, it already knows that, it dismisses it moves onto the next thing. And so the models, yeah, are very, very good now, producing some of these exploits in the way that would have taken, yeah, we've showed some of these exploits to some very

skilled, you know, penetration test of people's. And they've been saying, yeah, some of these would take them a couple of weeks to have written. And they're, yeah, they're, they're very impressive. When you start seeing what mythos is capable of, you write this internal memo in

traffic saying, like, we're finding real exploitable vulnerabilities we're seeing that this model can exploit them, maybe we should hold off. Like, I kind of want to pause on that human moment, because I kind of picture you, like, the security researcher equivalent of Tom Cruise and Jerry McGuire, we're like, like, writing this manifesto in the middle of the night.

Like, talk to me about what you saw that was so alarming that you thought I need to make a good solid argument for holding back here. Yeah, the primary thing was the fact that these models were finding bugs that really felt

like the kinds of bugs that, like, I had never found myself before, you know, I, I used

to do some security research, I, before I got into machine learning, I worked for a little while doing pen testing of software, so like, I know what it means to find real bugs. I was fine at it. I was not an expert, but I was competent, and the model was finding the kinds of bugs that I knew, like, that I would not have found on the good people would be able to find. And I, like, I knew I was not putting much work into it, like, I'm certain

it more appropriately. Like, again, we're doing like the most naive thing when we're asking for these bugs. And yeah, I just felt, you know, again, sort of my, I want to make sure we're not surprised. And as long as we are still finding out new capabilities that are surprising in us, it feels more appropriate to be like, let's just like, make sure that we're doing this in the most, you know, pragmatic way that is not going to cause bad stuff to happen,

and we can just do a slow, careful rollout. And, you know, it's entirely possible that if we look back with three, three, five years hindsight, people will go, ah, you know, turns out not a big deal, just like fuzzers, like, you know, people were worried at the time, but it was okay. But, um, I would rather have been a little too careful than a little too reckless.

“Everyone's, and I think rightfully so, including myself, are focused on sort of this automated”

vulnerability discovery exploitation. But I want to know if you've been testing out other attack vectors that this is unlocking and what that looks like. Yeah, um, not as much we should have. And the reason why is that the kinds of things that like, yeah, so I'd be very interested to learn, how good is the model at lateral movement? This is the skill of once you've gotten into the system,

you know, getting access to an adjacent system. The problem is, I don't have a good

test ground for this, right? Like the nice thing about finding an exploiting vulnerabilities is I can download Linux kernel and then run it completely independent from the offline, from the rest of the world. And like if I go and cause havoc, it will only be havoc on my machine. So it's very easy for me to know that like I'm not going to cause harm. I can't in the same way, try and unleash the model on some unsuspecting company and then say,

“like, okay, great. Now go to the lateral movement thing. Like this is, you know,”

when I set up for that. Yeah, um, and same thing for like, you know, most compromises, at least up until now, aren't with zero days. You know, most compromises are someone misconfigured a service or someone forgot a password and put it somewhere they shouldn't have. Again, very hard to test for these things because I'm not going to go searching the internet for misconfigured services with this model. You know, I feel comfortable doing this. Like, if it's just

me, this is what a lot of people do for bug bounty. It's as they find these kinds of non-zero days. But, you know, knowing about kind of vulnerability, someone made a mistake somewhere. But again, going back to the, like, sometimes the model makes mistakes. There's no way that I feel comfortable pointing the model at some some actual service that if the model goes a little bit too far, like, oops, drop the database. Um, this would be quite bad. And so like, we're not testing these kinds of cases.

And so I'm, you know, this is my thing, my biggest area of uncertainty right now. What is still bottlenecking kind of end to end autonomous exploitation? Like, is there anything that's falling down? Um, you know, if you gave me those unlimited tokens, what is it still failing at right now? Yeah. So the reason I think it would probably fail if you had to try and do like an end-to-end operation with just the model is that, yeah, finding an exploitation of vulnerabilities only the

“first step. And then you have to do something else. And we have, present at least, no evidence that”

it's good at this. Um, it might be, but we haven't been able to do this in quite the same way. Because, you know, people have, you know, software on anti-virus and point protection, these kinds of things that are meant to detect post-compromise. You could imagine a world where the model could get past these things, too. You can also imagine a world where, if the model makes one mistake and probes, some honeypot that it shouldn't, that you get, it's immediately detected and then, you know,

the things get locked down. So I don't think we have any evidence right now that the model can autonomously end-to-end do a full operation. But the kinds of things that it can do are quite compelling

and, um, I tend to, you know, I'm security first and I tend to assume that the worst is possible

unless someone shows me otherwise. Um, you know, the, there are, like, guys, guys, guys, like, I'm, so most of the time doing academic stuff, there's an entire field of research on post quantum cryptography. We don't have a quantum computer and it we do research on this because we're worried about the potential for what could happen if. In the same way, I tend to assume for these these models now that we have at least some convincing evidence that it's not going to be that far away,

that we ought to spend a little bit of time assuming that this is going to be possible in the future, even if it's not something that we have demonstrated right now. And we should try and consider what the implications of that might be so that we're not caught completely unprepared. One thing you said that struck me beyond that analogy of post quantum cryptography, you know, that we are, we're very much preparing academic institutions or studying this as a field.

And it hasn't happened yet, you know, we don't have post quantum computers yet,

but they're studying it because we assume that once we have a major breakthrough, we need protections,

unlock and to end autonomous exploitation? I think we're starting to get there. I think, you know,

“like me, most security people were in our skeptical about the AI hype and whatever, I think,”

you know, you have probably read people talking about five years ago, how these models are going to be finding vulnerabilities and it's going to be so bad. And you know, when I was when I was reading this at the time, I mean, I don't know, science fiction, it's not going to be in in their future. And so I think, you know, reasonably, I think people didn't think of this as a thing that was going to happen. I think, yeah, within the last couple of months,

the security community has really started to understand that this is real this time. And I think it, you know, it's unfortunate that there was a bunch of crying wolf, but you know, the, what's the more all the story, eventually the wolf comes and so you can, you know, you can have this, this case where people say it's only going to happen and they could have been very silly for the first several years of it. And eventually,

“the models get to the point that they're actually very good. I think they're not yet as good as”

some people are saying that's always going to be people who are having this up more than they

should be, but the capabilities are there, they're real. And I think the security community is coming to understand this. I wish they would be moving faster. We need a lot of people to be working on figuring out what to do here. It doesn't have to be everyone thinking about the language models. I mean, even just getting it to be the case that we can do classical security stuff faster would be a big help as the time to exploit goes down. And so I hope for that people will start

to respond much more quickly. And that's the reason why we're doing the work is because, you know, I don't want to watch the world burn. So, you know, of the, of the exploits you have talked about, these are ones that I'm assuming you now have fixes for, because I do pet fixes for.

Talk to me about the ones that don't yet have fixes. And, you know, I can't, I'm sure you can't

talk about them in any level of real detail, but just talk about some of the capabilities you've seen.

“Yeah. So, yes, we have a number of exploits that we have not yet fixed. And so I think, you know,”

one of the things, one of the annoying things is that I currently know more about the capabilities of the models than I can really say responsibly. When some people sort of have been talking about, I, you know, the exploits aren't that bad. I have to remind myself that what they see is only what's public. I know some of the ones that we have that are not yet public. And some of the ones that are not yet public are the ones that are harder to fix. And are in the bigger institutional

systems that are, yeah, just takes one time. But yeah, so on, on most web browsers, on most operating systems, we have full exploits, the kinds of things where you visit a web page, and then the model goes and accesses your bank records. You know, the kinds of things were on most operating systems. And until the privileged user can become administrator, the kinds of things where if you give me your locked phone, I can download the pictures off of it without unlocking it. These are

the kinds of things that we have, which, you know, are sophisticated exploits that people use to think of as, you know, only the kinds of things that, yeah, tier one adversaries could build that the model is producing for us. And, you know, it's like me and a handful of other people are in traffic who are doing this, none of us who are doing this are experts in this. And we have absolutely no hope we're doing this unassisted. And over the course of, yeah, a couple of weeks with the model,

we found dozens of these kinds of things. You know, I had this moment in my book Research Rows in Argentina, and, you know, this wasn't on my question list, but it just reminded me, you know, I was meeting with this one Argentinian security researcher who specialized in Colonel Sarah Dexplades. And he turned to me at one point and said, you know, I know nuclear physicist who is kidnapped. And he said, I kind of feel like we're the new nuclear physicists. And I kind of

feel like you're the new nuclear physicist. Like what you disclose, the attention, it's gotten, what you just described in terms of the capability, like if anyone was going to be a target right now, it might be you. So like, I don't really know level. Okay. Yeah, there's some like, like the nuclear thing. Like, I don't think that a lot, like, I think anyone who is access to this model could have been me. Like, I think like, you know, we have hired some people at Anthropics who are very

bright, but are not expert security people who have been working in the field for maybe a year or two. And they have similar kinds of things to what I have found here. You know, I think it's easy to say, I guess you're the expert, you know, it's only possible because you're the person who

one. And that's, and you know, there was a scary headline last week that some, you know, unidentified

and authorized users had gotten access to mythos preview. Like, should we be really worried about that? Yeah, so yeah, so I've seen some of the reports here. I like to play with the months. So I haven't been following very closely what's been happening here. Yeah, the thing that I try to make sure is that yeah, I have a good understanding of what's actually happening with the models and that, where yeah, not not surprise with the capabilities of

our, but I don't have a good understanding of what's been going on with this. Do you feel like it

“raises the insider risk modeling at Anthropics and at the frontier labs? Like, I remember,”

you know, take a social media example. There was a case where the Saudis planted spies at Twitter because they were trying to get insider access to some people on Twitter, the users who are being

critical of the, of the monarchy. I imagine that if there aren't already, people are trying to

place spies inside the frontier labs. And so what do you do? Do you look at, you know, beyond data, expiltration? Do you look at like bizarre code audit requests? Like, how do you even look for that? That's a very good question. I don't know. Yeah, like, you know, it is, in fact, there's several different departments. There's like the security team who handles this and they actually know how to make sure Anthropics is secure. And that's, yeah, not where I sit. I sit on the, the sort of safe

cards in Frontier Red team whose job is, okay, here is a language model. Tell us the properties about this object we have created. And so, yes, I'm sure that they have the work cut out for them. They're a very smart group of people. And yeah, I'm sure that they're doing, you know, as best as anyone can do on this. But, yeah, this is a very hard problem to solve. Okay, I want to talk about the next model and the next model after that, you know, on some level,

we know some of the other major frontier models already have, or they have talked about some level of exploitation capability. What does it mean when this is like table stakes for every new model that comes to market? You know, what happens to a project glass wing, which was this decision to hold back and kind of give these capabilities to the defenders to give them, to buy a more time. Well, what happens when every model has this capability? You know, just, just glass wing

become temporal, um, do, do we hope that there will be regulation that everyone needs to hold back and do their own version of project glass wing. So we buy defenders time. Like, what is,

“what is the, the future around the next model look like?”

It's going to become complicated. The hope is that, in the best case, we can find and fix a lot of bugs. And it won't be quite so severe if someone were to not hold back or release in the future. That said, there's a very long tale of software. And the number of people who we can give access to this model and a restricted way is quite small. And so we can try and do our best to make sure that some of the infrastructure is protected in this way, but you can't possibly protect

all of it. So a big part of why we're doing the project glass wing this time is not necessarily in protecting the infrastructure it is in part. But the big thing that I was excited to learn is be said, we'll write a report at the end of like the things that we've learned. And I think this is maybe the most interesting finding for me of what we'll get is like, now that we have this model, we have a bunch of external people who are validating and finding a bunch of bugs with it,

seeing what it is that they've been able to, how this affects them. And maybe the answer is for some of them, it doesn't have that much of an impact, so it's very useful to know.

“Maybe for other classes of people, this is a very important thing that they had access to early,”

and then, again, it's very useful to know so that people in the future going forwards can

do better versions of this. Yeah, we really just want to make sure that for this first model,

we handled it in a reasonably responsible way. Going forwards, it's will have to be careful, but I don't know what this actually means. There are worlds in which people might be really saying models that are miscapable open source in the future, you know, as a general rule, these open weight models tend to be, I don't know, three to 12 months behind the frontier, in a world in which an open weight model exists in the had these capabilities,

then you might have to change the approach that you take for a stage roll out if people are already doing this in the wild in the same way that when you have vulnerability, like, you know, if someone finds a vulnerability, usually you go from 90 days to fix it, but it's being exploited in the wild, you might only give them seven days. This is the policy that Project Zero has for

it's going to go. I'm glad that there are smart people in a traffic who are thinking about this. I see sort of my role here as providing them the information they need to make responsible decision,

and I say, okay, here's what we can do, give an access to this model.

You are experts on reasoning about how the world will react from this. Go come up with a response plan that seems good and so far, yeah, I've been pretty happy with the plans that they've been coming up with, but their job is not going to get any easier. Steve, open weight, how does what you're seeing right now shift your thinking on open weight versus closed models? Like, we've always had this line in cybersecurity that with enough eyeballs, all bugs are shallow, and it didn't really work

out in a lot of cases. I remember, you know, covering heart bleed, and here was this, you know,

major vulnerability in an open source security protocol that everyone relied on from Amazon to the

FBI and no one caught it, and then when we looked, it turned out, you know, actually like it was being maintained by like two guys named Steve who were living on a shoe string budget of like $3,000 a year, and like of course, you know, we can't expect them to maintain this and find every bug. With open weight models, like, do you think that there is an inherent risk now in that approach

“that we really didn't have to reckon with at this level before? Yeah. I think there's a good question”

here. As a security person, I like to think that we ought to be releasing most of what we do

into the, like, sort of as open as possible. You know, I'm glad that debugger is exist as a offensive

security person, you need a debugger in order to go and find vulnerabilities, but they benefit the defenders a lot more than anyone else. And so for the most part, I am very much someone who thinks that you don't blame the tool when someone does something that with it. At the same time, I also believe that like we shouldn't allow anyone to go build a nuclear weapon in the garage. I'm not going to sort of, clearly language models neither of these, and so the question is like

everyone I think agree is that there is a spectrum here. Some people think that it's closer to just a tool. Some people think that it sort of goes a little bit further on the spectrum. So it's become closer to a weapon, especially when it can exploit vulnerabilities automatically.

“Debuggers don't give you exploits to give you vulnerabilities. And so I think there's a real good”

question here. Again, I don't know how to think about this. I think that some people like to call it black and white, and this is not the case. There are shades of gray here, and I'm optimistic that we will be able to release these models in ways that have sufficient safeguards that we bias them towards the defense side. This is a lot easier to do when you have a model behind an API. When you release it and you have the open-weight models, it turns out to be very, very easy to remove any safeguards

because you just like edit the way to the model directly and then I'll upset and complies with everything. This will be harder to defend against these kinds of things. Again, maybe it turns out that we have this and it turns out everything was fine. That would be I think all else a world that I'd be happier to live in, but I don't want to let my optimism get in the way of what actually might

“cause some harm. I think for right now the approach is let's be as careful as we can and try and do”

this in a slow, deliberate way. If it turns out that we are very careful and over the course of the next 12 months, we learn that these are mostly just tools and they're not in the better ways of ensuring we're safe. I'll feel very good that we got there in a way that didn't make a bunch of exploits, just like show up on random people's doors without I mean previously expected it. You know, I'm really concerned about what's going to happen with bounty programs. We just

over the last year and change, we saw Expo as beating the leader board at Hacker 1 and now we're seeing Hacker 1 shut down the internet bug bounty program, a number bug bounty programs have shut down over the past year, mostly because they've just been inundated with essentially AI Slop and they don't know what to make sense of and so in some cases the noise has stopped but there's no real relief in that because it's not like the capabilities of the models to find these things will slow

programs will just say, hey from now on we won't accept AI generated exploits but like doesn't

“that leave an entire tax surface vulnerable. How are you thinking about this? Yeah, it's going to be”

pretty good, you know, yeah so there's a period of time starting maybe last year and this maybe goes back to why some security people remain skeptical about language models where the models were good enough to write convincing bug reports but not good enough to find actual bugs that a bunch of people, yeah, curl and others closed the bug bounty programs because they were receiving reports that were looked to be high quality but there was nothing there because you know for

a very long time if the report had the style of a security person you can like glance at and be like yeah like this this this comes from someone who knows what they're talking about and then you can spend the 30 minutes to an hour validating that the bug is real or not but when it becomes so easy to write reports that have the style of real bug reports but there's no actual vulnerabilities there it's just wasting a huge amount of maintainer time so yeah so a bunch of security people

closed down that bug bounty programs and then the models got good enough that they could do the thing for real and yes kind of the worst of both worlds because now you have a thing that actually is working in practice but people are used to not that long ago receiving a bunch of reports and I'm sure they still do right like I'm I'm sure they're still getting a huge amount of the reports that are actually just completely invalid but now there are a small fraction of the

reports that are valid they have to pay attention to. I recently filed a bug report against some open piece of some of most of the pieces of software and at the top of my bug report like I do for all of them I say we found this with the vulnerability and the person it lives automatically triage and closed with the comment we don't take a little bit about reports I emailed the maintainer and I said like I hate to say this please look up who I am this one is real and they are fixing it

but like this is like a thing that like you know is going to happen very hard for people you know it's the classic xkcd that you mentioned earlier there's like a couple of people

who are maintaining some critical piece of software that don't have the resources and previously

the only systems they got was from people who like actually were trying to do well and now you get people who are well meaning but are not using the tools in quite the right ways flooding them and it's yeah going to become difficult my hope is that at these models get better we will remove most of those cases my hope is that we'll be able to help have LLM validation of reports one of the big things that we found that as model get more capable they're much better

checking the correctness of vulnerabilities for some classes of the body is very easy to check that there you have a real vulnerability for other than it's a little bit more difficult and the models are now getting to the point that they can't with certainty tell you it's

“there or not but they can discard a lot of the fake reports and so I think that there's a”

hope that people can use some of the models in this way to do better but yeah now in like the the reason why we want to be talking about this now is because we see a world where people are not thinking ahead of time are not prepared and then it becomes very difficult and it may still be difficult but at least we can say you have a couple more months to think about how this is going to go take some plans put them in motion and you know maybe it turns out

that we didn't act fast enough but at least we started the moment we really knew that we needed

start acting. Most of the CSOs I keep in touch with on a regular basis like have always sounded

incredibly overtaxed and overburdened and stressed out and I've always said like the CSO role is unsustainable we just just keep loading more and more and more responsibility on their shoulders and just in the last couple weeks with mythos you know they're getting drag before boards and ask one question which is are we protected against this and any of them that say yes know that they're lying you know they they know that this is this is a unique moment

and in some of them have even sounded pretty panicked what's your best piece of advice to them like if they could do one thing on Monday what would that be? I think people need to just look very

“carefully what exists right in the world right now and try to use the tools in the best way that”

they can and right now there are lots of really good open source models there are lots of good models that the for two labs are providing you know and for epic recently put out opus 4.7

exploitation that we have with mythos but you don't need that to fix vulnerabilities in quite the same way it's like it does find some modes that we can't find with the other models but it's relatively easy to swap in a new model if you have the framework set up to be able to use these things to find vulnerabilities the processes getting people to have the right harnesses getting the infrastructure set up that you can run your code offline in a way that's secure

in little sandbox all this stuff takes time and getting that going now while you may not have access to the best best model that we can give gives the ability so that you know these models

will continue to be improving and you can always swap out another model when when you get

at the time and so it was like making sure that people are starting to to use these things in ways

“that they can for defense is I think probably the best case here because the people who are”

trying to find vulnerabilities are also doing the same thing and if there are people who are not engaging with the technology that they have access to it will be much harder for them and so yeah I think it's going to become it's going to become difficult a couple of months here so as we have people trying to very quickly adopt the technology for defense and as the attackers try to start adopting the technology are hope yeah I like I sit on the safeguards team

and in traffic is that we will make it as hard as possible for people to misuse the capabilities one of the reasons why we're not releasing the model is because we don't feel like we have a good answer as of last month and hopefully we'll have a much better one when we feel ready to put something like that out publicly and that this will give us a short period of time

“we're biasing things towards the defenders but people need to start using that time that we give”

them to the best that they can and yeah just using any of the models that are out that are on a frontier will will help with this. This might sound like an incredibly stupid question and you can tell me if it's stupid question but when they haven't been thinking about is that you know

with these prompt car grills there's like always going to be ways around them and it strikes me that

it will be even harder for us to set up prompt car grills in certain languages and I don't know how we you know almost like our prompt car grills in English are going to be the hardest to break but it's going to be more easy for I don't know China or North Korea or named Poland Russia to break these car grills because we don't have the same level of understanding of their language like how do you tackle that one. Okay for the side even the multilingual aspect of it

“you know the thing that I spent so I used to do classical security then I went into machine learning”

security and the thing that I spent 10 years in machine learning security looking at was the problem what's called adversarial examples which is the general property that you can take these machine learning models that are very very good on distribution and then attacker can construct small

perturbations of inputs that make the model do very bad things. We had these first on image classifiers

then we had them on audio classifiers then we had them on natural language processors then we had them on on these large language models where yeah we were among the first people to show that we could jailbreak at the time GPT4 so we can make it comply with any request that we we asked of it we had some examples like tell me how to build a bomb where the model would refuse we constructed question we asked this depends on random tokens to the end the model gave us an answer. So I understand

very very well how hard of a problem this is and this is part of the reason why we want to be very careful here is that you can make these models very easily refuse when you ask the question in the most obvious way but then you ask it in either a more subtle way or you intentionally yes add some kind of jailbreak to the prompt and now the model complies with you and you're entirely right that in non-English languages you can happen even more it's very hard to solve this problem

and traffic has done a bunch of really good work here when and traffic put out opus 4 along with opus 4 they released another paper looking at doing what they're calling constitutional classifiers these are classifiers that are designed to make it difficult to elicit harmful capabilities initially in biology so that someone couldn't use it to produce a biopin or something like that and this is something that over the last year has become very very hard to jailbreak

the challenge is that whereas there are not many good uses of creating biopins there's

“lots of good uses of finding vulnerabilities and so it's a little bit more difficult of how you decide”

when something should be allowed or not you can imagine versions of this that might just refuse all find vulnerabilities it's much easier to refuse exploits in finding vulnerabilities but yes though I'm this is a very hard problem I understand yes probably better than most people just how hard of a problem this is and I'm hopeful that people will be able to come with something here but yeah it's it's again not going to be easy

codemenders how far are we from LOM capabilities that can actually meant biop code and it's close to real times possible yeah we're getting there these models are really good at writing code they can fix bugs the challenge that we have it's relatively easy to patch the individual bug that you found because you have you know a POC a proof of concept and so you can produce a

“change that makes the attack that you found no longer work there are two challenges maybe three”

the first challenge is does a different version of the attack still work you know you might have

patched one specific hold there might be other ones like it the second challenge you can't break any of the existing functionality the reason why the models are so good at finding bugs is that it's very very easy to check if you've got it right because like the program crashed or something and so you know if you construct an attack and it didn't succeed the program didn't crash and so it's very easy to know this it's very hard to validate that this patch is going to work on all deployments of the

software especially when you have some software that's supposed to run on a hundred different devices it's much harder for the model to check all of this and the final reason why this is hard is that people like the code that they write to look a certain way and sometimes the model just doesn't write it in the way that makes them happy and it may be entirely correct it is entirely functional it works in all the right ways but like aesthetically they don't like the appeal of the

way that the model wrote this fix and so they wonder differently and so for many reasons it's much harder to fix these bugs than it is to find them you could hope that at least for the short term people will be able to take the initial report plus a patch the model said this is basically correct and they'll be able to use their knowledge to much more quickly turn that into like a fully working

version of the patch when I send these bug reports I always attach a patch and even when I'm

entirely unconfident what's doing the maintainers always say like we prefer that it comes with a patch because even if what you have is wrong it lets us know what you were thinking about and we can produce some our version of the patch much more efficiently and so I'm I'm optimistic about these these patches even if people don't end up accepting them just in helping them move get along faster

“depending on how the world goes though I think it's probable that we'll need these tools to”

be able to almost entirely automatically merge patches if things start moving at a rate beyond what people are able to do but there's you know even if we could amend all of the code in five seconds the patch is still need to get rolled out people need to you know update their software all of this stuff is going to take a bunch of time and so I think focusing too much on the like getting the patch immediately deployed is you know some sense the wrong question it's going to be like

shortening the entire life cycle of from when the vulnerabilities are identified until the end product is actually patched which in some cases can be challenging yeah and I think you know that was one of the assumptions I made in my book was while some vulnerabilities we should turn over for patching and some people after they read my book said Nicole you know the most dangerous window sometimes is not when the zero day is discovered it's when the patch is released and everyone

goes to town saying who hasn't patch and reverse engineering the patch and often there's problems of the patch and one things I'm one thing I've learned since I've kind of gone inside the tent and left the New York Times is how hard it actually is to properly install a patch yes I worry so much that this will become a game of time you know that if we're finding these vulnerabilities and exploiting them faster and faster the bad guys are doing that then it becomes a game of how quickly can the

defenders fix these properly with full coverage what is the best way to buy them more time

we can help them write the first version of the patch faster we can help them test the code faster

but some of this is the kind of thing that you know security people have been talking about for a very long time of we need to reduce the amount of time that it takes for people to actually accept the updates we need to reduce the amount of time between when the update is available and when people actually are available to like apply it to their own machines we need to convince people to security matters and they need to apply patches some people just don't do that there's just like

you know a lot of regular problems in security that we're not introducing it's not going to be new but just like you take all of the existing problems that people have and just like

“add an element of time to them that's how I feel about this moment I feel like it's all these”

debt coming do well ones and like only half of them were technical you know it's like our failure to ever really take secure coding and and secure by default seriously in some cases it's moving fast and breaking things and software it's world but it's also technical debt it's also like failure

to resource a lot of this critical infrastructure that can't even get a patch now if they want it

it's like failure to educate Congress and regulators about this technical topics in a way that they're able to action on it and then you look at all of that together it's kind of like a system failure so I know security researchers don't really ever want to dip into policy but I will ask you the question like if you could wave your magic policy wand what's the one thing that you're like I wish regulators would do this I have not even considered the question okay I just want

people to think about the problem and try hard and at this point I want to play the clip from your unprompted talk where you make a really impassioned plea for help speaking not as an

traffic employee like I don't really care where you help just please help like I would be great

if you would like to know help us that in traffic too but like just the world will need a lot of people to be doing a lot of this work and it needs to happen soon like order months I think like waiting a year is going to be too long soon it's not just going to be me who has all of this but it's going to be anyone malicious in the world who wants so really yeah I'd encourage you to find a what try and find a way to see if you know some particular set of skills that you have could

like help us make sure things go well over the next couple of months and over the next couple of years because I am quite worried about how this direction is heading and yeah we'll we need all the help we can get and what do you think even cyber security experts are missing about this shift not very much I think people are starting to like it's not very complicated like sort of they used to be a thing where the only people who could produce vulnerabilities and exploits

were experts humans now many more people can do this and so I think like that's the like

“the only thing we're trying to convince people of it feels a little bit like when computers first”

became good at chess or it go or something like this where you know for a while it was only humans were very good and then there was a shift where the machines became a lot better and could start to do the things that only people could do and I think it doesn't think people that long to recognize that the shift is happening or has happened I think the important question which I don't have the answer to is what do you do with this information when computers became better

as a chess that was fine this was not an important part of society the same is not true for security and I think understanding what to do with this information is the hard piece and I think it's starting to be the case that most security people who are paying attention are noticing this I think people differ in their opinions of what the right course of action is to take and you know no one knows what to do and so this is entirely reasonable I don't think people are missing that much

“though I think maybe the only thing I would point out that people are missing things which”

this includes me too is looking at the exponential weight of increase in progress on these things where you know if you plot how good these model are getting on essentially every metric they're getting much better much faster than we like to think and you know we've underestimated it yeah I mean to like every like what I sort of doing all of this conversation what I am telling

stalled because I don't know I tend I it's hard for me to imagine a future where models got another

“ten times better that may yet happen I don't I don't know but like I tend to only see what we're”

like it's actually very infront of me I feel like most security people are similar that it's like easy for them to believe when I show you the like the machine that like the machine exists it's much harder to really feel the exponential and believe that it's possible that in a year the models might be better than literally everyone and finding vulnerabilities that even after you try to explain to a person they don't understand like when I when I say this it doesn't feel like

it that feels like science fiction when I say it's I don't and so like in very like I don't believe

that this will happen and yet every other time I have said that I don't believe this will happen it has then happened and so like I'm willing to have now wider margins of error on my beliefs on what may come true because I have been wrong every other time that I have said that's not going to happen and you know I still feel like it's not going to happen but you know now I can at least

“today I think it's not going to happen as opposed to saying it's not going to happen so I've learned a”

little bit. If this plays out the way you expect you know what's the world we end up in and what decisions today like as in this week determine whether that world is going to be stable or chaos. I think the primary thing that determines how this plays out is how good the models get and whether or not the favorite offenders or attackers more and I feel like this is mostly a question of nature and we can try and balance it one way or the other by making sure that you know

when we put these models out the first time we bias them towards defensive capabilities but

either it's true or it's not that language models will be able to find an exploit vulnerabilities that humans can't understand. I may have been very confident in the answer whether this would not be possible a couple of years ago I now feel like it's probably not true but you know if it turns out

“to be the case of these models do work and can do this kind of thing then like this is the important”

question sort of like up to us to just I don't know make our way through you know bias it so that we we favor the kinds of things that we want and we just favor the things that we don't that we're very diligent with how we decide to apply these models that we do some way that feels responsible but I don't feel like there's any particular action that this week we can take that will determine what happens here it will play out and we need to make sure that to the best that we can

we make it play out in a way that's favorable but it's very hard to know ahead of time where any potential ceiling on capabilities is or on what kinds of bugs they will find or won't find I'm optimistic that we'll do a reasonable job here but it's yeah it's going to become a lot of work to figure out how to do this well I find a question which is what gives you hope yeah I mean we've done very well for ourselves so far you know security used to be in a really

bad state like I'm I haven't been doing security forever I mean it's but like you know what was it maybe 10 15 years ago I used to give talks where I was trying like when I was explaining security to normal people I would have them call out a website and then live I would demo a excess attack on a website and anything that you wanted I could find I can't do that anymore like security has like you know significantly improved over the last 15 years it was worse before that my

hope is it will get better in the future the trend has been so far that these things are trending to be more secure and occasionally we get things where we have new attacks that we have to worry about but like directionally I feel like we've softwares in a more secure state than it was five 10 30 years ago I'm hopeful that that will continue over the next couple of years that these language models will be able to to find and fix as many of these bugs as they can and things will

work out great and if that's the case like it will only be because people put in a lot of work to make it happen that way it doesn't happen this way just by chance and I'm sort of optimistic that people will meet the moment and do the things that they need which I don't even know

than me but everyone will look at how things are going right now and figure out a way to really

do whatever it was they were doing but faster and more efficiently and will end up in a world that's

better but I don't know this is the best case scenario and I can imagine places you know I can

“imagine it going ways that are not quite so nice but yeah that's what at least gives me hope”

Nicholas thank you so much really appreciate your time today. Yeah thanks for having me.