The ‘But China!’ Dilemma Driving the A.I. Race

at the Times.

“Together we host the Popcast, a weekly pop culture chat show where we speak to the biggest”

musicians, actors, internet celebrities, and more.

We've had Bad Bunny, Olivia Rodrigo, and Hathaway. We've also hosted live performances with Andre 3000 in Erika Badoo, plus we end every conversation with a snack. Catch new episodes of Popcast every week on YouTube, and anywhere you get your podcasts. [MUSIC PLAYING]

Look, I don't know if you are following every article, every tweet, every blog post from the labs right now on AI. But it's frightening. We are in a frightening place. Open AI says it's AI system hacked another AI company on its own, and what the company

called an unprecedented cyber incident. Turns out it may be much worse than we thought.

People don't know that in this investigation, there is a third round of the hacking,

in which it hacked open AI itself. It happened again this time, it's anthropic. Meta is now the latest company to say it's AI agent, broke past the guardrails. And so there is this growing sense of we actually need to do something. We need to pace the frontier.

We need to slow all this down. But if you talk to anyone in Washington about this, where you talk to anybody at the

“AI labs about this, you just crash into the shells of, well, what about China?”

If we slow down, we will lose the air race to China. And as dangerous as it is to build these things we can't control, it is even more dangerous

to have them in China's hands if they're not in hours.

They're going to be killer robots. I'd rather they'd be American killer robots at night, Chinese killer robots. We're on the eve of talks right now. Between Donald Trump and Xi Jinping, behind that there can be talks between Scott Bessent and his counterpart on the Chinese side that are more tightly focused on AI.

The expectations for these talks are not very high, both because of the broad relationship between the US and China. And because neither side religions do know what they want to do. But these talks are at least to be getting. They are the beginning of relationships and maybe frameworks and approaches that if things

continue to get crazier, and action is needed. Maybe they are a platform we can stand on. So I wanted to talk to somebody today who's an expert on China and AI, how they regulate it, how they approach it, the relationship between China and America on this topic. And also somebody who's thought about what talks like this could achieve what is realistic

within the operating frameworks of the two superpowers. My guest today is Matt Sheen, a senior fellow at the Carnegie Endowment for International Peace. He has been closely following and studying China's regulations and governmental structure on AI.

He's been involved in U.S. China AI talks. He has a great sub-stack on these topics and he's the author of the 2019 book, The Trans-Pacific Experiment. The China and California collaborate and compete for our future. Welcome to the show.

Thanks very much for having me. So the dominant metaphor for the relationship between China and America on AI that exists in Silicon Valley, that exists in Washington, D.C. is this metaphor of the race. And the ending of this race is superintelligence. That some company or some country is going to have the moment where they're hopefully

well aligned, safe model, moves into recursive self-improvement and goes, "This got used to get called in the rationalist community the foome moment." And at varying levels of explicitness, people in D.C. to be seem to have this model in their heads that we are racing China towards this kind of supremacy.

“Does China buy this race models at how they see it?”

And I guess, do you buy this race models at how you see it? In terms of does China buy this race model, it's definitely not the dominant paradigm that has been informing AI policy across the country writ large. And it doesn't have the chokehold that it does in U.S. China it's like, huh, okay, that could happen.

We're kind of looking for evidence on this. We see that America is very concerned about this, but China's not taking the steps that

you might think they would take if they were sort of ultimately laser focused on that

type of thing. China is very constrained on compute, they have far, far less compute than the U.S. They have, I mean, some estimates are they have one eighth, the compute of the U.S. May a one tenth of the compute, then the U.S. does. That's right.

The chips, the G, the GPUs that all of these programs, programs, AIs are trained on and then run on. Exactly.

“And, you know, most people think that one of the key determinants of how powerful”

your model is is how much compute, how many chips are you using to train it. And with China being so compute constrained, if they were really just laser focused on

this massive takeoff scenario, you might expect them to start consolidating all that compute,

make your bet on, you know, deep seek or another company and go from there. And we haven't seen that. Actually, the, in terms of the major AI policy documents that have come out, they've taken a very diffuse approach to compute. They said like our number one concern is AI applications, and we want to incentivize every

mayor, every governor, every state on enterprise, want you to look for ways to apply AI to manufacturing, apply AI to your traffic lights, apply AI to upgrading your robotics industry. And those actions of focusing on application, then really diffusing your compute throughout the country are not what you would expect for a government that is laser focused on this takeoff.

It's possible that changes. It changes very quickly, and you know, I think as America keeps like beating this drum

“louder and louder, some people in America beat it louder and louder, you have to imagine”

that it's going to seep into their consciousness in that way or seep into their beliefs about the way this is going, but so far we have not seen that evidence. So every single conversation I have with politicians, with AI lab leaders, about regulating

the frontier AI, always falls apart on this but China problem.

Maybe other things we could do to regulate the pace of the frontier here in America. But China will race forward, but China will create recursive self-improving AI, and either we have the same dangers that we would have if we're here, but now it is under control of a competitive foreign country with a very different political system than ours. So how do you see the, the butt China conversation and the butt China problem?

There's a reality to it, you know, we, this is a, this is a competition. These are the two leading countries. The only two countries that really matter at this point of time. China is not that far behind, and they have outperformed kind of all of our expectations along the way.

And so the idea that you just totally surrender competition, you surrender the playing field to another country that's a geopolitical rival, and that probably has less safe AI practices than you, like that's, that's not a good idea to just abandon the field. But there's also an irony in this, and that, especially when we're talking about regulation of AI, like China has had the world's strictest, most comprehensive, most burdensome AI

regulations on its companies for three or four years at this point in time, and it's during that period of time when they had these heavy and maybe burdensome regulations that they did a lot of their catching up. So the idea that this is just a total binary of like any obligations you put on companies automatically puts you behind this totally wild, unconstrained Chinese juggernaut.

That is just not true. That is not based in reality. You said two things there that can sound like they're in conflict. One is that China's AI practices are less safe than ours. The other is that China has a much more burdensome severe intrusive regulatory apparatus.

So tell me a bit about what they are doing that is so much stronger than what we are doing from a regulatory perspective, and then why you also say that they are in a less safe place than we are. So most of Chinese AI regulations, the early ones, especially, were really focused on online content, on information, you know, when the CCP encounters a new information technology,

“the first question is always, how is this going to affect our controls on information?”

And so when AI came into the picture, that is what they looked at. They looked at recommendation algorithms, and they said, "Why is everybody getting their own news feed? Why can't we sort of set the news agenda?" So they regulate a recommendation algorithms.

They looked at deepfix with sort of obvious implications there. They regulated deepfix. They looked at gender to AI and they did the same thing. And these do impose like real costs on the companies. The companies have to do mandatory pre-deployment testing.

They have to file their sort of safety report cards with the main regulator in China. It's like a real burden of time and money and effort on the companies.

content environment, what we would call censorship, obviously.

“From 2024 on, they've kind of expanded the scope a little bit, and they brought in new concerns.”

They've started regulating AI companions. So they're concerned about the psychological impact on kids. They're concerned about over reliance and self-harm. But all of this so far is not focused on the type of frontier AI safety risks that are really the focus of a lot of people in Silicon Valley, on loss of control, on bio-uplift,

chem bio-weapon stuff like that. That's coming into the Chinese conversation now. But it's coming in much later, it's a much less mature ecosystem over there. It really kind of needs to get up to speed. Something will hear at least in America sometimes, is that much of the closeness in the race

comes from China in different ways, generously building a top-or-models, less generously, sort of stealing them. The key term here is distilling their ways to train a model on the answers in another model gives. So, if that is true, then it's not just like a race, it's like a race in which the people are

tied together. Like the faster America runs, like the faster China's going to run, because it's actually amazing in America too, how close a lot of the different labs are, you know, they're just

like always like a month or two around each other.

How much do you buy that everybody's bunched up because, in fact, the race is governed by the leader dragging everybody else with them?

“I think it definitely plays a role and maybe a pretty significant role and just for”

audience to visualize this, I saw a great meme of this where it's a speedboat pulling like a, when you call it a weak surfer, you know, a person behind who's essentially, you know, trailing behind the boat, going over the waves and the people on the boat are like, they're so close. We need to go faster.

Yeah, right. Just pulling them along with you. I mean, this is one of the arguments people are making about, you know, when all these AI labs in America, like, well, we can't possibly slow down because China will speed up a boat.

China's going so fast because you're going so fast. Maybe if you slow down, China would be going so fast either. Yeah. But at least in part, I think we can say the distillation probably plays a significant role in cutting into the US lead.

My sort of mental model for it is, you know, China's so short on compute and that distillation is probably a way for them to essentially, like, train more efficiently, increase the intelligence more efficiently given that they have so little compute. So it's essentially making up for one of their biggest shortcomings. I don't think that if we suddenly found a way to block all distillation, the Chinese labs

would just stagnate, you know, China has, you know, in AI nuclear weapons and almost every technical field over the last 30, 40 years, they consistently outperform expectations and just do things that we don't think they should be able to give in their level of economic development and their capabilities.

They have an amazing AI research ecosystem over there, like one of the reasons we're

ahead is because we keep taking Chinese AI researchers and employing them in our labs. We are, you know, siphoning off a lot of their top talent. So they have a really thriving ecosystem on its own, but I do think distillation plays a big role. You know, it might be the difference between six months and a year.

It might be the difference between six months and two years.

“We don't know, but I think the fact there's pretty strong evidence that the Chinese labs”

are doing it and they wouldn't be doing it if it wasn't to their benefit. How does China see us on AI? How do they see what our goal actually is, what our goal is vis-à-vis them? So as we talk about this question of can these countries cooperate if they need to, what is China's perception of America's AI industry?

I think the number one perception is that the U.S. wants to hold China down and wants to constrain China, you know, especially with the export controls, it's come under Biden, I should say. Yeah, export controls under Biden on these advanced chips. This uses itself as being sort of boxed in by this hedge-man that wants to kind of keep

trying an permanent position of subservience. That's a kind of a meta narrative across Chinese modern history, and it's one that's crystallized in AI.

So I think in some ways, that's the first thing.

Another element is they see us often as being pretty irresponsible, deregulatory, just let it all, let it all rip, let it all hang out. So they see sort of chaos within our government. They say, oftentimes, and actually in the-- That's crazy because it looks so orderly from here.

In the run-up to these potential AI talks that might be happening in the next couple of weeks, China is issuing sort of op-eds by its state media where it kind of lays down its markers. It tries to position itself in advance of the talks. And one of the marks that they lay down is, you know, America wants to lecture us.

They want to define all the stuff unilaterally, and they don't even impose any requirements on their own companies. So don't come to us with that stuff unless you're going to take care of your own house. It doesn't seem totally unreasonable to me. Not totally unreasonable.

Self-serving, in a way, but yes, I mean, too. But I think it's interesting. I mean, this is a point I was made, but to China, we look like the ones who are not regulating AI. But, you know, there might be this whole discourse and the countries need to cooperate.

But in fact, what they see is us racing forward, trying to attain AI supremacy before them, and kind of, you know, we're a diffuse way calling for regulation of something that might like destroy all of humanity, but we're not actually doing any serious regulation of the thing that might destroy humanity. And, you know, when I read some of these state op-eds, you're talking about the way they

end up framing it is instantiary, that I'd ever know how, when I talk to people in Chinese government, their sense of American politics is actually not often as sophisticated as I would imagine it to be, maybe I don't get to talk to the right people.

“But I think sometimes they look at us and assume that the things that are said are have”

a more orderly structure. And in the way that everybody has to use, she's language there. But if you look at a thing that doesn't make sense and you come from their perspective, well, maybe the reason doesn't make sense is the counter party is not serious. They're just making a bunch of different moves that are all different forms of strategy

to stay ahead in the race. As a macro perception, I mean, I see this all the time talking to Chinese about the US political system, talking to Americans about the Chinese political system is if you don't understand the system at a pretty kind of ground level, if you don't have an intuitive feel for the two systems, the tendency is to look at the other side and to connect a bunch

of dots and see a grand conspiracy. And it usually is a conspiracy against you. And the CCP has a very conspiratorial view of the world. They see conspiracies everywhere. And there have been times when the United States and other countries have conspired to hold

down China, but the way that they will connect dots that from a US perspective are just

wildly unconnected is, you know, it's worrying and we basically do the same thing over

there. We do not have the ability to see through rhetoric that's like, that's just what they have to say.

“That's what they have to say to start and then the real protein, the real like meat of this”

conversation is here. That's the real signal. So that's kind of a permanent issue in US-China Mutual perception. I don't know, from the Chinese perspective, and from the American perspective, as somebody who does have a good ground level intuition for who is saying what in our system and why.

You know, the Biden export controls on ships were quite explicitly an effort to maintain AI supremacy for America, which is not a crazy thing to do for a country, but if you're the country that is being denied, the exports, I think that's a little bit provocative and then Trump comes in, right? You know, orient is an entire trade war against you.

You know, Dari Ahmeday, the leader of arguably the most important American AI company in

anthropic, he had this big essay on the adolescence of technology, and he says of China. They have hands down the clearest path to the AI enabled to tell Terian nightmare I laid out of buff. It may even be the default outcome within China, as well as within other autocratic states to whom the CCP exports surveillance technology.

I have written often about the threat of the CCP taking the lead in AI and the existential imperative to prevent them from doing so. So, you know, from China, I see the leader of the frontier AI lab saying it's an existential imperative to keep China down. I really worry about this atmosphere being the one in which these momentous technologies

are potentially being developed because when I talk to American policymakers, they don't really feel like they understand what is happening in China. And there's a lot of skepticism that an agreement with NSUB verifiable or followed.

“And I think China is pretty good reason to be skeptical of what our real intentions are.”

Our intentions to make AI safe for everyone, or our intentions to make sure America is

a first one with the AI that ensures American dominance of the global order for another

hundred or two hundred or five hundred years. And that kind of miasma of mistrust is a tough space for negotiating. It's a very tough space, you know, I mean, this is arguably one of the worst times for a technology. This momentous to be coming online, you know, at a real moment of deep geopolitical competition between two superpowers that distrust each other that have, you know, interests

that are fundamentally in conflict in some areas. I think one of the key points here is, yes, trust is good. It is kind of the lubrication

this work or not. Like the thing that makes this work or not, in my opinion, is going

to be whether or not both sides for their own reasons genuinely believe that this is a potentially catastrophic risk and believe that they need to take action on that for their own safety and security. The Chinese technical AI community needs to like believe deeply in its bones that if we push into this area without the right safeguards and testing in evaluation, then we run

a real risk of losing control of this technology. And, you know, Xi Jinping, other Chinese

“leaders, they do not want that in their own country for their own reasons. And so I think”

that is the area of mutual interest as opposed to mutual trust that things have to be built on. So, when I'm looking at, you know, US China interactions in this space, I think that

one of the most important things, at least as a starting point, is can we build up a mutual

understanding of the risk? Can we share information? What are we seeing about emerging risks and how do we test for those risks? What are the best practices for securing a model that has cyber capabilities that you don't understand? What are the best practices for sort of defanging a model that might have bio capabilities that you don't want? Like that work in the United States is just much more mature. The regulation in the US is much less

mature, but within the companies, within the labs, this has been worked. They've been doing seriously and investing a lot of money and a lot of people and resources in for a long time.

“And, I think that's kind of one of the misunderstandings in China as they look entirely at”

our regulatory ecosystem and they say you're not doing anything. Whereas the labs here are

voluntarily doing far more on this than the Chinese labs are doing in sort of a mandatory regulatory environment. So, with all that work in that knowledge that we've gained, can we find ways to safely share some of that with China to essentially seed bolster and help grow their existing AI safety ecosystem. They're technically AI ecosystem over there that is concerned wants to do good work on this, but is just starting five plus years later and just has

invested far less people, money, computing resources in that work. So, you have some personal experience here. You've hosted some of these China, US, AI dialogues, not the official high-level US government ones, but these more informal ones that are, I think it's some way supposed to help lay long-term groundwork for this. What are those felt like? What have you learned from interacting with Chinese colleagues and counterparts? Like give me some of your texture on

this. These conversations are normally very, very technocratic, you know, maybe a little bit

“boring, productive, but, you know, calm affairs. And I think one thing I've seen a couple times”

when you get a little bit of heat, like a little bit of spark in the conversation happens, oftentimes when the Americans are pointing at, you know, these trend lines in AI and bio, or these emerging safety issues scale, you know, say, like, look at this, like, why aren't you more concerned about this? Why aren't you doing more on safety? And you'll see the trend side getting actually really frustrated and be like, you guys don't get it. We, we are doing a ton on safety.

We have these AI companion regulations. We have mandatory labeling of AI generated content. We have all of these rules and regulations. They're just not the exact thing that you want. And I think that's a lot of the disconnect between the two sides. Is the Chinese side feels like they have been doing serious work for a long time? And that's just not seen. It's not understood or not respected outside of the country.

We're learning the morning and he answered seven decades in space. We want to see what's beyond the next star. A thousand consumer product guides. Today I am testing seven mattresses. 60 super Sundays. They were probably the best team in the league every single year. 53,500 puzzles. I use a different starting word every day. 25,000 recipes. And I love a one-can recipe because I don't love washing cans.

The New York Times, celebrating 175 years of helping you understand the world and make the most of each day. Subscribe now for a special offer at nytimes.com/subscribe. I think talking about this requires some sense of how China's AI industry differs from ours. How do you describe the difference between what the sort of culture and approach of the frontier AI makers in China is compared to sort of the anthropic, open AI, you know,

let's say that's where the cultural similarities are the closest and it's much more of the broader

AI industry and the policy ecosystem where the differences are much wider. And I'll start with

“that sort of broader ecosystem and then kind of bring it into the frontier. I think in many ways”

in the US, a huge portion of the AI industry and the policy world really started from this idea of one day we will reach superintelligence. That is the goal that we'll bring with it catastrophic risks. That will require heavy focus on safety. It's been this magnet that's drawing us into this future and that's of course especially true at anthropic and at open AI in DeepMind. But I think that's a pretty significant portion of the policy ecosystem here too. And so it's almost like

it's like this teleological thing of we're endlessly being drawn towards that. And in China,

there are some people, a couple of people who lead the frontier labs who have that take. But in terms of a broad culture throughout the industry, the investors, the engineers, the policy people of government, that has not been this kind of magnet drawing them into the future in the same way. It's more like they've been developing an industry, developing applications of it, as they develop a new application, they develop new policy to deal with that. It's a pretty

fundamental difference in the way the ecosystems have kind of grown and expanded. That thought about this a lot in the American context. Something I sometimes say to American policymakers, like where do you start? Well, someone say where you start by starting, that you learn how to regulate things, you learn how to legislate on them, by regulating and legislating on them. And the Chinese approach that they already

are learning day by day, how to interface with their labs, how to craft regulations and revise them, that they're building up practical regulatory experience, that then, if or when they need to come in with things that are much more potent, they sort of know how to do that. And so it's not that their regulations are what the American Frontier labs believe

“are needed or what I believe are needed. They're not, although frankly, I would like us to be”

more thoughtful about AI companion bots than we've been. But that we're actually behind in practical experience here is meaningful. In some ways, it's more meaningful than, like, neat conceptual arguments about, you know, the worst case outcomes. You know, one of the characteristics of Chinese policy making, but especially in AI is that it's very iterative. They'll roll out a regulation. They'll see how it's working. They'll roll out a technical standard that specifies it.

And it's not quite achieving in that they want and they'll roll out another regulation that

basically just overlaps on the first one. And with each one of these, they've built up these reusable

regulatory tools. So the main one is this registration system for AI models. And that the CAC, the cyberspace administration of China, the main regulator there, needs to be able to read, needs to be able to understand. In some cases, maybe do the tests on their own. And, you know, when they first started this in 2021, the regulators were totally out of their depth. Like the CAC is traditionally an internet regulator. It's focused on, you know, content and political content

stuff like that. But that was four years ago. And, you know, that, like I said, has been focused initially all on this controlling content. It's now been expanded. These other areas about, you know, emotional dependency around sort of labeling of content. You know, can you impose and then remove a label on AI generated content stuff like that. But they have been sort of constantly in touch with the labs for her now about four years. So that's, that's a lot of regulatory practice and

regulatory muscle. Part of the question is, with these frontier safety risks, is that something that you can kind of just easily tack onto this? Is it just another test that they run? Or is it something

“significantly more complicated? And I think, you know, it's kind of in between the two. Like they”

have a lot of mechanisms. They have a lot of habits and, and touch points that are very good. But they do need to increase their technical capabilities in these specific areas of frontier risk and control. Tell me about the way in which China's evolved to emphasize open weight models versus our main models and throughout the open AI are closed weight. And maybe begin for people don't know this terms by defining them. Sure. So closed models are the way that when you use

chat GPT or Claude or Gemini, those are closed models. And you interact with it kind of on the company's terms through their portal. You cannot sort of edit the model. You cannot download it to your computer and run it yourself. An open weight model can be downloaded from the internet. And if you know how to

You can sort of tailor it to your own purposes. If you need to use a model, you need to make thousands or tens of thousands of calls of it every single day to run your own startup. You do not want to be paying anthropic and open AI for every single one of those tokens. Every time you ask the model a question. And this has been a divide that's really emerged starting, especially in like 2024 or so,

where it wasn't always a given that this is how the two ecosystem would develop. But the way it

has developed is that Chinese labs primarily release their models open weight and the U.S. labs primarily release them close weight. So at the very beginning, sort of in the aftermath of chat CPT, when China's first regulating generative AI, they actually started off taking a relatively cautious approach to open weight models and putting regulatory burdens on them that would have made it much harder to use open weight models in China. And the reason they were doing that is

because at the time, the primary, the leading open weight model was Lama from Meta, from Facebook. And China was worried, you know, our Chinese developers are going to take in Lama. They're

going to build their applications on top of it. It's going to kind of poison our ecosystem with

their information that we don't want. But over the next year or so, we saw a couple of the leading Chinese labs decide to release their models open weight. And you know, deep sequels really the big kind of kaboom moment in this. In that, when they released it open weight, it took the world by sore storm, you know, the entire global AI community was able to actually play with it and look at it and see that it really is that impressive. And since then, it's kind of snowballed from there.

And I think in some ways the CCP might have stumbled into this outcome, but I think they're pretty happy with it. And it makes sense both for the companies to a certain extent and for the government. Doesn't it make it harder to control these models? One thing going on in, and I mean, this is to debate in the American AI ecosystem where, you know, Dio, Amade and Sam Altman and like they often find Mark Zuckerberg over this. There's a view that when you get these very

powerful models like something like, you know, mythos, which has these incredible cyberhacking

implications. You don't want anybody to be able to just download mythos and do what they want with

“it. I mean, these are potent things you need to have some control over them. The CCP is,”

both has a more aggressive regulatory stance and is more control obsessed than the US government tends to be. And yet China's the center of the open weight ecosystem, like how did that, how did those things hold together? And I think one factor is like, what was needed for the companies to be seen as globally competitive? You know, I think if it deep-seek and late 2024 had just announced to the world, hey, we've got a great model, man, feel free to use it. It'll go to

Chinese servers and we'll give you back the answers. I think there would have been a level of suspicion about that. I don't think it would have seen this rapid global proliferation because people have a certain distrust of Chinese technology. And by releasing it open weight, they can essentially say, hey, you look at it, you change it, you do whatever you want to it. It's that good,

“and you will see that. And then we'll figure out how to make money other ways. I think that's part”

of the business aspect to this. It's hugely reputation enhancing for Chinese companies. And now for China's AI ecosystem as a whole to release these open weight. And therefore overcome some of the suspicion that normally falls on Chinese companies when they go global. So that's one part of it. Another part is that, you know, frankly, these are probably undermining the future valuation of anthropic and open AI. And I don't think that that was a scheme going back to 2023 or 2024,

when this world we're in wasn't totally foreseeable. Now that they're here, I think it says one of that, you know, that is to our benefit in terms of a long-term competitiveness. Do the Chinese models give extremely different answers or come with very different approaches in the American models is a more fundamentally different worldview detectable if you kind of run testing across the two

“rate of more skepticism of democracy generally. I think American models very much do have an”

American outlook on the world. Do you see the models as being very different when Americans are talking to deep-sake? It depends a little bit on how you're using the model. Like the most censored version of a Chinese model will be when you're using it through the app or you're using it through the API when you're going to deep-sake.com and asking it questions. That's the version that has sort of the most controls built into it. If you download the model, an open-weight model, you download it,

you run it on your own computer, you will have a different set of safeguards, not entirely removed

add new training data. You can change the way that the model functions. And some American companies like curse or others, they feel that they can get the models into a place where they

“are not propaganda machines for the CCP. And I think the CCP would say that there are a lot of”

countries, Singapore, Southeast Asian countries that are building sovereign AI models on top of these open-weight models. Say, add your own language data, add your own cultural data to sort of post-training these models in a way and tweak them to your needs. So that's part of China's pitch to the world, to the global south is America is the technological hedge-man that wants to restrict your access to this technology. It wants to impose its own values. It wants to, you know,

blood out your own local culture and it won't let you in any way adapt or play with their models. We're just giving you the model and you can do with it what you want, you can change it, you can adapt it and you can run it for just the cost of, you know, the cloud computing that you're using. That's the pitch. I don't think it's, I think it's 100% honest. I don't think it's actually going to play out in that exact way, but that's the divide that China has been trying to pitch

to the rest of the world. So I find this really interesting. So the way the internet developed and a lot of the modern mega online platforms developed, China and the US have pretty separated digital ecosystems. I mean, you're not using a lot of Google search in China. We're not using WeChat here. We are much more integrated on AI than we are on what came before. A pretty large number of American companies, which are consuming AI tokens at a level where you actually have to pay real money to keep

going. They're using Chinese models. I mean, Airbnb, Coinbase are famously using Chinese models for significant parts of their AI infrastructure. So this is not just like the Chinese ecosystem over here and the American ecosystem over here. They're already somewhat combined. I don't know how much open AI or clutter a lot in China because I assume they're not censoring in the way that the CCP would want them to. But the Chinese models are here and in widespread commercial use.

Yeah, this is really one of the great ironies of this current AI moment that we're in. The firewall really came down and kicked out the American technology companies, Google, Facebook, Twitter, et cetera, 2008, 2010. We had very separated product ecosystems. They were building their

own products. We were building our own products. The products didn't really cross over. But we always

actually had pretty integrated sort of technology ecosystems. You had a huge flow of Chinese people coming to the US and working in companies. A lot of them would go back and cross pollinate the two ecosystems with ideas. There were a lot of American money going into Chinese startups. A lot of Chinese money going into American startups. It was all quite integrated outside of the product layer up to about 2017, 2018. That's when we began the American project of technology

decoupling with China. We want to pull apart these connections because we think this is how China is catching up. It's catching up because they're stealing. It's catching up because they're

learning at our universities, et cetera, et cetera. The first Trump administration to a certain

extent, the Biden administration did a lot to cut down the flows of people, to cut off the flows of money, to kind of reduce the flow of ideas between the two ecosystems. That was

“relatively successful. I think it probably would have continued to be quite segmented in this way”

except for the fact that the Chinese models going open weight. It's almost like the open weight ecosystems kind of reintegrated these ecosystems in a way that I don't think anybody could have foreseen three, five, ten years ago. Xi Jinping recently gave a pretty big speech on AI. What seemed new to you in that speech? So this speech was up the world AI conference, which is China's premiere AI event every year. They try to get the whole world to come out. It's a big to do.

And this is the first year that Xi Jinping has attended and given a speech there. So it's really

as a biggest AI speech, maybe ever. It's been watching very closely. I think a good portion of it was China's pitch to the rest of the world. It's the one I outlined earlier. And then the other part that stuck out to me was the conclusion. And it ended with some pretty striking metaphors using an ancient Chinese idiom that I don't have left at the top of my head about how sort of the

“wise adapt to circumstances. And they do not get stuck on one path. And I think the one of the key”

terms was that we need to be able to act to forced all loss of control of AI. And this has been a long-term concern in the West. You know, as AI get out of our human control, it's a long-term

Are they talking about party control? Are they talking about, you know, the control of an

operator? Or are they talking about human control over AI? And so he put down a marker there around loss of control. And I read it as leaving this space open. These are all signals to people in the system. When he says forced all loss of control, that means that AI researchers all throughout China. When they're applying for the next grant, they're going to use that term. If you use a term that was in a big sheet speech, you're just more likely to get grant funding. Like these things are

markers that everyone, the policy makers are interpreting and they're trying to figure out how can I do that in my area. Researchers looking for funding are adopting it. Companies are looking for

signals about what will and won't be sort of, you know, in bounds. So the words really matter.

“And I think that conclusion was at least putting down some markers that are shown that China”

is shifting pretty quickly on a couple of these fronts. My model this is that political pressure, political possibility doesn't build linearly. And it particularly will not on AI. That what happens is you have issues. They stagnate. They are not at the front of the agenda. And then something happens. And the window of possibility blows open. So I think in America here, the open AI, hugging face hacks, the, I mean, almost more consequently, the fact that open

AI systems hacked open AI and took over a part of their research clusters, the kind of social engineering and effort to upload malicious code from, you know, frontier anthropic models that this sort of swarm behavior, the peer behavior that, that this summer of weird AI incidents has blown this open a bit in America. And now I'll start to remember talking about pacing the frontier. And as I guess a question is, China knows these things are happening.

“So how are they responding to these same events that are transforming our conversation?”

So you're right that they're taking it in. There's tons of coverage in Chinese state media about the hugging face incident about pretty much all the major safety developments. You know, recently there was an anthropic researcher who resigned and who issued these pretty dire warnings like that was in state media today. I was reading that. And so they take it all in. They are much more tuned to our conversation than we are to theirs. I think part of it is they react more incrementally

than we do. And that's in part, I think due to this kind of long-term different relationship to say superintelligence and catastrophic risk. For a lot of Americans who have been thinking about this for, you know, a decade, this they've been predicting this will happen. And then this happened and it's the ultimate illustration that they were right all along and it's happening. For the Chinese side, this is just newer. They're taking it on board and they take the data points like, okay,

that's interesting. It hacked out of a system like was this issue with the safeguards, with the tooling, could this have been, you know, constrained with pretty like mundane security measures, or is this a sign of something bigger. And it's, it's been really interesting to watch for the last year, really year and a half at this point. As a lot of this safety terminology has worked its way into important Chinese government documents and important technical standards documents by

“their lead regulator. I think just last week, Chinese main AI regulator, the cyberspace administration”

of China, issued sort of a public statement on its top five AI risks. And number two on that list

included, what they call extreme loss of control, Ziduan Shukong. That's the first time that I've seen

that specific phrase, extreme loss of control. They'll talk about control ability in the past when they talked about that. That was more like party state control ability. But around 2021 and then really in 2023, they started talking about human control over AI. And now it's essentially it's working its way into more and more practical and specific AI policy and technical documents. And so they're taking it on board. I think they are, you know, essentially they are moving in

the right direction on a lot of this. And to me, the big open question is like, do they move fast enough? I'm Winnello. I write the game connections, one of the puzzles from New York Times games, and I love horror movies. I love my dog. And I love trying to trick you. I'm Tracy Bennett. I get to pick the word word every day, which is not as easy as it sounds.

New York Times games are made by people like the ones you just heard from go to nytimes.com/games

to start playing today. There was another incident that it didn't, I mean it literally made headlines here. But it has not been greeted as such a big deal here, even though to me it was quite scary. So frontier AI models were able to find a vulnerability in wechat, which maybe you can describe for an American audience, the centrality of wechat to the Chinese digital ecosystem. And they're able to build this attack on it that they dubbed we worm. And it would have given

control of somebody's phone just by calling that phone. Now, this was then conveyed to Tencent, the developers of wechat and the, according to them, the vulnerability has been patched.

But it seemed like a hell of an example to China that the hacking capabilities here at the point

“where it could compromise major foundational Chinese digital infrastructure. How has that been covered?”

I haven't seen that much coverage of it in mainstream media. And that might be because when an American company finds a huge vulnerability in like these central, you know, digital platform in China, it's not really seen as in everybody's interest to publicize that a ton. So that might be part of it. I think another part of it is that this is a question of like offensive hacking capabilities. And I think for them, like the real wake-up moment for that came with mythos when they, you know,

the US develops a system that they're not releasing to the public that they're only releasing to a set number of companies and also the NSA. And China has to assume at that point that it is being deployed, you know, far and wide against Chinese systems. So a lot of the discussion in the aftermath of that was about how do we harden our own system against these type of cyber attacks. You know, in some sense, this type of cyber warfare between the two countries is inevitable

in long term. And like it's almost like we shouldn't take it too personally. China shouldn't take it too personally when we, you know, hack them in a bunch of ways. We shouldn't take it too personally when they didn't. That's that's kind of our job and their job. It's, it's the, it's the, it's the question of like when something happens. A lot, a lot, a lot, not often there. I'm not a question, but yeah, just just putting a pin that, uh,

yeah, a lot, it's, it's the job of the NSA and it is the job of the MSS to try to hack each other.

There should be limits, you know, critical infrastructure, all that kind of stuff. But in some ways,

“I think that's baked into both countries worldview that we're both going to be using it intentionally”

against each other. The issue is when it's something that's not being done intentionally by a state. When it's happening by a non-state actor that neither of us wants these tools and the hands of. When it's out of control and neither of us has the ability to sort of understand or control it, those are kind of the areas where I would expect, you know, some level of overlap and it's all, that's, that's, that's a newer phenomenon that shines grappling with. What about recursive self-improvement?

Sometimes we talk about loss of control. I get as a passive thing. I don't intend to lose my keys, but I do it all the time, loss of keys. Recurs of self-improvement is handing of control over to AS. Right? Recurs of self-improvement is where the AS systems autonomously build the next system. Right? That they're now moving faster and improvements than human beings can possibly keep up with. We are dependent on the AS system to tell us what it is doing. We're dependent on those

descriptions of what is happening being correct. We have seen AS exhibiting this up to behavior. We see the frontier lab saying our capacity to monitor is already degrading. We are seeing a lab that are currently racing towards recursive self-improvement, expressing very high levels of concern about what it will mean to achieve this thing that they are desperately trying to achieve. It is a very strange situation. And then, of course, when you say maybe you

should do this, you get but China. Now, she also keeps talking about how AS should be developed by humanity. She'll be under humanity's control. Recurs of self-improvement is the simplest way to give up human control of AI. But to me, that's a place where some international standards seem really

“needed. And not like we can wait five years on that, because, you know, I think the American”

AI lab's think we're going to hit RSI in the next 18 months or so. Is there the possibility of cooperation on this or constructive dialogue on this or is something that is outside a crisis point not even possible than the conversation? So, like a lot of these concepts or developments, RSI is

talking about RSI this summer, like mid-late summer, whereas I think this has been, you know,

in the conversation and Silicon Valley for much longer than that. Now, they say, wow, you know, this is the next thing, this is where things are going, because in many ways, as has creative and innovative as the Chinese ecosystem is, they still do a lot of times essentially look to

“Silicon Valley for these type of directional shifts. You know, what is the next paradigm?”

And so, with so many of the US labs, beating the drum on RSI and saying, like, this is where it's going, I think they're, the train labs are kind of following into that space. I don't think they have as much experience with it. I don't think they've done as much technical work with it,

or maybe even thought as much about the risks of it. I do think that this is one of those places where

we might just have to draw a line. And whether it is done bilaterally at the exact same time, or whether it is something done unilaterally with the expectation or intense negotiation to try to get trying to agree to the same limitation, that might be the point. I mean, this is a word to be more likely to get them to agree to it if we drew that line unilaterally. Yes, you know, if we do it unilaterally, it increases the chances that China does it. You know, it also increases

some risks that you do it unilaterally and then China catches up or forges ahead. So that's a double edge sword and I won't pretend that it's just like the solve all for us to do it unilaterally. But

you know, a lot of this is a matter of sending like costly signals. You talked about all of the

misinterpretation and sort of conspiratorial thinking between the two sides. And so when we just say a bunch of stuff about the dangers of RSI and we talk about it, we don't actually have any regulations. We don't do anything about it. We are not sending any costly signals and our lead in this technology allows us to have access to information to see threats and to see risks that they just haven't seen yet. They're not going to trust everything that we say. They're not going to trust all the

information that we share. But that is it's a card that we can play in these areas. And whether it's a unilateral pause or it's an information sharing mechanism that we set up now, something where we share information on incidents like the hugging face incident. You know, if the US and China are going to sit down and talk about AI in the coming weeks, that's a great opportunity to put on the table a lot of information. That's not sensitive in the sense that it doesn't

undermine the US lead. But it lays out very clearly and in deeply technical terms like this is what we saw and this is why we're worried about it. Do with that what you want. But this is what we saw.

“I think that's that's the space that we want to be working in. And then we want to be”

other than just sharing the information on the risk. We want to be trying to plant seeds or enhance the technical AI safety capabilities within China. They really need to catch up. The practices there are just much further behind the leading US labs. The capabilities aren't that far behind but I think the safety practices are further behind. And so it's in our interest, you know, loss of control if something goes out of control in China doesn't stop at the borders there.

So it's in our interests for them to have good safety practices and they have a lot of catching up to do. So we're on the cusp of there being talks. They're being led on the US side by Treasury Secretary Scott Bessent. He's got Chinese counterpart. There's also going to be the Trump and Xi Jinping meetings coming up. I think that you've sort of been pouring a little bit of cold water for people and what to expect out of these. But what to you is a constructive outcome here,

right? The sort of beginning of a space in which, you know, possibilities can emerge. And what to you would be, you know, a negative signal about what's possible. So I think a negative signal would be a statement that sounds good and gestures at some thing that nobody has a problem with. So if the two sides get together and they say, look, we both care about AI and we both care about child safety. And so, you know, we each affirm our commitment to child safety and AI. And it's like,

sure, yeah, that is an important issue. But that is not kind of the key issue between the two countries. I think that would be a sign that we hadn't really, we didn't really have traction yet, at least. I think what would be positive to me is one, establish this as an ongoing recurring dialogue that will have staff that will sort of build over time. So, you know, maybe the US-China Strategic AI dialogue that meets every four months. We've, in the past we've established these on security issues,

“on economic issues. And you need to have a real structure in place where it's not just a one-off.”

The next thing I'd like to see maybe two things. One would be a working group between the leading

Chinese system. So in the US, you know, there's a lot of bureaucratic fighting over this, but the

“center for AI standards and innovation, the KC, is really, I think, the center of knowledge”

when it comes to testing a frontier models. China has a new working group, I'll work in group nine, that is essentially tasked with developing technical standards related to AI safety, broadly defined, but also starting to look at catastrophic risks. I think something that creates a space where those two teams can safely talk to each other and exchange best practices, say this is what we're seeing, this is what we're worried about, this is how we test for it and this is how we mitigate it.

Maybe the second thing would be a crisis communication line, a good way for the US and China to

get in touch if something emerges rapidly, that is a AI driven crisis that could get spun even further out of control because of US-trained dynamics. So, you know, the hugging face incident somewhat luckily, open AI hacked, hugging face. They were able to more or less get in touch with each other and sort of sorted out and it wasn't a big deal. And that was somewhat untangled by hugging face using Chinese open-white models, Chinese models, which in which

Chinese state media quite enjoyed. Absolutely. You know, imagine just a couple different like twists on that, like what if that is a deep-seek model that's hacking, hugging face or what if it's an open AI model that for whatever reason decides it really needs to acquire more compute resources. I know I can find this, you know, insecure, compute cluster that happens to be in,

“you know, judge young province in China. What happens if it takes over a compute cluster there?”

Like, how is China going to read that signal? What's their response going to be? Or even if you take it out of a just a purely bilateral context, if we start to get information intelligence that a swarm of AI agents is draining bank accounts in Pakistan. And we don't know what their intentions are. We cannot read their communications and we cannot shut it down right away. We need to be in touch with the other leading AI superpower on that

issue. So, a way that these two countries can get in touch share information in a crisis situation. It's a very fraught issue. We have had a lot of these crisis communication lines on military

issues and the US complaint is always the Chinese side doesn't pick up the phone when we call them.

And that's a real issue. I think, you know, one mitigation to that is to use somewhat ironic, not use a phone but use a fax machine like literal faxes, literal faxes. And it's, it has a logic to it too because the political system there is not a system of empowered individuals. It's a system of committees and a system of documents. And so when our, you know, Treasury Secretary someone who feels very empowered on the US side picks up the

phone and it's like, give me some answers, you know, Hilly Fong or other Chinese counterpart. And we're not really ready to give you answers on the fly. Much better to send a document over to their system that they can review. They can bring it to their committee. They can come up with their understanding and response and send something back. So, something in that vein

“that at least puts a little bit of a safety net on these incidents that I think, you know,”

it's pretty like something like that is pretty likely to happen in the next year. Everything you're saying here makes sense to me about the fax not phone dynamic and documents. And, but man, when the whole thing we're facing down is the acceleration of an incidents and things happening at faster than human speeds. And now we're dealing with governments that work at frankly slower than human speeds. It really creates quite a mismatch. I mean, you know,

the big language you're right now is pacing the frontier, but we don't even have a plan to keep the frontier from accelerating. I forget pacing at the moment. We are currently accelerating the frontier. It's concerning. It's deeply concerning. And if it's a matter of a race in decision-making between an agent and a person, like the agent is going to win that race, I think, you know, we hopefully won't be engaged in that very specific race. And I think the Chinese system,

it's interesting as it's in some ways it can be so slow and incremental and that can be so fast, you know, their response to COVID was initially so halting. It was screwed up by information gaps where the local officials don't want to report the bad news to the higher officials. It has all these kind of, you know, neuroses and idiosyncrasies. But when they decide, like, we need to shut down this city. We need to wall in this city and not let anybody in or out. That happens pretty fast.

And I'm not saying that's the solution on AI that had a ton of human costs. It always,

when China takes action like that, it always has a ton of human costs. But each side kind of has its strengths and weaknesses in this area. And I think there is a chance that while China's moving pretty incrementally now, as the evidence builds, I think there is a chance that they shift

are coming into? You know, Trump rose in politics with a very skeptical, to say the least,

take on China. In a second term, after the beginning terrorist deliberation day,

they tried to pivot toward trade growth. China, China fought back. We functionally back down. And since then, for all the blustery, sometimes here, Trump seems to be trying to build a better direct relationship with Xi. And so to what degree is the current status of the U.S. China relationship? In particular, the Trump Xi relationship may be more flexible than one might assume

“just knowing sort of where it was at the beginning of Trump's second term?”

Yeah, you know, Trump does both extremes when it comes to China. You know, he totally, he really changed the direction of American policy, you know, much more hawkous direction, taking like seriously, you know, harmful aggressive actions against China. And at the same time, he seems to personally really like Xi Jinping. He seems to admire him. He seems to see a kindred spirit in some way, and these two strong leaders of countries.

And I think that affects a lot of U.S. policy making. There's a lot of evidence that, you know, different potentially aggressive actions against China been watered down, because Trump doesn't want to screw things up ahead of the meeting. You know, during the Biden administration, I think a fair number of people were thinking, sort of thinking ahead to saying like, maybe we do need to be engaging China on AI safety. Maybe we do need to be sharing information. But they felt very

constrained by the idea that, well, if if Democrats do that, if the Biden administration does that, we're going to get roasted as soft on China. And, you know, we're going to be seen as kind of, you know, giving away the store on AI. And Trump just creates his own political gravity, his own political environment, where that same action will be read in a very different way

“that he doesn't have to share the same concerns as past administrations. So I think that's”

that's a dynamic. On the Chinese side, I think she is a much more systematic thinker and a much less, much less relying on these individual relationships. And seeing this is a structural long-term contest between two systems, between two countries. And the, you know, the day-to-day wavering of, we love China, we hate China, we're we're blockading, we want to have double the investment. I don't think he sees that as a meaningful change in the overall trajectory

between the two countries. And so, well, I think the kind of the one-to-one relationship, you know, does she like Trump, not not all that relevant, but the the changes in the overton window of what we think is possible when it comes to engagement. I think that is meaningful. You're pretty calm, seeming person, temperamentally. If you're talking honestly to maybe, you know, Chinese counterparts who are regulating, but not on the most profound set of risks,

or American counterparts who are worrying but not actually doing all that much. What's your real level of alarm? Like, what would you tell them about the moment we're actually, and now what you think is possible, not what you think is likely to happen in the bilateral talks, but if

“everybody was where you were, the way they would see this issue right now? I think that the moment”

this period of time is terrifying. Like, we should be terrified on a certain level. But like, you know, I've been working in AI policy when we are in other since about 2017, and I've been

hearing these warnings since then, and I've always tried to maintain some type of like

neutrality on how real are these risks. I'm like, you know, these scientists say this, these scientists say that, I'm not the one to adjudicate this. I'm not going to be the one who solves it, so I'm just going to try to sort of keep both these things in mind and work forward from there. But it, you know, the evidence is mounting. The evidence is growing that the people have been making some of the most dire warnings for the longest time that they have probably been right, at least about a lot of things,

and the warnings that they're issuing are increasingly dire and increasingly on short timelines. For someone who's been looking at this for a while and is tried to maintain a position of like not panic and neutrality, it's, it's very, it's very worth it. I think that's the place to end. I'll also find a question. What if three books you'd recommend to the audience? I'll do two

China books and one fun one. So the first China book is Country Driving by Peter Hessler,

New Yorker Correspondent, and a lot of way, for people of my generation who went over there and live there, he's kind of like the godfather. He's the guy who inspired me to become a journalist to,

of of Chinese society at the micro level that I think we're just, we're missing, we're missing that

“in so much of policy today. And I hope young people today will start going back over there and”

getting in the mix. We need that like textured understanding. So that's one. Another one a little bit

more obscure. It's called From the Soil, the Foundations of Chinese Society. It's a book by a Chinese

“sociologist in the '30s and '40s, got in facial tone, who was trained in the West, went back to”

China, applied kind of Western sociological paradigms to studying Chinese villages in agriculture.

And it's just one of the most insightful books about Chinese culture. So I'd encourage people to seek that one out. I read it every three or four years. And the last one just for fun,

“Sadie Smith's on Beauty. You know, you had Sadie on the show. I just, I think she's the goat. I think she's”

the best. And on Beauty is just a hilarious novel of an academic family and her ability to pierce into the psychology and the insecurities of each of us and and put that on blast in a way is just, I don't know, it just brings me a lot of joy. So on Beauty. That's you, thank you very much. Thanks for having me.