Lawfare Daily: AI Targeting Systems Are Coming—But Not as Fast as You Think

Running isn't enough anymore. With PWC's managed services, your operations don't just run, they evolve continuously,

powered by AI embedded directly into your workflows. So instead of maintaining yesterday's model, you're building tomorrow's advantage. PWC's managed services, we run your operations with tech and talent, so you can run faster, scale smarter, lead stronger. Yes, there are growing capabilities linked to AI that affect the battlefield and war,

and yes, we should be concerned about that, but we also shouldn't jump to conclusions without understanding

and a more granular detail what it takes to make these systems work in the way that we saw in the Iran War. It's the Law Fair podcast. I'm Kate Klanick, Senior Editor at Law Fair with Steve Feldstein, Senior Fellow at the Carnegie Endowment for International Peace.

“So I think we'll see a lot of push when it comes to AI integration on the edge, when it's on devices that are being used directly in the battlefield.”

But for a full-fledged, comprehensive system, like Maven, I think that's still going to remain in place for a small group of high-capacity military for the near future. Today we're talking about his recent article in the bulletin of the atomic scientists on AI targeting systems. So I thought that your take in this piece was super kind of contrarian and a little bit like unexpected, and therefore, as I'm kind of a little bit of a contrarian myself, valuable and kind of like, put up like sent out my spidey sense as it were.

So there's kind of an ambient panic in the room for people that are not familiar with this space, which is that military AI is extremely real, it's consequential, and it's genuine, but it's genuinely hard to build. It's a hard to get the components, it's hard to get the factories and the setups to make the components turn into usable products. And then, so like, let's just lay out the whole thesis before we, you know, you start pulling the threads, and that you do so well, and so quickly and clearly in this piece, which we'll link to in the show notes.

So the conventional wisdom right now is post around war post Ukraine, that like your title is doing a lot of work, AI targeting is coming, but not as fast as many assume, watching AI warfare proliferate in real time.

“So what is the story that everyone else is kind of like hung up on and grasping to and is like very much has been in the ether for almost as long as I can remember.”

What is that story getting wrong? Yeah, so, you know, I think that with the Iran war there's been so much attention paid to how the targeting and the kind of pace of strikes have occurred. And I think the question on everyone's mind as well is this now the future of war, so if you look to other conflicts, future conflicts, battles around, you know, is this how war will now be fought? Will we see something happen where every 70 seconds, another target is struck whether it's, you know, in theaters like India Pakistan or China, you know, in another theater or something, another conflict in the Middle East, is this the new way that things are going to look.

And so I wanted to kind of delve beneath that a little bit and sort of ask some questions, you know, in particular, how, like how do these all these different pieces work together?

“To what extent is this replicable? What do you need to actually make an AI targeting system work at scale as effectively as what the US showed in the Iran war?”

And so that's where I got some interesting insights as I kind of delve beneath the surface there and kind of looked at all the component pieces necessary to make the system operate in a fast, in a fast pace, in destructive manner. Yeah, and so you kind of open with this staggering data point that during the Iran war, which was 38 days, I think by the time when at the time of this piece published, that the Pentagon's use of Maven surge to something like what 20 billion tokens a day.

And it was a over 4,000 percent increase and 13,000 targets struck. And if the diffusion story is over hyped, how do you square that with numbers like these though? I mean, that's a huge amount.

That's a huge amount of kind of the reason that people I think see numbers like that and then say, oh my gosh, like this is an absolutely incredible amount of warfare. I don't know, is that should we contextualize that? Is the argument that the US is exceptional or that the numbers themselves overstate what AI is actually doing in the supply chain? Yeah, well, so I mean, I think the question for me really is, the US clearly has been able to strike a lot of things.

And the question for me is, can others do the same thing? Is this reputable? Are we going to see this happen with other militaries? And the answer for me is not really that this was unique over time, you know, as the technologies embedded in these systems become more ubiquitous. As they get lower in price. As we've seen, when it comes to other types of technologies elsewhere, sure, if they will spread, but they're not going to spread, it's not going to be like an issue of turning the switch, where one day the US has it and then in six months time Saudi Arabia, UAE and everyone else has it right away. It takes a lot of investment, both in terms of the data infrastructure needed training the algorithms appropriately,

“tying that together in a targeting system that is linked to a cohesive command and control structure.”

So in other words, you can't just simply say, "Oh, we're going to take this off the shelf, give it to X country, and now they have this same capability that the US has."

So we can walk through all the different levels beneath that, but the top line argument is essentially that, yes, they are growing capabilities linked to AI that affect the battlefield and war. And yes, we should be concerned about that, but we also shouldn't jump to conclusions without understanding and a more granular detail what it takes to make these systems work in the way that we saw in the Iran War. Can you quickly give me a description of just like a kind of a two or three sentence description of volunteers like Mavin, just for people to differentiate it between models like kind of clawed or things like that, like what Mavin is and like how they can do.

Yeah, so Paltiers Mavin is essentially a user interface software packaging that helps to bring together different streams of data to facilitate decision making.

So it's something you can think of it in a business perspective when it comes to allowing a person to move along decisions through clicks so that you can go from identifying a target to assembling a targeting passage towards a final decision. About whether to actually authorize a strike to destroy that particular asset. Perfect. What I thought was like so so useful in this piece is really kind of talking about what it takes to make an AI targeting system, which is something that I think most people don't understand and also kind of frames up like the anatomy as it were of like a kill chain.

“So if you could kind of walk us through what it actually takes to build these AI targeting systems because I think most listeners imagine that you kind of like if they if they're familiar with this problem.”

Most listeners are imagine you buy an AI model and you point it or train it on a satellite imagery right or like you describe something basically closer to a many decades long national infrastructure project.

And I thought that that was like it was just a really great way of really kind of building out the complexity and scale. So if you could just talk about that a little bit.

Yeah sure. No I mean and the first thing and a lot of part of this actually relates to my earlier work research have done when it comes to surveillance systems because there's a lot of similarities there and that's where I kind of got the idea to kind of think about how that relates from one to the other.

“But essentially what you need is you need to have an intelligent surveillance and reconnaissance capability ISR as it is in order to actually have data that's usable.”

So what do we mean by that? So it means especially when you're looking at dynamic targeting which is the idea that as you're in a war new targets pop up new information comes up that allows you then to position assets and munitions to strike those targets. So for example, let's say you're looking at a command and control cell or a several high. You know senior generals and you're looking to strike that that group well they're moving constantly and so as new information comes in that then gives you a better understanding of where they are where let's say a missile battery is and so it's not something that you can just sort of assume will be in the same place that it was maybe three months prior.

What do you mean what I mean by dynamic targeting so how do you how do you get the information well that requires inputting tons of data right so that means not only looking at cell imagery kind of in real time. It also means examining signals intelligence intercepting phone calls oftentimes social media telegram has proven to be a great trove and resource especially in the Ukraine war when it comes to identifying where different Russian military assets currently are located so you put that all all in.

In order to get that information that takes a pretty high technological capability it's not like you can just sort of make it up overnight. I mean you really need to have a pretty deep rooted surveillance apparatus and a more deep rooted the more accurate it is in terms of feeding into the algorithm.

You know it's not obviously it's not just satellite imagery you just kind of spell that out beautifully but like does that information is that including the vetting time you have to do with all of this information before you feed into the system where you feed it all in and like you're trusting the highest system to effectively sort it out and vet it for you.

“You know I honestly don't think there's that much vetting that occurs because and partially because we're looking at so many different points of data I mean essentially what we're talking about is raw intelligence coming in right so.”

If you're you know intercepting phone calls you know to what extent are you able to vet and say is that exactly the person we think I mean you have a reasonable certainty that that. These interceptions are correct and you put that all in and what you actually hope for is essentially you're putting a bunch of noise and then you're hoping the system will be able to kind of peer through the noise and derive insights but.

I mean the the reason you have a system the first place is that there's so much data coming in it's impossible for humans to sort through that I mean that was the original premise for what.

I mean I mean the talent here essentially came up within 2003 when it was started up they basically said you had all this data when it came to potential terrorist activity related to 911 you were unable to sift through that because of. Of just human capacity limitation let us provide software that can do that for you now. Does that software work accurately are there errors associated with it I mean that's a whole other question but the general idea is that there's way too much data. To little add it to capacity especially at the front end to do something with it which is why then you sort of feed it into the algorithm and then use try to then sift through it throughout that process to figure out what's actually legitimate and what is it.

Great so I'm sorry I cut you off but I just kind of want to loop back is there so you were kind of going to the supply chain right okay so you would say you have this like you know you want your building up this surveillance apparatus over time and and frankly the kind of more in depth and the better you're able to make it. You have it constructed the better ultimately the the target generations will be at the end that's the theory of the case so you so you there is in incentive to make sure the information coming in is good you don't want just.

Bad information but you also recognize that that you're bouncing with the fact that more information is generally good things so the more data that comes in hopefully the right information will sift through so that's one piece of it the second.

A piece of that isn't actually having a well trained algorithm that can identify particularly if you're trying to identify military objects or individuals something is able to kind of.

Better filter through what's real or you know color combinations are so forth when it comes to computer vision and what's not so an example is this idea that when the use military was developing Mavin they found that models that had 70% success rates in theaters like Afghanistan that are dusty desert like and so forth.

“plummeted to less than 30% when they applied it to theaters like the Philippines which are you know dense rain forest foliage right and and so you need to can context matters.”

And you need to have an algorithm that is properly trained and so to do that takes a lot of time it's again it's not impossible but each one of these steps requires.

Dedicated personnel feeding in labeled images making sure that you have an algorithm that is properly attuned to the conflict and terrain in which it's operating.

Yeah, I actually was going to bring up that incredible Mavin example because it reminded me very much of kind of the flaw like why don't know if you remember this from like your civics in American history but like the flaw of the revolutionary war which was like the British marching in with like their red coats and like becoming like literal targets versus like the like the gorilla revolutionaries obviously and rebels and I just like this was like it just was like wow so much has changed and then still so much has not changed that we have.

change that we have like there is just something so simple is like what are people wearing and and what context and then completely changes the game of war. Yeah, absolutely and look at you know enemies have an incentive to try to hide and camouflage as much as they can I mean at this point this has become kind of the key part of the battlefield in Ukraine where you have different ways that soldiers and who are in like small little encampments try to blend in so that they detect. You know identification from drones above and so forth so you know even a well-trained algorithm will struggle at times to find accurate imagery let alone one that isn't right for the context so that.

“You know that that's that's another input in the kind of targeting value chain that requires investment but then you know this sort of kind of infrastructure piece that I think is also really interesting is that.”

More data that you collect the more phone calls that you're storing and then trying to sift through the more you actually need the right physical infrastructure the compute infrastructure in which to then analyze this and this is you know an interesting example.

You know in just like a million phone calls a day or so forth intercepted from from Palestinians residing in West Bank in Gaza.

But it's soon ran out of space on his servers internally so I didn't have the data I'm just talking about one of the most sophisticated militaries and countries in the world. So what they had to do was turn to Microsoft as and some other companies but Microsoft in particular and basically say look you know let's come up with a contract so that unit a 200 you know the kind of Israelis equivalent to the NSA would actually be able to store this data and then process it on Microsoft as your servers. Based in the Netherlands in Ireland right so this shows you that I mean it's not again something where you can just like sort of come up with an off the shelf system.

“Have a bunch you know intercept a bunch of phone calls and be you have to store somewhere you have to do something with the unit the compute capacity and that cost a ton of money.”

I think project nimbus like this cloud computing contract that was signed between Google AWS and the IDF that is really defense forces in 2021 is a 1.2 billion dollar contract. It provides a suite of machine learning tools other sorts of storage and infrastructure capacity. But we're talking about really large expenditures that a lot of countries will struggle to match from a resource perspective.

Yeah so this is also something I love which is unfortunately like unfortunately it's always about logistics.

“You know one of the things that I think is really incredible is like just people just because they're just dealing with these systems in these models or these chat bots in front of their own computers.”

Just seems to slip running on your computer and they're like missing from this equation is just like the absolute the petabytes of just huge huge swaths of like every sources and data and energy infrastructures that are like invisibly falling into place. But like not so invisibly because we're seeing all of this controversy and data centers and everything else. Like I mean just it's it's really true it's like these are not just like to create this artificial intelligence to create this capacity to crunch this kind of data in the way that human beings would crunch it is like costing us billions and billions of dollars like we're creating artificial human intelligence sure.

And it's turns out it's very expensive like it is that's very very expensive to build an artificial human brain that is like as good as Steve Feldstein.

So like or take clonics right but no but I was exactly I was just going to make that exact analogy to the data centers is I think this is cognitive dissonance where on the one hand we don't want data centers where angry about them by justifiably we are worried about the environmental damage we're worried about the rise in electricity costs on the other hand like don't take away my chatbot.

“Right let me use that as much as I can but get keep those data centers out of my backyard I mean you know you this sort of one goes with the other but we forget that when it comes to these these complex systems.”

This is a story that like is very familiar to me I've studied speech platforms and private governance my whole life and I'll kind of get to that that was kind of where my question was going to go but I was just going to also say that is a very familiar question of people want to the good parts and want to take away the bad parts of technology. And they do not understand how difficult it is to extricate either through policy or even in practicality if you can write the right policy to like incentivize the correct practicality.

These systems to preserve the things that we like about these technologies without having these negative externalities if it's even possible at all most of life has negative externalities to. And these are the things of convenience and certain types of benefits that we create and so like I you know trying to hit that balance I mean like it you know took us like a hundred years with cars to make them safe. And things like that and so I just you know I kind of wonder about that all the time but anyways that's a different we can talk what the history of cars and a different podcast Steve.

You know reading your piece it really struck me particularly the private part of this right and so I talk about like kind of I've talked about the private governors of speech but here you kind of have a lot of the same players but not in their speech capacities just in their private company capacities Google Amazon Microsoft. Open AI Palantir and Thropic have become something more like kind of private governors of this kilching essentially right and Israel's targeting capacity is you're describing it in the piece and you just kind of described it now runs substantially on American commercial cloud and AI infrastructure.

So like who is going to be accountable in this arrangement who is the best target for accountability when you have something like this I mean is this what we're going to think of like the defense kind of supply chain in the future is it going to be not like who has access to like Patriot like kind of systems or whatever and to build actual like armature but like kind of like who has access to Microsoft Azure.

On the one hand you know it's not fair to sort of say Microsoft your infrastructure is responsible for you know largely or or wholly responsible for strikes that are being carried out by drones that are manufactured by domestic Israeli companies.

There are then used in in gossip targeted yet I mean there is a crucial aspect to the supply chain that Microsoft has provided and when called out upon it.

Reputationally by a guardian investigation they have pulled out from the contract so clearly they feel that it's not you know they they have some amount of culpability.

“Even if they are not sort of directly responsible for the end result of that so that becomes complicated I mean I think sort of as you get further down to the kill chain and as you get to the actual.”

Weapons there being you so you know the thing is you have all state of infrastructure is place you have all the surveillance apparatus that kind of comes in and the end of it you still need two things right you need to have a software interface or some way to make decisions so that. What information that's presented in the targeting packages that come together are then actually used for something to destroy something that's where bound to your maven comes in so you can make an argument that. That interface that puts it all together that compresses the kill chain that gives you four clicks towards destroying a target that's probably closer in line at least from a software perspective to the end product then.

The data infrastructure behind it and then from there you still need to hand over to a drone or a ballistic missile or you know a man to aircraft or something else it actually will deliver the munition that destroys the target whatever that target is and so I mean I kind of work backwards that way and that like you know you start with who actually fires the shot or directly authorizes the kind of final orders that kind of go into.

The end targeting but then you kind of work your way back and say well there is the interface and to what extent to that bias or lee towards decision making.

Erroneous or not that led to that target and then behind that you have data infrastructure and so forth so there's all these sort of different pieces to it. Do your current managed services really help run your operations or are they just running in circles running isn't enough anymore with PWC's managed services your operations don't just run. They evolve continuously powered by AI embedded directly into your workflows.

So instead of maintaining yesterday's model you're building tomorrow's advantage PWC's managed services we run your operations with tech and talent so you can run faster scale smarter lead stronger.

We have a pre-exampsi met erhöter proteinori motor mun by 15 cm CTG is granted. We have a halting design to em Hinterkopf. Push, the two types stuff of the medical drama series. Hitstream MCTF. So this is kind of a great point you give us this anatomy of the kill chain and the thing that I really kind of also want to focus on

Specifically when we're talking about kind of all of this kind of array of private actors that take place in this and that you just described.

“Is that I think it's contrary to the notion that when you talk about a chain there's kind of this idea well if you break one link.”

The whole thing is like a bit of like a it's a chain and so it's at every point in it is like if you can disrupt one part of it you can like have a out like a choke point. Essentially and I think the one of the things that is really great about what you're describing and what this piece describes is that there really isn't a chain. It's kind of a web or it's kind of a stack and the stack is diffuse and has many options on how it can go. But I kind of just wanted to see if you could add a little bit of color to that and if that's something you agree with.

“Yeah, no, I think you've actually described it really well. I mean, I think this idea that all of a sudden you can stop a kill chain or disrupt it through taking out one thing doesn't really work.”

I mean, kill chains just describes a process of getting from identifying something to destroying that thing and all the steps along the way in which to go from. And so, you know, kill chains existed before AI and this, you know, text acts and so forth. They were just more analog and they were more manual.

But at the end of the day, you're still going to go from A to B in either case and it really is something that you can't sort of stop.

“Also, I think it's important to note that there is an owner of the kill chain that's the Pentagon. You know, these different private entities and service providers give software.”

They give options. Someone's those options. There's a design element that's biased towards a certain outcome or another. But I don't think that means that you sort of take away, you know, legal or military accountability from the Pentagon itself. Most of me are the ones who have to make the clicks. They're the ones who can demand different pieces to how the process is set up.

And they're the ones who have to sort of make the final decision, say, this target should be destroyed. The evidence coming in points to ex conclusion or to say, you know what?

We need to take more time. I don't feel comfortable with this. The pace is too fast. The scale is too fast.

“We need to slow this down because I'm not confident that in the accuracy of what's being recommended to me right now.”

Yeah, I'm increasingly a fan of and I'm planning a podcaster in this about the idea of like deciding to opt out of certain types of technology and our human autonomy and ability to say, like, let's slow this down. Of course, there's not incentives as you say in the fog of war, but even in more in general to always do that, but it is something that's certainly possible and like policy as a has a role in shaping, which we'll get to in a second.

But I kind of want to talk really quickly before we kind of flip to the policy discussion of kind of like the diffusion like what's next, who's next, what actually spreads.

And so you kind of bring up the UAE and like kind of the idea that this is like the forward looking thing that we should kind of talk about. And you know, just to be clear, just to find a fusion in this context for people. Sometimes I try to imagine my parents listening to this podcast and like, there's so many things that I'd have to define, but just to find what you mean by diffusion in this context. Sorry, Mom and Dad. But diffusion, what I mean is sort of the spread, the emulation, the replication and then adaptation use of a particular technology.

It's the idea that one country has an innovation and others will get it. And the questions behind diffusion is how quickly and who can replicate it and what does it take to do that. People look at a classic example of nuclear weapons, right? That's, you know, an example of something that actually has diffuse quite slowly. It's a closed system. It's very hard to replicate takes a significant amount of capital investment in order to operate. And at the end of the day, the costs required are not commensurate with the benefit that most countries would get out of owning them. So it's something that doesn't diffuse quickly.

The converse of that is the spread of something like open a eyes, you know, like chatbot, like chat GPT. I mean, that shows you something that was only owned by a few software developers that was opened up to the public and now has taken off like wildfire in a less couple years. I mean, that's an example of rapid democratized diffusion across the globe. Yeah, and so if you take kind of like an internet policy or internet history background internet, you generally require a lot of infrastructure, but once it was kind of in place, there was a massive diffusion of software based products like a massive diffusion of access to certain types of products.

And then kind of similarly, I would say here kind of one of the things about diffusion with, you know, systems of war systems of defense are or are that like it requires again, like a huge infrastructure kind of buy in. But once you have it, there is like maybe it diffuses faster. How is this or isn't this different in your mind. Then something like, I think actually you're kind of your comparison between like a chat GPT diffusion and nuclear like capability diffusion is like absolutely perfect because we kind of have both in this setting.

Like you like just having the intelligence, just having the access to certain things like does not make this a better or easier way to let, you know, doesn't actually change the game. It's the marriage of that with the with the physical kind of kinetic capability of a drone or a strike system or something like that, which is a different thing. That's right. Yeah, and so what do you what are you seeing with like the diffusion kind of issues here and why is the UAE the example that you use?

“Yeah, so I mean, I think you've explained it exactly right in the sense that I mean, first of all, you need to have capacity, right?”

And so the first question I asked is, who has who doesn't have the systems yet, but has the necessary resources so they have the money to be able to acquire the necessary components for this, or who potentially has some elements already in place like a surveillance apparatus that they've built up.

They actually have invested over a period of many years, a pretty substantial surveillance infrastructure anyway. They are also, you know, have signed lots of deals when it comes to AI data centers and so forth, so a lot of data infrastructure elements are already in place for that. So it's very logical and then what we also know from the kind of weaponry standpoint is they've really are investing a lot in their this military conglomerate edge, so they are seeking to build homegrown drones to build up their own arms industry to a greater extent.

“These other pieces are there and, you know, but most importantly is the political incentive. They, you know, see a very volatile unpredictable world and they want to protect themselves, and so investing in a system like that is natural.”

Now, there are other, a few other countries like that as well. I mean, I would sort of think of them as sort of rising powers. You know, Saudi Arabia is in a similar situation.

You can plausibly argue that India may be Brazil, you know, our other countries that have significant capacity and that over time, you know, would be kind of next in line to want to take up and adopt these sort of systems. So that's kind of what I looked at as like a sort of next set of actors who potentially, you know, I would anticipate would take the systems up in the coming years. Yeah, I really wonder how much of that hinges on frankly, just money, like how much of that hinges on the wealth of the nation. You have Brazil, India, Pakistan, Singapore, Turkey, Saudi Arabia, that you kind of list in this, the list in UAE is to the, and the reason Steve is to those who are are listing and not watching the reason Steve was laughing and kind of giving his answer was like he was like describing the UAE who has like,

who has a neighbor that is like very like, and I was like, stroking my chin like this was a very hard question like who's incentivized to kind of do this.

“But I think that this is like, you know, this is a really great list. Who is the most incentivized and then who has the resources to actually build this?”

Because obviously, like, you know, I'm going to go out and eliminate say that Turkey is not quite as well resource as the UAE and like, you know, and it doesn't have as a minute set of threats. And so like, you know, we're going to see it, but maybe we won't see it first. So I think that the UAE is a well selected kind of, you know, as the dominos fall, who's going to have this diffusion.

But you draw kind of a sharp contrast. I kind of want to use this example. You draw sharp contrast with drones, which kind of diffused everywhere.

I mean, frankly, like from militias to like people in the park like spying on you is you're like lying on your picnic blanket type of thing. Drones are, and I know that we're not, I'm like, hopefully they're not armed those drones, whether you're in the hanging out in central park. But just generally, I kind of like the idea that like this, these, these, that let's something could like be so much in your personal space so quickly and so seamlessly is a huge, huge change with drones. And it diffused so fast. It just diffused so fast when I was in 2014 killer robots and killer drones was at the very forefront of like when I was starting my PhD.

And, you know, now it's just to give it like this is just how kind of, and that that point people are like, oh drones like, oh, here we are in like the future of sci-fi kind of, but no, it's now it's just like how war is done. And so is AI targeting can like categorically different or just earlier on the curve than drones right now like what's the load bearing bottleneck here is it data is it talent is it ISR compute.

“Well, like, what do you think actually holds like this back from diffusing?”

Yeah, that's, it's really interesting to kind of think back to the drone analogy. I mean, like yourself, you know, I sort of watched the evolution of drones from being these very exquisite 30 million dollar reaper drones. It's only the US and the handful of other countries we're using for like signatures strikes against a Taliban or in Pakistan against al-Qaeda or against al-Shabab and in Somalia. And then over time, but really accelerating, I think, between the Islamic State and investments by Turkey and to kind of low cost medium range drones their TB2s.

You've all of a sudden saw this like explosion and then with 22 with the Ukraine war and their innovations with FPVs, they're able to take something, you know, they're able to take something that was essentially consumer grade, you know, and make it mass produced and show that a very simple quadcopter that you could buy for $500 off Amazon, you could outfit it with a basic munition and create tremendous damage.

I mean, I think we will not necessarily for the kind of targeting system that I mentioned. I mean, I think that's going to be a ways away from just becoming ubiquitous around the world.

“But when I think you will see our like lots of different experimentation with like autonomous drones, other autonomous weapons. So things like the last mile where you have a chip embedded in the device.”

You've already seen this in Ukraine and where you lose contact with the operator and the drone is able to use computer vision to kind of lock on to particular targets and destroy them. The targets are actually the right targets or maybe they're not. I mean, this is the problem when you sort of like leave your hands and sort of say, well, it's up to it's up to the machine to make the the final decision. So I think we'll see a lot of push when it comes to AI integration on the edge when it's on devices that are being used directly in battlefield.

But for a full-fledged, comprehensive system like Maven, I think that's still going to remain in place for a small group of high capacity militarys for the near future. Yeah, and so this has been so so clearly like kind of laid out and I think is a is a very persuasive argument and so I like I kind of want to move to like the so what so like what do you think that we should do. If we decide that like this we have a little bit more time than maybe people are forecasting or people are doomsday predicting.

Your prescription kind of has three parts as I kind of like put it to get those the restrict cutting edge chips right curve kind of access to the most advanced models and limit kind of military specific software. Like Maven and so the chip piece we kind of know how to do put there are parts of it that we're doing already.

“Elysian theory but model access how does that work like how are we going I mean and this is something that I think that we've been we've talked about in and a lot of different rooms you and I Steve like that we've been in.”

And and we just kind of talked about in like the nature of what diffusion is like how things diffuse software is the hardest thing to stop kind of from diffusion.

I mean the internet make information wants to be free systems want to be free how do you like how do you basically do that.

Wades leak capabilities like get distilled into open models and then the API is a global product like you know. Is export controls for a model even like a coherent way of thinking about this I know that there there have been a lot of people and experts in this. In this field calling for export control on models and I think that that's just not. One of the most effective ways that we've controlled software in the past so I'm just kind of interested in your thoughts on that.

“Yeah yeah I look I mean I I think you laid out I mean that I think all the challenges right and I like I don't know that I can like.”

legitimately come out here and say look I have a plan for stopping the export software in area models because even though like as we're going around we can see that not happening everywhere like. I think the at least on software and modeling the cats out of the bat I don't really think there's much we can do on that front. I don't think that's a realistic choke point and I'm not going to pretend otherwise that like there's some plan we can do so I think that leaves other pressure points so you know.

Certainly one of the pressure points is I mean you can you can you can make an argument maybe that the very most advanced models and mythos of the world or fables or whatnot. Maybe that has a slightly extra capabilities and so as new frontier models come out we should that and and look for for ways to safeguard and think carefully about how that could be used but I think that that to me doesn't really solve the problem.

I mean the second part of the problem is is you know you try to restrict the infrastructure so it's the chips it's the data centers right it's it's the kind of surveillance apparatus that I mentioned.

I think that extent you can't stop it but you can slow it down and I think we've seen that happen successfully to some degree when it comes generally to surveillance so you know a bad dictator who doesn't have access to a lot of money. In a second or third you know third tier country wants to get this type of equipment so that's where you can export controls can make a difference so you won't stop major militaries. But you can stop second to third tier actors maybe from acquiring as easily as they would like these types of systems and I think I you know I also think I mentioned is normative pressure public pressure right now there's very little friction.

It sort of feels like it's kind of an open space there's few rules there's people the public still is confused exactly about how these systems work. The more you can start to create a political cost for countries that are thinking about getting them the more that it becomes a resource trade off question where. You know leaders say well I could spend 150 million investing in this system but I'm going to I'm going to pay a political price I'm not even sure if it's exactly going to accomplish my objectives maybe I'll hold off that I think and have a bit of an effect as well so I think normative pressure matters I think infrastructural pressure can play a role.

You I like your normative pressure take and I actually kind of paired it in my mind with what you just said which was kind of actually not just normative but diplomatic like a decision that has to kind of. Kind of happen across nations to see the tradeoffs as they're coming and that there could be infinite amounts of money and resources devoted to basically like you know not a cold war but a very very hot war depending on whether or not you're like near a day to center you know and the climate kind of impact the the trade off of like putting money into that and not into like improving people's lives healthcare systems food like medicine like all of these types of things that are very real problems and like.

“If you're everyone is scrambling I gave like a list of just like you gave a list of just seven or eight countries that are like in various ways of like would be.”

Gearing up to get these products and to build systems that that create these products and these weapons that those are you know there has to be some type of truth there has to be some type of agreement that we're not all going to be searching for this all the time and maybe that's kind of. high in the sky type of idea right now particularly with our current leadership but I I do think that that is something that has to happen and I kind of leads me to kind of the lawyers question in this you know as we talk about treaties we talk about kind of choices but this is like kind of in the international humanitarian law like kind of context.

It assumes that human judgment is a point of targeting right and we talked you know just spent the whole time just destroying this kind of idea of the chain of the kill chain.

It's a much more amorphous kind of system that doesn't have a clear choke point but one of the things a humanitarian law and a lot of people argue for generally is to build in that choke point by having a human in the loop this is like a phrase you'll hear over and over human in the loop and people generally think that this is like I would say like I will summarize that the arguments against that which are basically that if you have a human in the loop. You slow down all of the benefits that you get from a very fast very sophisticated AI system you get it backlog that that it's like it's not a it's not a one ray ratches not like you can have everything flowing and then like magically a human can like snatch the one bad thing out of like the ether and also as we know.

In like various types of experiments from make medical diagnosis to all kinds of pattern recognition actually now these machines these now these systems are better at pattern recognition and spotting errors than even humans do and humans often introduce errors to types to a lot of these systems so it's like do we want a human in the loop right or will it introduce it or to like these like sophisticated machines but to get back to the point of autonomy and choice.

International humanitarian law such that it is and such that it has kind of power to shape this space.

Do you think that you know when lavender generates kill lists almost automatically and when maven compresses like detection to strike like into minutes right it's just minutes is meaningful human control like even a real doctrine that we are going to be able to deploy is that something we.

“get choice as like politically legally that like we're going to necessitate that type of thing also who would want that job honestly.”

But like we're kind of like that we're going to necessitate that type of choke point to be introduced to the kill chain. Is it still a real doctor is it kind of just this legal fiction that's doing a lot of work that doesn't really kind of maintain.

What we want to accomplish in the first place like do we want these systems to be safer yes but like is a human in the loop going to do that.

“Right right well I mean there's actually two levels to I mean I think one one question is do we think that their ought to be an equivalent balance between accountability.”

Or is the new theory of war is that faster wins battles and therefore by sacrificing speed pace and scale you are essentially causing yourself to lose and this is the theory that a lot of people are putting out this is the theory behind why there's such a push between. For more powerful algorithms bigger systems powered by AI that can allow for a compression of identification to to target destruction right and the argument that you you'll hear is that yes you might get more accountability. But then the war will be lost by by the US now I think that's a little bit of a false choice but I don't think you can dismiss those those arguments either.

And and you know the question on the kind of human in the loop in meaningful oversight and some respects we're already answering that question and the answer is sort of like no where we don't really I mean meaningful human oversight is a starting to become something a little more distant. Certainly for them the most sophisticated military like I don't know if 70 second reviews count as meaningful human oversight when it comes to the targeting cycle that we've described with with maven like I'm not convinced of that and the idea that you would have a human who's collecting information sifting through it and looking at it.

Like that's not really how war is being fought and certainly not where we're we're moving towards so you know what I think we need to kind of keep a hold up is accountability there needs to be accountability period now is that some combination of human on the loop combined with machine generated recommendations that.

“still is able to bring about accountability for violations of IHL that occur like we've got to figure out a way to get there and I think right now honestly we're we're behind when it comes to figuring out how these.”

How IHL applies with the new technologies on the field that are being deployed I think there's a big gap in that and I don't have a good answer to that but I think we need to think carefully about it and take it seriously because it's only proliferating in the in the months and years ahead.

So last question for your Steve is that hopefully in the near future we'll get to talk to you again because you have a book coming out.

Tell us what it's called tell us what is you know when it's due and like tell us kind of what it's about is it dealing more with this kind of idea. Is it going to expand on kind of some of the really interesting points that you make in this article. Yeah, thanks Kate so I do have a new book it's called bison bullets colon global rivalries big tech and then new shape of modern war. It's coming out next week actually in the UK and it's coming out in September in the US so in the book really kind of looks broadly at the way in which tech is being used to exercise power both geopolitically and war so it looks at different levels of tech competition in rivalry.

Between the US China and Russia on the one hand it talks a lot about how private companies are playing a bigger role as we've discussed here in terms of.

In terms of increasingly making decisions about how price will be used and who gets access to them and then it also talks about this idea of diffusion or spread how do new actors, you know, in searching groups and others get access to disruptive technologies that help them increase influence and attain political objective. So it tries to sort of present a big exposition of what the landscape looks like and where we're headed. So I'd love to talk about it when it's out. Yeah and we'd love to have you thank you so much for coming on Steve and I'm looking forward to talking to you about the book in a couple of months and thanks for coming.

Yeah thanks for having me Kay I appreciate it.

“The law fair podcast is produced by the law fair institute if you want to support the show and listen at free.”

You can become a law fair material supporter at law fairmedia.org/support. Supporters also get access to special events and other bonus content we don't share anywhere else. If you enjoyed the podcast please rate and review us wherever you listen it really does help and be sure to check out our other shows. Scaling laws, rational security, allies, the aftermath and escalation are latest law fair presents podcast series about the war in Ukraine. You can also find all of our written work at law fairmedia.org.

The podcast is edited by Jim Potia with audio engineering by Goat rodeo.

Our theme song is from Alibi Music and as always, thanks for listening.

“Do your current managed services really help run your operations or are they just running in circles?”

Running isn't enough anymore. With PWC's managed services your operations don't just run, they evolve continuously, powered by AI embedded directly into your workflows. So instead of maintaining yesterday's model you're building tomorrow's advantage. PWC's managed services, we run your operations with tech and talent so you can run faster, scale smarter, lead stronger.