Lawfare Daily: ‘The Reverse Centaur's Guide to Life After AI’—A Conversation with Cory Doctorow

Running isn't enough anymore. With PWC's managed services, your operations don't just run, they evolve continuously,

powered by AI embedded directly into your workflows. So instead of maintaining yesterday's model, you're building tomorrow's advantage. PWC's managed services, we run your operations with tech and talent, so you can run faster, scale smarter, lead stronger. Booking from com is the easiest way for me to lose. To an influx dot in, body-side, kind, role-spieled.

Uba Alhin, ganz einfach. Booking.com. Booking. Yeah.

“Under conditions of patriarchy, you can be certain that you can sell anti-aging cream, even if you don't think anti-aging cream works, right?”

And under conditions of bosses having these confrontations with their workers and under the conditions of the class for, you don't have to think that the software can replace a worker to think that you can convince a boss to replace the worker with the software. It's the Laugh Fair podcast. I'm Kate Klonex, senior editor of Laugh Fair, here with my co-host Laugh Fair, senior editor, Alan Rosenstein. Our guest today is Cory Doctor, the science fiction author, activist, and journalist. The fact that we are finding these information security defects in our software using AI to whatever extent we are is good because we need to find those defects and fix them.

If we can clean that up with AI that's great, but if what we do is we get the AI to write the software, it's going to be full of those defects. The fact that AI can find the software defects doesn't mean that it won't produce them at scale. Today we're talking about his new book, the reverse centros guide to life after AI.

“So I guess foundation question for those uninitiated and you're so good at the neologisms, but can you give us the metaphor?”

I'm sure this is like you've done this in a million times, but can you give us the metaphor for the reverse centor?

Sure. So you know, this is not the first time we've had to ask ourselves how labor and automation go together. So there's pretty rich literature here, and one of its trueisms is that when workers adopt automation, it's typically in service to increasing quality, whereas when capital adopts automation, it's in service to increasing throughput, because when you acquire an asset, you want to get as much value out of that asset as you can before to appreciate software books. And so typically they work the machines, which means they work the people who work the machines as hard as they can.

And so that's where we get centours and reverse centours in automation theory, a centoris someone who's assisted by a machine.

“And you're out of bicycle, you actually look like a centor, but you know, using a spell checker makes you a centor as well.”

The point is that you are the human brain directing a machine and the machine has capacities that you lack as a person. It might be stronger than you were faster than you, able to pay my new detention without a break in a way that you couldn't ever do. It can certainly work longer than you can, but it is taking directions from you. You're, you're the brain. It's the, it's the tool. The reverse centor is one in which you are the tool of the machine. The machine is the brain atop your frail human body. It is directing your body's actions in order to achieve parts of the task that the machine can to achieve on its own.

So it's using you, but it's also using you up because you are always going to be slower or weaker or have less stamina than a machine,

which means that when the machine sets the pace, it is setting a pace that is always going to be at the outer limit of your capacity because you are the bottleneck. And so this is why, for example, the most automated warehouses in America, which are Amazon warehouses, have three times the injury rate of other warehouses because when you are the slowest factor in a, in a facility with a seven figure automation investment that you want to maximize the return on, that automation is going to set a pace for you that is at your absolute limit, and it's going to do it over and over again until you tap out.

And if you get that wrong, you're going to get impaled by a foreclift. And so AI for reasons that I hope we'll get into is not intrinsically organized around reverse centorship, but the way that the capital markets have funded AI and, you know, to a very extraordinary degree set of trillion, it's 1.4 trillion so far in counting, that necessitates a degree of reverse centorship. So Corey, so walk us through how you view then the AI market, because I think, you know, if your book was just about AI is going to have these distributional effects,

But I think what's really provocative about your book is that you're arguing that this explains the structure of the AI market and its fragility. So just given overview of that.

“Well, there is a seeming paradox. There are a couple at the heart of AI. One is a labor paradox, which is that you meet people who are skilled workers,”

historically reliable narrators of their own experience, and they tell you I use AI, and it makes everything I make better. There's lots of programmers who fit that bill, right? And then you meet skilled workers, also reliable narrators, who say I'm being me to use AI in the workplace,

and the products we're making are so defective, you know, I'm an aviation software developer, never get on an airplane again, you know,

and how do you resolve that conundrum? Well, you ask yourself whether that first group are centours in the second group is reverse centours, and then the other paradox here is the paradox of the monopoly, because the markets love a firm that is growing, and the trajectory of a firm that's growing and growing growing is that it's going to saturate its market. And the problem is that when it saturates its market has to stop growing, right? When you have a 90% market share, you don't grow again, you know, you can try breeding a billion humans and raising them to be your customers.

Google has a product called Google Classroom, it's very promising, but it takes 15 years to pay off. And in the meantime, the markets want to seek growth now, and as the director of such a firm, as key employees or managers of such a firm, you also want that growth and not for ideological reasons, not because growth is the ideology of a tumor, but because firms that are growing are worth more than firms that aren't, because a share in a firm is a claim on its future earnings and the future earnings of a growing

firm are larger than the future earnings of a static firm, and when you are highly valued, when your shares have a very high ratio relative to your earnings, when they are very liquid,

“you can use them like cash. So if you want to grow about hiring key personnel, you can give them shares.”

If you want to grow by buying other firms, you can buy those firms with shares, and the cool thing about shares instead of cash is that shares are an endogenous substance produced within the four walls of the firm. You make them by typing zeros into a spreadsheet. You are not allowed to make money within the four walls of your firm. The Treasury Department really frowns on it, and so if you are bidding against a boring old mature company that

has to get money from creditors or customers or from investors, they're always going to be

out bid by you typing zeros into your spreadsheet, and the corollary is that when you stop growing, you become grossly overvalued. Your share takes a huge nose dive. We see this over and over again, whenever tech companies have a small bubble and their growth trajectory and we get these panic sell-offs, because everyone is worried that this is the beginning at the end of the growth.

“And when you're no longer liquid, well, first of all, all the top executives who have been”

compensated in shares are now worth half as much or a quarter as much as they were yesterday. And second of all, you can't buy other companies and restart the growth by using shares to do it. So there is a very good material reason to want to grow. And so we have lived through all of these tech bubbles that have coincided with the saturation markets by monopolies, where they have come up with ever more outlandish narrative strategies to describe imaginary markets they can go on conquer,

because claim you're going to conquer existing markets has the drawback of people who currently run those markets insisting that you can't. When Google says Google Plus will make us Facebook, Facebook has lots of reasons why Google Plus will make them Facebook. When you say instead, I'm going to conquer the Metaverse, like who is to say what the Metaverse is worth? Well, on the one hand, that might people make people skeptical that it's worth anything,

but on the other hand, as the person who just thought it up, you're the world expert on how big the opportunity is. And so in the manner of a man running across the river on the backs of alligators without losing a leg, so long as you can come up with a new narrative, as soon as the old one gets a little washed and stale, you can grow. And now we get to AI. We go, you know, Metaverse NFTs, cryptocurrency, Web3, Dows, and so on. And now we land on AI and AI is bigger, a lot bigger.

Meta spent $61 billion in the Metaverse. They spent in the first three years, $150 billion on AI.

In this year, another $150 billion is doubled, and they say it's going to grow again next year. And so that kind of expenditure is clearly different. It's serving the same purpose, but the capital markets are willing to fund it to a much larger degree. And so that brings us to the ideological part of this, which is that what they're promising the capital markets is that you can fire workers and replace them with AI. And even if you don't believe that the

AI is good enough to do that job, you might believe that bosses are so infinitely horny for replacing workers with software that never melts off to them and tells them why they're brilliant

have to have ego-shattering confrontations with people who know how to do things, and clearly

“think that you're an idiot. You don't have to confront the possibility that you're not in the”

driver's seat, that you're in the back seat with a toy steering wheel. When you know that that's the sales call you're funding, you don't have to believe in the product. You might believe in the product, but you don't have to, because you know that you can sell a hell of a lot of that product. Like, under conditions of patriarchy, you can be certain that you can sell anti-aging cream, even if you don't think anti-aging cream works, right? And under conditions of bosses

having these confrontations with their workers and under the conditions of the class war, you don't have to think that the software can replace a worker to think that you can convince a boss to replace the worker with the software. Okay, that's like 73 ideas, which is awesome. So let's see how many of these we can get into. Sure. So the first one, and I just want to clarify something that was somewhat unclear to me from the book, which is when you invoke the

idea of a bubble and you invoke past tech bubbles to understand AI, what do you mean? Because it's strikes me that there are two different kinds of bubbles fundamentally. There are bubbles where the product

“itself is just not very useful. Like, there's just not a lot of there there. And so I think crypto”

is a good example, web three, the metaverse. These are all examples of Flon, and Ron, NFTs, right? Like, they're just not a lot there, right? So you just pure value destruction. On the other hand, there are bubbles like the original.com bubble, or the fiber optic bubble, or, you know, there have been bubbles in semiconductors, right? Where what you have is a temporary, since over investment and over capacity, that pops, it's a big problem, right? That can have long-term effects,

right? The housing bubble is actually another example of this, right? Housing prices are much higher, right now than they were even during the housing bubble. It seems to me that AI, if it is a bubble, and that seems quite plausible, is a bubble of the second, not the first, right? Like, there's an actual product there, and Thropic does, in fact, make a lot of money, right? Maybe we're over investing, maybe meta's bets on it will go to hell, but it strikes me that AI is going to

endure. And so I'm curious, if you agree with that, and be, if it's a bubble of the second kind, not the first kind, what implication does that have on your broader account? Because that it seems to me like the NFT and the metaverse and stuff, there are very useful to understand AI. So, it's crazy about that. So, I think that you're describing a spectrum, and I want to have a two-by-two grid. So, there is the spectrum, right? Does it do anything good? Doesn't it do anything

good? Right? And then there's the other piece of it, which is like, can you ever have a successful business given the way that this is currently constructed? And if not, we'll there be something left over when the bubble pops that you can build a successful business out of, right? Is there a

productive residue? So, WorldCom was a fraud. The WorldCom could never be profitable. WorldCom's

stole money to put fiber in the ground, as a convencer. When WorldCom was gone, the fiber was still there. My connection at home in Los Angeles is WorldCom fiber operated by AT&T to give it

“two gigabits, a magical link, and that's how we got the fiber, right? And so, that productive”

residue is a thing that will have after the AI bubble goes. I think we are going to have, you know, GPUs and data centers at 10 cents of the dollar. We're going to have an army of skill technical workers who can hire each other to build the products. Their bosses insisted, no one wanted, incorrectly, which is very like web two vibes. And we're also going to have a different kind of model. We're going to have these open source models that have released as convencers by the AI

companies, and that aren't frontier models. The frontier models are, I think, they have

unit economics that can never be improved. You know, the unit economics of the early web was

such that every time a new user signed up for the web, the web got less unprofitable and then eventually more profitable. Every new use of the web made the web more profitable and less unprofitable. Every new generation of the web was more profitable. That's not true of AI. AI is the reverse. Every customer that you acquire for your AI company makes you poor. Every time that customer touches their keyboard and interacts with your product, you get poorer still, your best customers

of the ones bankrupting you fastest. And every generation of the technology is more money losing than the generation before. They have $50 billion in global revenue on $1.4 trillion in capex, 700 billion of it in the last year. The assets depreciate out every two to three years and they really depreciate out. So like in order to make it worthwhile for people to buy all new GPUs from Nvidia every year, Nvidia has thrown out any guardrails on things that you can and can't change

with the GPUs, which means that new generations of GPUs have radically different power consumption,

heat dissipation and networking requirements, which means that it's not always possible to retrofitted

Right? So like you cannot turn $50 billion in annual revenue into profit if you're spending

$700 billion a year and you've got to do it ever and you've got to start over every three years in your capital expenditure. That's a very different proposition to the one we had with the web. So is AI useful? Yeah. Like it would be very weird to be opposed to statistical inference and it would be very weird to look at the returns to scale that we got from applying this minor variation to existing machine learning techniques 10 years ago and say that's not interesting

as a computer science banner. It totally is. It's fascinating. It's interesting. We're finding out that a lot of things that we thought you would have to understand in order to make inferences about them, we can actually just do through theory, free statistical inference where you don't understand

“it at all. You just say, I think the future will be like the past, the word that comes after”

this one is almost certainly this other word based on a large enough corpus. That's all super interesting and it's telling us stuff about like even the nature of reality that we didn't know before, but it's not a business and it's it's so money losing that even if like the pentagon becomes dependent on it and they decide to bail these companies out when the bubble bursts. They're going to have to do it again the next year and the next year. It's like building an entire

other DOD, right? Like building the world's most efficient money for us is a very weird piece of

performance are, but it's not a business. Yeah, I wanted to kind of get into that for a second,

which is, and I think Alan has a follow up again, but I'm going to kind of, I'm really curious about how this bubble bursts, right? Because there's like the clean bubble bursts, right? And there is this idea that like just like the early web, like it will just kind of just pop and it

“will burst very cleanly. And then there's the, I think the alternative that you're spelling out”

and for the people that I know and I talk to that say that that AI is not a traditional bubble, but are the people who say that it will not burst cleanly specifically for the reasons that you describe, specifically because exactly because it's fused with the state precisely. And so AI related investment is like now, I don't know, 30% of the S&P 500. 35? 35? That's seven companies.

Really. Okay. So that's like, I mean, so are you going to make the whole thing

systemically embedded, right? Too big to fail is like really literally the definition of too big to fail. So you're not going to get a 2000 style crash. You're going to get a slow kind of deflation on this kind of stuff. And a 20% to 30% condition rather than like a 70% like just collapse popped up by the Fed and investors and all of these other things. And so like you kind of just mentioned this that the state isn't better, that there's all of these things like is that?

Like I want to know what it looks like. Like what is it going to look like to us? Is it going to

“see it to kind of follow onto Alan's questions? Is it going to seem like the housing, the crisis in 2008?”

Is it going to seem like that? I mean, I think we've established that it's maybe not the same as the 2000s.com burst. And I certainly don't think that it's Enron style fraud, although there are certainly going to be precise companies and firms that are certainly running that griffed. I'm just kind of curious what your thoughts are on that. Well, you know, I don't think that there's ever been a bubble burst where the ambition of policy makers and market overseers was any different from the

from the ambition you just described a slow deflation. No one has ever like, let's just rip the band aid off this dot com thing. Right? The dot com bubble bursting was in spite of the best efforts that, you know, financial regulators and policy makers could engage in to prevent that from happening. And it still happened. We might not have a choice about what kind of burst we get. Yeah, it's totally true. Yeah. I'm guided by two sort of pulls of financial wisdom. So

one is, I think, called Stein's Law, which is that anything that can't go on forever eventually stops. And the other one is John Kenneth Galbriss, famous statement that the market can remain irrational longer than you can remain solvent. So, you know, I don't, I, I'm not like trying to time the, the rupture here. Like, if you want stock advice, I don't have it except maybe like go long on laser tag because that's something we could use data centers for. But oh, it's like what we

can do when none of us have jobs anymore. So that's a separate, see, I don't think it can, so I don't think it can do your job. I think a salesman can convince your boss to fire you replace it with a thing that can't do your job. We, we've seen that before. Like, if there's a job that you don't care if it's done well, but you feel required to do it. And especially if you're in a concentrated market where if everyone decides they're just going to do the job badly, consumers just have to eat

it. Like, that's sending customer service to the Pacific Rim. Right. We did that 10, 15 years ago.

people in the, in the Pacific Rim. And then we gave them a three-ring vider with a script. They

weren't allowed to deviate from that they weren't allowed to solve any of your problems. So that they could just absorb the abuse. Right. They could be, no, no. It's like literally like half of what it might like my very research has been on for the last 10 years. It's like insane. I forget her name. The woman who came up with the term moral crumple zone, but you know, they're the moral crumple zone. Yeah. Madeline the less. That's it. You're right. You're right.

You know, that in Dan Davies idea of a, of a accountability sink. So, you know, firing those people and replacing the chat box that can't solve your problem makes a lot of sense. Right. So anyway, back to this, back to the finance set. So you're right that it's not a pure stock's window, but there's a lot of stocks windling there. So as it runs done a lot of work in the

last couple of weeks on these leaked financials from OpenAI, one of the claims of the industry about

how it's going to sustain itself into the long term and improve its unit economics is with making inference cheaper. So inferences the, when it, when it makes a guess about how to answer your question. And so that's the operating expense. So you have these upfront cost training the model, but if you can make the operating expense lower, then you maybe you don't have to train the model again, because you've got a chap out that's good enough. And then you can find you can

tinker with the pricing model until you get to the right number. So, Ed looked at this and he's like, but wait, they've got this marketing budget that doesn't make any sense at all. Like if this is their marketing budget, they're spending more on marketing than like three Coca-Cola's. And there would just be agencies with like tens of thousands of people working on that marketing, if that's what they're spending and they're not. And so Ed's theory, which is the best theory I've heard

so far, is that what they're doing is calling the subsidized inference marketing. They're selling

“$100 bills at a bill of $1 a piece and they're calling it marketing not inference. And so that's how”

their inference costs are low. Is this just in a different line on the, on the balance sheet. And even if you don't agree that that's the answer, what we have seen with Claude and this is something, I think, Dan Davies wrote about it. What we've seen with Claude is that because the switching costs are low for AI, as soon as a model is significantly better, everyone leaves the old model and goes to the new one. This is how anthropic came out ahead of opening AI,

which means that opening. Right. Right. I didn't see that. Yeah, no, I didn't see it coming on, which means that opening AI can't say, oh, the chap out's good enough. We don't have to do another round of brutally expensive training because so long as there's a competitor that does. So there's a, there's an endless, bigger neighbor strategy there. That's that is going to drive more more capital expenditures. So the bubble does have these like unsound financials. They're not exactly

unsound in the way that, you know, pets.com was or NRON was or, or Worldcom was, but they are unsound.

“And I think you're right that they have fused with the state and the state will want to bail them out.”

Right. If we fire workers who do jobs that matter and replace them with chapods that are bad at that job, that's bad. But then if the servers that chapods run on get turned off and can't be, you know, turn back on again, that's worse. Right. That's like filling the walls with this bestest and, you know, we're going to be digging it out for generations, right, retraining workers to do those jobs and putting them together so they can acquire the process knowledge to do the

jobs as an ensemble and so on. So I think the state will be incentivized to do it. I just don't know if it can. Like, you know, obviously the US can make as many US dollars as it needs. But we're talking about claims on real resources. Energy and so on. And I just don't know that they're going to be able to fulfill those claims over the long term. So I'm not ruling out the possibility of a catastrophic collapse. And those seven companies that are 35% of the S&P 500. So six of them

lose money. The seventh one is as in video there. That's the one they're losing the money to. So when they go away, so does Nvidia, they're all trading around the same 100 billion dollar IOU really fast and pretending it's in all of their bank accounts at once. And they're, they're death financing and some of their capital financing is coming from entities who are really in precarious situations, like Gulf states that suddenly need to use their capital to rebuild their LNG terminals because

for some reason that no one can figure out a situation in which a ran was never bombing them

changed and now a ran bomb the shit out of them. And so maybe those people decide that their

“contractual obligations to build data centers aren't as important as having an LNG terminals so”

that they will have money tomorrow. And so they walk away from those financing deals. So there's so many contingencies that could just blow this up. And you know, I don't know if you've been following the part of the Epstein files where it looks like maybe he's the one who forced Layman Brothers into bankruptcy, right? Like we don't know where all of their weaknesses are, but we know that there are so many low-bearing Jenga blocks that are made out of Hulva and you know, riddled with ants.

And so the idea that we'll like take apart the Jenga tower really gracefully relies on none of those crumbling while we're doing it. I want to turn to the capabilities

You said, you don't think I will take your job. And I think it's actually worth digging into that

“because my sense is that a lot of the skepticism that you have of whether it's the kind of AI”

boosters on the one hand or the AI doomers and like the safety people on their hand is just you are much more skeptical of the capabilities of the technology. And so I think it's especially useful because you are not yourself an AI denialist, right? No. So why are you skeptical that in the, I don't know, let's call it medium-term, let's call it three to five years, right? Right, just to say long enough that new things could happen, but not so long that who the hell knows.

AI could get to the point where it really could meaningfully automate large swaths of white-collar work that would have real profound labor impacts that are worth taking seriously. Well, you know, the basis for that claim about labor displacement is typically the accounts of those workers who say that AI is doing my job really, really well. And those workers of the workers who are using AI as an adjunct to themselves with error driving it, which is to say where the

number of workers that remain static, right? The quality of the output goes up, but the number of

“workers you need to produce it doesn't go down. And I do think that that is disruptive an amazing”

thing and we'll see lots of cool stuff as a result. Although how much we get depends on whether those companies can continue to get normal people to give them the money to give away $100 bills at a dollar a piece. A lot of those claim benefits are related to that great subsidy. And you know, if you're allowed to lose thousands of dollars every time you touch your computer,

you can do amazing things with your computer in terms of the goods you can produce within

that cost envelope. And so those claims that that AI automation is very good, they relate to people who are not making productivity gains, they're making quality gains. And when you look at the inverse situation, right, where workers are being asked to mark the AI's homework. So the AI is doing the job of 10 coders. You fired those 10 coders. You kept one behind they mark the AI's homework. That person tells you that what we're doing is accumulating tech debt at scale, that they can't

mark the AI's homework at the pace that they're being asked to do it at that the arrangement in which they write the code in the AI gives them help with the boring parts of the parts that require perfect monotonic attention or whatever. That that's really good. But the bit where the code is produced at incredible volume and then humans review it just doesn't produce the robust code that we need. It just produces low quality goods at high volume. And are you are you confident

though that that is a question is always a timeframe. Is that a true for the next six months,

true for the next year, true for the next five years, true for the next 10 years? The same question about the productivity gains, right? Because you could say, well, maybe at some point AI will get good enough. And then inference costs will go down, right? I mean, we have seen like a thousand x decrease in the cost for the same level of AI capability. My question is just like, what set of facts, right? Would get you in like three or four years to change your mind and say,

oh, wow, like the lines, the lines crossed in a different way than I thought they would. So I,

“I think that I just don't believe that's statistical inference produces that like if you do”

enough of it, it produces the phase change where it becomes understanding. And I think understanding is why you need a human to mark the homework, right? I think the idea that understanding arises out of doing enough inference is like the idea that if you breed horses to run fast enough, eventually, one of the gives birth to a locomotive, right? I do think that like humans are brains are made of material things, which means that it is possible that at some point,

we will figure out how to make other material things that can think and reason and understand. I just don't think that it's going to be done by expanding the training data and the granularity with which you analyze it in order to make guesses. So I just didn't interview with someone who said, if you have two partners who've been in a marriage for a long time and they can finish each other sentences. It's not because they're guessing words. It's because they understand each other.

And that's true, right? It turns out that you can get a word guessing program to guess what word you're going to say next to. It doesn't mean it's understanding what you're saying, which means that when it makes that guess, that guess needs, if it's a consequential guess, if it's a salient guess, a human needs to review it. And I just don't think that you get the kinds of labor market changes we are betting we can get $1.4 trillion back from by having human

beings review those guesses. I just don't think the math pencils out. So, you know, if someone has a different technique in computer science that does understanding, right? Which, you know, I got

the reasoning models turned out what they were doing is using inference to guess how something that

“was reasoning would narrate its reasoning. But when you actually looked at its account of its reasoning,”

it wasn't how it was reasoning, right? It was just writing down shrewd guesses about what a chain of logic would look like. Meanwhile, burning a reptilian tokens and creating buildings for the people who were doing it, right? But it was a reasoning. Do your current managed services really help run your operations? Or are they just running in circles? Running isn't enough anymore. With PWC's managed services, your operations don't just run, they evolve continuously, powered by AI

embedded directly into your workflows. So instead of maintaining yesterday's model, you're building tomorrow's advantage. PWC's managed services, we run your operations with tech and talent, so you can run faster, scale smarter, lead stronger. No, no, no, no, no, no. You can now have a six hour conversation about this thread entirely, and both Alan and I are like, okay, this is like, let's push on this because this is throughout

all the other questions we had because so my background is in cognitive neuroscience before I became a lawyer. That is literally pattern recognition. Literally substantial similarity judgment. Literally all the quantum introversky stuff, the stuff that came after it, a lot of experimental philosophy, a lot of experimental psychology that kind of has been asking these questions for a long time. And like one of the things that for years before this AI stuff, I was doing AI stuff

specifically because it pushed on this question a pattern recognition and what we define is knowledge. It turns out like that's like what is like the cognitive science, the hand made in a philosophy years philosophy, the hand made in a cognitive science who knows, but like the general idea is like, I love this because this is the question that we're asking 10 years ago, 25 years ago,

like basically back to touring, right? Which is essentially this question of like, how does one

measure an external process? How does men measure without explainability? Without an ability to understand what it is as thinking or how it can explain itself, and it turns out that actually the stochastic parent actually, like the thing that's turning us all into bojack horsemen, walking around, like kind of like in its service, is essentially the same thing that like is we have no other way of defining it. And one of the things that I also think is really super interesting

and drives me a little bit insane is the, I don't want to disparage it, but the kind of the tech

“bro kind of's proclivity, or just I think the general human proclivity to describe these systems”

and these LLMs as human beings, as using words like thinking, using words like feeling, and if you look at Caitlin's words, being Richard Dawkins. Right, exactly. If you look at any of the work that's been done for the last 15 years on robots and all of these other things, things that are like decidedly not thinking, people will describe feeling and emotion and anthropomorphize the shit out of things that are not anthropomorphic, that are mildly anthropomorphic,

or dinosaur shaped, or whatever the heck, right? And so one of the things I think is really fascinating

and is like, is this just a philosophical question? Like, first of all, I don't know that this

is actually getting into the economics of this. I want to also segregate the fact that we're talking about economics, we've ended up in this place and so maybe we're in the more sociological, philosophical edge of this conversation, but humans are bad at explaining their own reasoning. They're really bad at it. They make judgments all the time. In fact, we don't even like, the worst kind of science is survey science, right? Like, where you ask people what they think,

and like the best type is the stuff that's involuntary, where you like, or like, you know, or you watch them from afar where they think they have privacy and they're silently making

“them their own choices, thinking that they're in their own bubble. But the thing that I think is”

super interesting here is that is human understanding, like different than pattern recognition, and if so, like, do we have to alter our understanding of AI, or do we have to alter our understanding of humanity? Well, you know, maybe we can talk about art here, because that's that's maybe a realm in which we see those questions raised. And I think answered, so there are things in the

and as an atheist, I don't think a sunset is saying anything to me or trying to say anything to me.

“I think that a sunset is just doing some stuff with physics. But when you take a picture of a sunset,”

or paint a picture of a sunset, and then you show it to me, even if I don't know you, my experience of that painting is on the one hand, the faithfulness with which you have captured, the physical phenomenon of the sunset. But the thing that makes it art is that we have this involuntary step that we take when we see a thing that someone is intentionally made or framed and shown to a second person to you, we try to understand what they meant. Right? So I don't know that we ever do. I mean, look,

I'm a novelist. My job is to pretend that one person has ever had access to another person's

interiority, a thing that has never happened in the history of the human race. But that is an

exercise that we find endlessly fascinating, that produces an aesthetic effect that is very profound, that can change the way you understand yourself in the world. It seems critical to like us being fulfilled people that we both make and experience this art in this way. And when we first encounter AI art, art, we encounter for the first time a painting that doesn't have a painter, words that don't have a speaker. And because we have never seen that before, we impute, or to use

a very modern term, we hallucinate an intention behind it. And sure, the person who prompted it, right, you type three sentences, make me a beautiful painting. You are a world-renowned expert, make no mistakes, that you put some communicative intent into that prompt. But the amount of communicative intent that is present when you dilute three sentences across a million pixels, it's like it's like homeopathic, right? There's just like there's no there that there's nothing to say.

And so when we get over the fact that this has nothing to say, it is disenchanted, it just

“quickly becomes merely striking. And I think that is the effect of even the most aesthetically”

interesting AI art, whether that's words or pixels or moving images, is that the best it can

hope for a striking, but it's never, moving. It's never art the way art is because we know that

as much as the pixel gasser or the word gasser can assemble things that feel like they've, they've got an intender that this is in, this is the seeming of intent without an intender. So my, my daughter just finished her first year at Santa Cruz. And there's this thing called the mystery spot in Santa Cruz. Very famous everyone's seen the bumper sticker. And it's a place where water seems to flow uphill. And if there was another mystery spot where when you threw leaves in

the air, they fell down and made a sentence. We would all go and look at the sentences that the leaves made. But after a while would be like, these just aren't that interesting sentences. Like as a matter of physics, I want to know what's going on with the leaves. As a matter of literature, the leaves just stop being interesting. They were grasped to the mean really fast in terms of their, their artistic impact on us, the way that they, they stirred our thoughts the way art does.

And so I think that buried in there is a lesson about why the job that the AI does is not, the job that the person does. And I'll give you one more example from law school just because I

“really, I just remember that I'm talking to a law professor here. I've a friend who's a law professor”

before LLM came along. Their grad students would not ask them for letters of reference unless they were good because it was pain in the ass to write them. But once everyone knew that a letter of reference was three bullet points. And then it shits out five paragraphs of flored nonsense. Everyone started demanding that they write letters of reference. And now whenever they advertise for a postdoc, they're inundated by candidates who have letters of reference from all their

props. So they shove them back in an LLM and say, give me three bullet points about this person. And we say that and we just like put our hands in our hands, our head in our hands because we know the bullet points are the same bullet points because the AI doesn't know anything about the law student. Right? All it can produce is the seeming of intention without the intent. And so in the same way, the three bullet points you put into the, make me a picture of a sunset you are a rower and

an expert, make no mistakes prompt. It knows nothing about why you wanted to make the, the sunset. And so there is no, there's no substance to it as striking as it might be. There is no substance to it. I just want to say that I, I totally, I totally agree with this. I love the art metaphor. And one of the things that I think is so true about this is the question of intentionality. I think that this is the one and more of this part of all of this, meaning

and you know, finally enough like you make the sunset comparison and you make like the

But I do think that like, I, I just want to say that, you know, we could talk about this forever.

“But I, I love this idea. And I think that I think that that is like kind of the right answer to this”

question. That there is a missing piece of intentionality. And it also relates back to my initial constant kind of like push back on the, on the, on the AI domerism and like the, the huge worry about kind of like this idea that they're going to just decide we're going to link them up to like our, our, our, our drone systems and they're going to decide to bomb us type of family. They just don't have intentionality like they can approximate everything. But they are just not actually

intentional so that like that would not happen. And then you have a little bit of something like open claw and it like makes me ask questions like the open claw debacle. But I mean, we, I want to leave that there and let Alan answer questions one to follow up. Well, so let me, let me, let me be the, the grumpy spoiler in, in the room then since both of you agree on the intentionality because I want to push on that for a second because it seems like there are two different versions

of this argument. And I just want to understand which one, which one you're making, Corey. So one version of the argument is to say, look, there is something inherently limited about AI, LLM,

transformer architecture, the data, whatever the case is such that they will never be able to

demonstrate a particular level of capability, right? Like they will never create art, quote unquote, whatever, that in a double blind test, right, will be as rich and meaningful as whatever,

“you know, Shakespeare, Milton, you know, whatever this Taylor Swift, whatever you want to do, right?”

Okay, that's an interesting empirical capabilities question, right? And that's one version of the argument. A second version of the argument is, no, no, no, no, there is just something about human consciousness, human sentience, human moral agency, mind-body, dualism, you know, whatever the case is, right, that will make us as humans never accept AI output as equivalent. And in that's why no matter as no matter how good AI is yet, we will always demand, right, that there be a

conscious human on the other end of the line, and that will, and that demand will itself be a kind of limit on, let's say, for example, the ability of AI is to replace all labor. So one might say, one is intentionality as like instrumentally important because it leads to good outputs, but that's an empirical question. And the other is as intentionality being kind of inherently important, which fine, that's a different question. So which version of the argument because, like, they're

very, very different arguments. I don't think I'm making precisely either of those. I'm definitely

“not making the second one because I'm not a mind-body dualist. I'm a materialist. I think our”

brains are made of atoms. I think we might make another brain out of atoms, and it might understand things. What I'm saying is that there is a thing we call understanding that is not statistical inference, but is instead a synthetic process that does not arise out of statistical inference, but out of some other physical process. And, you know, when we look back through the history of AI from the meeting and the fifties where they coined the term to now, what you see is a series of

technique breakthroughs that have been philosophically interesting because they made us question, if you can accomplish task A using technique X, and you can also accomplish it using technique Y, are X and Y the same thing. Right? If you can play chess with your human brain and you can play chess with an algorithm, is your human brain using the algorithm. And I don't think it is,

and I think the answer is it's not. Right? That what we're doing is we're finding two different ways

to do something. And there are limits to how well it works. And those limits arise out of the discontinuity between technique X and technique Y. They're not the same thing. And so they they don't perform the same way. They can do some of the same things but not all of them. And I think that this thing we call reasoning or understanding, which includes synthesis and coping with novel situations that are discontinuous with what we might call our own training

data, our personal experience. Right? That capacity is a thing that is foundationally at odds with the extremely impressive results that we can get out of statistical inference in which we assume the future will be like the past. And as a result, you get these hallucinations, right? These mistakes that these computers make. And that, you know, further I would say that in terms back to the labor market story, that because these hallucinations are mistaking the future for

like the past, that they tend to be the kinds of errors that are really hard to spot. And so you need humans to pay really close attention to that output. You know, I use the example in the book which is then called Slop squatting. So when your programmer, you include certain functions,

And so there might be like a bunch of libraries for parsing different kinds of text files. Like, you know, lib.text.pdf, lib.text.html, lib.text.markdown. And because reality is messy and has rough edges and because sometimes you merge two different groups of libraries into one. Maybe the one for like docax is lib.doticax.text instead of

lib.text.docax. And the AI will always guess, right? If it's just making inferences, that it's

going to have a regularity in the naming convention. And because we know the AI is going to make that wrong guess, it's going to make that misfire. And because we know the AI is producing code at scale and it's being compiled in malicious software authors can create a library with the right name, though name the AI is going to guess. They can include all the functions of the real library so that it passes all the unit tests and then they can add in malicious payloads. Right?

Now, as a programmer looking at that error, that error is as statistically indistinguishable from correct code and as it is possible to be. If the universe were regular, that would be correct code,

“which means that the kind of attention you need to pay to catch that kind of error, which is”

the errors that the AI makes most often. That kind of attention is very hard to muster. It's very hard to sustain. And it means that when you say to the AI, make as much code as you can and then have a person market, that that code is full of these hard to detect errors. And one of the really exciting stories about AI right now is when you ask AI to find errors, it finds a bunch of

security defects. We've heard a lot about this. We never know exactly to what extent the stories

are true because all the stories we're hearing, but the security defects are people who have secret software who are making claims about it. You know, oh, the five eyes say that their secret software no one's allowed to know about is had a bunch of bugs that were found by AI. Well, like one of the things we know is that if no one gets to look at your stuff and mark your homework that you cheat. And so we can't tell whether the five eyes have garbage software or whether the AI is doing a

really good job. Same with mythos where it's like, well, we're about to have an IPO and we've built a thing that's so powerful. You're not allowed to look at it. You're just going to have to take your word for it. That is like, that is a, that is like federal wallet inspector grade rhetoric about your, your looming IPO and people who are taken in by it at least without interrogating whether it's true have so little object permanence they would lose a game of peekaboo. And so, you know,

“I think that like the fact that we are finding these information security defects in our software”

using AI to whatever extent we are is good because we need to find those defects and fix them because our software fucking sucks. And if we can clean that up with AI, that's great. But if what we do is we get the AI to write the software, it's going to be full of those defects. The fact that AI can find the software defects doesn't mean that it won't produce them at scale. Yeah. So last big idea I want to kind of test on you and I think that we've, we've talked around this, but I think that

the idea of the entire book is kind of premised on this and what you just said is premised on this, which is that everyone talks about model collapse. The idea that AI is kind of trained on itself and it gets, and it eventually AI output is going to get lossy and in bread and what people kind of

call happed for AI. Thank you, Jthensekowski for your amazing coinage. I know it's really good.

So we're all going to be walking in with like giant, the kind of like fits really well with the reverse mentor if we all have giant chains like a horse or something, right? I mean it's like very, it's very good, but I've been describing like basically a human professionalized judgment version, which it turns out and repeaterscent had already kind of come up with, which is this idea of

“knowledge collapse, which is I think the exact idea that you're describing in reverse mentor,”

which is that weirdly, like as you kind of said, we have this domain knowledge, we can spot these errors, but what happens when we have the gradual narrowing of generations of humans that, you know, generally as we lean on these systems, we don't know what to do and look kind of like there's plenty of like data points on this, but just arbitrarily pick one, the MIT media lab recently, did a study that basically showed that like when people wrote with AI assistance for a few months, they just

like had dramatically reduced like neural connectivity, then people who wrote on their own. And so I think that that, I mean, you couldn't, this is just like an FMRI kind of as much as you trust FMRIs in their flawed, but like this might be the one kind of like measuring blood flow in the brain is like literally what the one thing FMRIs are good at. So I'm just kind of curious, let's okay, let's assume that that's one of the things that you kind of assume in the book is going to happen.

What is, and just like kind of as our last question, what is the solution to that? Like is this solution, there is this one model that I have that I'm kind of increasingly like forming around, and I had kind of mentioned it before in the pre-show that with like Benjamin Wright who's a computer scientist at Berkeley has wrote a book recently called the Irrational Decision, how we gave computers the power to choose for us. And there's the question of like can we,

I should say, LLNs are just this think or like are this writing, reading calculator, and we decided that people could learn math, but they had to show their work on paper, and we wouldn't let them use a calculator when they were learning long division, and they had to learn how to do it. And the same thing with like all the way up to calculus, like you could check your work, but you were only going to get one point for the right answer, and you were going to get,

you know, you needed to be able to show the proof and show that you actually understood it, and so like, is there a version in which we could cast LLMs, and they're in their place and education, the place that like people to avoid the knowledge collapse, and which we just simply to say, we walk away and say, no more Chromebooks in schools, no more like no more cut, and I'm not an anti-tech

“person. I think the Chromebooks have their time in place, maybe in a computer lab,”

like we used to use like in like the 90s or the odds, you know, like there's a place in time for them, like a calculator, and it's a really important place in time. They're huge, huge, really powerful tools, but like is that, is that a solution that you're interested in or what are your solutions that you come up with in the book? So, well, I don't have solutions in the book for that.

I mean, my solutions broadly are in the book, our skilled workers, not always, but more often than

other people know how to use tools to make their products better. Make the things they make better, and that, that, you know, within the community of say radiologists or lawyers or doctors or whatever, that's a discussion they need to have about when they use it this way, you get better outcomes and when they use it this other way, you get worse ones. I think that's, that's, it will be domain-specific, and this is why we have professional academies in licensure and professional ethics,

“as well. One of the reasons I think that firms are so interested in AI is they like the idea of”

prioritizing workers who right now are legally obliged to say no, right? Doctors aren't allowed to say yes when to a hospital administrator who wants to do things that that indeed your people's lives, lawyers, there's codes of ethics that they're supposed to hue to, and AI is a way to sidestep a lot of that. And, you know, again, if this were coming from the bottom up, it would be different. And it's one thing for like someone to say, oh, I'd like, I'd like the AI to mix me some

peptides and for them to be like wrong and grow at tentacle. But it's another thing, you know, the money for for AI is like coming with the expectation that you can go to a hospital administrator, and then they can use some combination of AI to do labor discipline so that they can turn away indigent patients, right? Like that's the, you know, by firing all the doctors who would say no to that, you know, so that's the, that I think that's the risk and so this is where we're professionals

“and comes in. In terms of the classroom, you know, there's a joke for me Eastern Canada,”

like all the best Americans I am Canadian, and there's a joke for me Eastern Canada whose punchline is, if you wanted to get there, I wouldn't start from here. And when I look at kids using AI and classrooms to do stuff, that's just crap, it's because the assignments are crap, right? Like, like, the five paragraph APSA, the DBQ, like these are, like, if AI is what you use when you don't care something is done well, you could not ask for a better example than then standardize curriculum and

standardize testing and standardize assessment. You know, software is always going to be able to

write a better formulaic, five paragraph essay that is organized around consistent ability to grade it so that any two graders will produce the same mark from it because it has no qualitative elements. It is, it is, it is, you've removed all qualitative aspects from, from essay writing and left it in the realm of the purely quantitative. Of course, software can hit that benchmark better than you can. I mean, this is just the living proof of the law that every metric becomes a

target and ceases to be useful as a metric. And so, you know, it's a big, it's a tall order, but I would like to refactor education around continuous individualized assessment in which case it wouldn't matter if the software is doing it. You know, ideally what you want, like, I, I was just, I just spoke at a conference at Yale and there was a, um, there was a sign up in the hallway for the undergraduate writing program, like their freshman comp program and I asked someone

about it and they said, yeah, that's some, the most expensive course Yale delivers. There's one instructor per two students. And the idea is to, like, basically, instill a love of, of language in those students such that asking the AI to write your essay for you would be like asking the AI to eat your pizza for you. Like, why would I ever want an AI to do that for me, right? Like, this is the good part, you know, and I, I went to weird alternative schools where we had great

writers workshops and I would never, ever want to use AI to compose sentences for me. Find

typos all day long, right? Maybe even suggest some fact checking lines to, to work through.

this is a typo, I'm like, no, that's a coinage, right? The reason you think it's a typo is that

“you're dumb and I'm smart and I thought up something that no one's ever done before. So that's why”

it's there. You know, and, and so I can ignore it, but, you know, asking the AI to do my coinages? No way. Yeah. So I think this is a super interesting question, a little bit off topic,

but we should, you know, as always, we should do this again sometime, Corey. But it's a really

interesting question of whether AI ends up giving us the resources to do this bespoke Athenian like education and what you have like one, one teacher to every, like two to three students or whether it ends up becoming using AI to create bespoke prompts and masks that really like tailor and know a student and kind of like turn this in to like this recruit, like the teacher begins the AI versus like the teacher frees up labor to be for tip for humans to become teachers.

“Yeah. So I think the problem is, but there's, there's a slight conflation, which is that the”

problem isn't just that we don't put enough money in education. The problem is that we're so obsessed

with accountability for teachers who are another one of those professional classes who get to say no to their bosses that we have removed all agency from teachers by standardizing curriculum. So it sucks. And I think that without raising the price, you could just give teachers the ability to use the discernment they're taught in teachers college and that they develop through their professional lives and go back from this robotic teaching. Of course software can be a

“better robot teacher than a teacher can, but that starts with the stipulation that robot”

teaching is the way to teach. Well, Corey, thanks for coming on the show to talk about your great book. Really appreciate it. As pleasure. I love having these chewy, gnarly discussions. These discussions, I kind of wish I could have a very eye when I read the book anyway, it was as opposed to like, well, you know, when is the word guessing machine

going to turn this into paper clips, which like God, if I never after have another one of those,

that'll be very good. The law fair podcast is produced by the law fair institute. If you want to support the show and listen at free, you can become a law fair material supporter at lawfairmedia.org/support. Supporters also get access to special events and other bonus content we don't share anywhere else. If you enjoyed the podcast, please rate and review us wherever you listen, it really does help and be sure to check out our other shows. Scaling laws, rational security, allies,

the aftermath, and escalation are latest law fair presents podcast series about the war in Ukraine. You can also find all of our written work at lawfairmedia.org. The podcast is edited by Jim Patia with audio engineering by Cara Shillin of Goat Rodeo. Our theme song is from Alabama music and is always thanks for listening. Do your current managed services really help run your operations or are they just running in circles? Running isn't enough anymore. With PWC's

managed services, your operations don't just run, they evolve continuously, powered by AI embedded directly into your workflows. So instead of maintaining yesterday's model, you're building tomorrow's advantage. PWC's managed services, we run your operations with tech and talent, so you can run faster, scale smarter, lead stronger.