Inside the Fight Against Non-Consensual Deepfake Porn

Today's episode discusses sexual violence including rape, graphic sexual content, and suicide.

“So you might want to keep this in mind when you decide if and where to listen.”

Hi everyone from New York Magazine in the Vox Media Podcast Network. This is Bond with Kara Swisher and I'm Kara Swisher. My guest today are Attorney Kerry Goldberg, tech journalist and CEO of Mostly Human Media Lori Seagull, and VS Supermonin, the Walter P. Murphy Professor of Computer Science at Northwestern University.

We're talking about non-consensual, deep fake pornography or AI-generated explicit videos and images that use real people's likenesses without their consent. ories were often the target of early pornographic deep fakes, now according to one survey, one in 18s personally knows someone who has been targeted. Nearly all deep fake porn depicts women, despite laws and moderation efforts, pornographic

deep fake websites and so-called "nutify apps" continue to multiply. And a recent analysis from wired found that Elon Musk's grock, chatbot, is still being used to generate and host non-consensual deep fakes. I am not surprised in the least. Kerry Goldberg is a victim's rights attorney, her law firm specializes in fighting Internet

abuse and revenge porn. Journalists Lori Seagull recently released, searching for Mr. Deepfakes in investigative docu-series created for TikTok and partnership with Paris Hilton. It follows an investigation to unmasse the anonymous operator behind one of the internet's most notorious Deepfake porn platforms.

D.S. Supermoneyan is a top computer scientist in Deepfake expert. He led the team that developed the global online deepfake detection system, a free platform to help journalists detect deepfakes.

“I think it's critically important to talk about this issue when I started my career.”

pornography was at this heart of the problems with the early internet and the uses of it and the abuse of it was built into the system at the start. A lot of people were aware of it then and have had a very hard time dealing with it, even as technology has gotten better and better at being more and more abusive.

So it's critical that we keep talking about it.

We criminalize this and we start to hold platforms, big platforms responsible for the stuff that they distribute. Our expert question today comes from Wall Street Journal, Tech Reporter, Georgia Wells. This is an important conversation and one we will keep having so stick around. And one more thing before we get started, if you're going to be in Washington, DC on July

16, please join us live at the Johns Hopkins University Bloomberg Center for a taping of on. I'll be talking to Gina Remando, Commerce Secretary under President Biden and former Rhode Island Governor about what AI means for the American workforce and as a special bonus before that conversation, I'll be speaking with Johns Hopkins University President Ron Daniels

and the University of Notre Dame President, Father Robert Dowd about how universities are approaching AI and workforce issues. You can get tickets and learn more at boxmedia.com/caroswisherlive. Support for the show comes from KPMG. In any organization, disruption is inevitable, but struggling through it doesn't have to

be. KPMG adaptability index is your blueprint for building capabilities to handle what comes next. It uses real data to look at how your culture, strategy and partnerships all work together to help your business thrive.

Stop reacting and start adapting. Visit kpmg.com/us/adaptability to explore the adaptability index and pulse surveys today.

“How many third-party vendors does your company use?”

20? 200? Thanks to AI, someone on your team probably added three more this week and your security lead has no idea.

Traditional third-party risk management can't keep up.

Vanta gives you continuous coverage across every vendor, automatically, so you actually know what's in your stack and what to do about it. Vanta.com/TPRAM Hey, it's Ryan Reynolds here for Mint Mobile. Now, I was looking for fun ways to tell you that Mint's offer of unlimited premium

wireless for $15 a month is back. So I thought it would be fun if we made $15 bills, but it turns out that's very illegal. So there goes my big idea for the commercial. Give it a try at midmobile.com/switch. Up from payment of $45 for 3 months, $90 for 6 months, or $180 for 12 month plan required,

Initial plan term only greater than $50 gigabytes in slow and network is busy, C-terms. Kerry, Lori, and VS, thanks for coming on on. Good to be here. Thrilled to be here. Hello.

Hello. So I talked to experts back in January about the proliferation of sex-wise non-consensual AI deep fakes, including images of children on Elon Musk X that were generated by its grok chatbot. Obviously, AI created deep fake porn hasn't gone away on grok or elsewhere, and the technology continues to improve.

But I'd like each of you to explain the current scope of the problem and how pervasive it is and who it's affecting and how technically advanced it is. We're going to get more into detail. So go high level here. Kerry, we'll start with you, then Lori, then VS.

So I'm a plaintiff's attorney that suits tech companies, and generally I see problems at the very early stages. So you know, back probably seven years ago we started getting situations where celebrities and really high level content creators were being deep-faked, and then over the last just I'd say year or so, it started trickling down to middle schoolers getting deep-faked.

All the apps involved were these pretty obscure things, but then everything totally changed when Elon started sending around pictures of toasters and bikinis, and it just unleashed

this never before seen sort of situation where everyone was able to create deep-faced,

and then immediately publish them on the X platform. So that's where you saw it. Lori? Yeah, when we started looking at this years ago, it was very much focused on celebrities,

“oftentimes women in power, politicians, it seems like the first line of defense, and I remember”

thinking at the time, "Oh no, you might not care about these people, we should care about these people." But what this means is this is going to happen in our high schools, and unfortunately that's what we're seeing happening. So I would just say it's rampant, and it's happening in our communities, it's happening

to women, to all sorts of folks, and unfortunately it's something that teenagers are learning, and it's becoming commonplace. Go ahead, VS. I'm a technologist, and so I've spent a lot more time thinking about the technology. Over the last five, six years ago, the ability to create deep-faced was in its infancy,

but over the last five years, and in particular over the last year or two, it's really gotten to a stage where even experts like me, on the detection of deep-faced, have difficulty eyeballing something, listening to something, and telling if it's real or fake. So what we're seeing is something where you really need humans and AI to come together, to work as some kind of partnership, to try and figure this out.

And, you know, until we can get that right, I don't see an end in sight.

“And how has the technology improved in the last year?”

Just ease of use?

These reviews have always been there. The improvement has been in the quality of the generator deep-fakes.

So today, for example, we're seeing things like face swapping. So I could record something of myself saying something thoroughly objectionable, and then put, you know, replace my face with that of a celebrity. In the past, that was difficult to do. There were things around the edges of the face that would be noticeable.

They much of that has been worked out. And ease of use for normal people to do so, correct? Indeed, absolutely. So Mr. D. Fakes launched in 2018, and was once again the largest deep-fake porn site online getting 17 million visits a month at its peak.

Lori, talk about your investigation into Mr. D. Fakes and how it began. But again, a bit with an obsession to be quite honest with you. I remember it was 2022, chat, GPT had launched. Everybody was talking about how AI was going

to just do these incredible things.

It's going to cure cancer. It's going to, you know, give us all this free time. And of course, I mean, Cara, you know, there's better than anyone. We're all kind of looking at this and saying, okay, there are a lot of unintended consequences and this is moving very quickly without the correct guard rails.

And someone told me about Mr. Deepfakes.

“I will never forget going to the site and opening it up and seeing thousands, thousands”

of highly, sexually graphic images, deepfakes of women who did not consent. And they were violent in nature. It was a website where a creator could say, you know, could go and make money. And someone could go and say, I want this woman doing this and they could pay for it. So it was beyond just a website, it was a platform.

And then what was even more concerning as we were digging into it was, you know, there was everything from training data of women's faces to create hyper realistic deepfake chronography. And then there were forums where people would exchange notes on here's the latest deepfake

Here's how you can do this. You can pay a premium and get a deepfake training manual. And so my initial thought was, this isn't going to just be for these high profile women. And high profile, I would say, let's take it with a grain of salt.

“And I remember thinking at the time, well, it's not going to stop.”

Here, you know, this is just the beginning of it. And by the way, you're looking at these forums and you're seeing people say, is it wrong that I want to do this to my sister and law? And, oh, this is so easy to do. And so it's almost creating entertainment around deepfake abuse.

And so as I was looking at this, I thought, well, surely there's something these women can do to help themselves. And the reality was absolutely not, there was nothing they could do because the man behind it was anonymous and he'd been anonymous for six years, and even so, even if he wasn't anonymous, the laws hadn't caught up.

And even when I went out and started talking about this, and by the way, you never

want to be the person talking about deepfake point out the dinner table, but turns out I was, people look, well, the images aren't real, that it's so it doesn't matter. And I'm sure everybody on here is kind of just shaking their head. So we also had a cultural problem. We needed people to realize that these are so hyper realistic.

Actually, this causes deep, deep psychological abuse, and it can be used to try to digitally abuse a woman, I say, a woman, anyone. And so that was what was alarming. And so we decided we had to go find him and raise awareness. So that was the beginning of the investigation.

Let's hear from Joanne Chu. She's an artist and actor. One of the women featured in Lori's investigation. She's talking about the moment when she realized her image had been used in porn on Mr. Deepfakes.

I was just bored one evening and I'm just going to Google my name and see what comes up.

“I kept scrolling down and then I said, what, what, what is this?”

It's like my face on very disturbing and graphic content. It's, you know, Joanne Chu or Joanne Chu in the beginning, it was just a few listings. Mr. Deepfakes came up social media girls, e-parner, and then when I typed it in again, it multiplied. It's very much akin to physical assault.

I have, like, less than 3,000 followers, but it doesn't matter if you're famous or you're not just looking at Mr. Deepfakes, it's like they have a whole alphabetized database.

At one point, when I typed in my name, my stuff wasn't coming up first anymore.

It was all of this content. So she compares it being physically assaulted, carry, does this reflect what you've heard from your clients? Because it sounds like perpetrators are telling themselves that it isn't real harm. Yeah, I remember this when I first started my firm back in 2014 and it was everyone's attitude

about revenge porn and how, you know, this is just an online problem and back then there were all these revenge porn websites and people were searchable through Google, which would lead people to search engine results and it would just be pages and pages of nude images. Like all the harm was really, really underestimated back then and it took a huge amount of advocacy to change the public opinion about the shame around being nude online.

And we're just, we're seeing something really, really similar. The difference though is that back, you know, 10 years ago, women were being blamed for having shared an intimate image in the first place and with deep fakes, I don't see that same kind of victim blaming because here no one, no one consented to the image in the first

“place, but there is still a lot of, I think, marginalization and minimizing of the harm”

that it's not physical assault, that it's, yeah, I feel it is conceptually different than a physical assault, but that doesn't mean that it's not still really, really harmful and humiliating for victims, especially like I see so many young people who are victims and it's other young people's doing it to them, you know, where it's peer on peer in junior high school and high school settings and a lot of times the offenders just think that they're

doing a prank or if, you know, and always the defense as well, I barely shared it with anybody.

I didn't post it online, but these things find a way to get published. So deep picked technology, first of March around 2015, the S talk about the online communities that exist for sharing information and making so-called new-to-fi tools accessible and how is it involved over the past 10 years in terms of how it's used? So, Cara, there are a number of notification AI tools, so these are tools that are of

various varieties. One, as I mentioned earlier, is face swapping where you've got an individual who's nude, performing some sex act and their face has been replaced with the face of somebody like Mr. Chu who you alluded to a couple of minutes back, so that's one class of techniques.

and generate new news of people who are hyperrealistic, who are performing acts,

which the real individual portrayed never performed, so that's another class of techniques.

And I don't want to go into the gory details of how that's done, but there are a number of techniques, these techniques have evolved better and better every single year. And so, as a consequence, what we see is really a huge number of websites that can disseminate such stuff, not just the produced content, but also the tools that I use to create and disseminate this stuff, and that's typically done on the dark web through some kind of distribution network on the dark web.

And what changed when Elon made it easy to do so, because for all eyes on the dark web,

“you have to actually make an attempt to go get it, right? It's like going to a porn store or”

going to a porn movie or something back in the old days. Talk a little bit about how that changed.

Well, I can't speak specifically to Elon and what he did, but what I can say is that the more

these tools are available through normal channels, the more the audience that gets access to them. And as soon as you have a big audience, you have a bigger audience of badguides, or as carry mentioned, just teenagers who don't know better and haven't been taught better, possibly because they teachers and their parents, however decent they may be, well, not aware of the threat. And by the time they figured out, it was too late.

Deepfakes and new-to-fi apps are also a big problem in schools, as we mentioned earlier, a recent survey of roughly 550 U.S. teens found that over half had created at least one image using notification tools while the third of the teens reported having their image created and shared non-consensual, I'm sure the numbers are higher actually. Sites like Mr. Deepfakes and apps like Grock have, as we know, have normalized, creating non-consensual, in some cases,

violence, sexual imagery, often of women. Laurie, talk about the consequences for young people.

Yeah. You know, I'll start. I always like to say to try to make it a little bit personal.

I'll never forget, and I'm going to take it here, and I promise I'll get to young folks,

“but I'll never forget, I think it was 2023 when a group of technologists approach me and”

say, can we have AI attack you and see what it could find? And because I am a crazy person, I said, "Sure, let's see." And what they did was they had AI come up with an attack against me, and it took some true things, like I've interviewed Mark Zuckerberg before, or I'm a technology journalist, and then it ended up in front of an audience. It ended up creating the created sexually explicit deepfakes of me. And I'll never forget, because this, I had consented, I didn't realize they were

going to do that, but I had consented to this demo, and I'll never forget looking at the audience, look at these sexually explicit deepfakes of me, and I could, as a person who generally formed sentences for a living, I stopped being able to form sentences. I felt shame, I felt humiliation, I felt like the world had seen me naked to be quite honest with you, and then I thought to myself, well, if that's how I feel, and I know what this technology is, imagine what that's going to be like

“for an 11-year-old girl. And I think that's what we started hearing. So I've spoken to a lot of”

parents, a lot of teenagers who've had this happen to them. I spoke to one woman who said, and you know, in her early 20s, she said she, after this was happening, and she went out and she tried to get this stopped, and people said there's nothing you can do, they blamed her, very similar to what Carrie is talking about, back when we were talking about non-consensual pornography, people didn't quite understand it. She told me she walked to the roof of her building, and she considered jumping off.

And you talked to a politician who describes this as digital rape. And so for me, it was really confusing because I was looking at this website. That for me, this website represented this dystopian era where we lose consent and an AI era where, you know, our most intimate qualities can be weaponized against us. And then externally, we were hearing, but it's not real, it doesn't matter. When it really impacts the way a young person, any person walks through the world. We spoke to a woman

a molly who was a mother who had this, her husband's best friend, deepfaked the whole neighborhood, 80 women, and the community, and it created sexually explicit deepfakes. And she said, it impacted how she would go on the bus and look at people and wonder if they'd seen this. And she had to go on a, she said, a porn hunt for her own face on the web. And I think it really impacts from a human perspective how you walk through the world and how you hold yourself and it can have reputational

harm. Right. So, Terry, your law firm in your words, Sue's, quote, psychos, purves, trolls, and toxic tech. In January, you're from filed a lawsuit against XA on behalf of Ashley St.

chatbot was used to create and disseminate sexually explicit non-conceptual images of her,

“talk about the lawsuit and how is it representative of bigger legal issues around deepfaked porn.”

When Grock suddenly had this new capability of being able to create

notification images, this was the first time in history that a new-defying tool was integrated

into a widely used social media platform, which combined the ability to just generate an image with the widespread dissemination network. And so, suddenly an image could just be viewable by millions and millions of people. But also, I mean, you know, XA's for people 13 and up. And so, children were also then consumers of the images, which is illegal. And we're also, there were children that were the victims of it. And the images posted were not just people in

bikinis, but Grock would pose them in sexual positions. It would respond to prompts telling

“them to drench their body in blood and seeming like fluids, hold suggestive props, and just”

really vulgar and explicit images. And it seemed very clear that this was an intended rollout

of a product that was known to be harmful. They were like something like 1.8 million posts

of women and children being deep-faked, which accounted for like 41 percent of all the images posted during this nine-day period. So, Ashley St. Claire is somebody who is really demonized on the X-platform because of some of the hostile things that Elon Musk has said about her was a huge target of the deep-fakes. And there were just so many images that Grock generated and published on her own page. And so, we first got attempted to get a temporary restraining order

to stop Grock from deep-faking her. And then we sued them for defective design and others

of sort of product liability types of lawsuits, negligence. And the claim that I like the best

is that we sued them for public nuisance. There's just no reasonable, safe use of this product, but to harass people publicly. So, those, that's where it stands now. And yeah, well, we had to give X-A-I statutory notice that we were filing for a temporary restraining order. And they immediately sued Ashley in Texas claiming that she had breached the X-A-I terms of service by even threatening to sue them in her home state of New York. So, now we are litigating the issue in two different

venues. They're also trying to use the X-terms of service to get the X-A-I case transferred to New York. And the judge initially last week said that X-A-I can rely on the X-terms of service to transfer the case to Texas. Do you mean to Texas? Yeah, so not even though X is not a defendant in our case,

“only X-A-I is. The judge still let them borrow from a different Elon company, which I think is”

really scary. So, is that where it's going? Well, we filed an emergency petition for a man damage to try to get the stay held up, but I don't know, we don't have the outcome on that. We'll be back in a minute. Support for this show comes from Quince. A nice linen set lets you get away with a lot of outfit repetitions in the summer. You could wear the same one to a laid-back barbecue or upscale

seaside restaurant and still feel classy. For elevated summer wardrobe pieces, check out Quince. They offer just that and they use premium materials like European linen, organic cotton, and washable silk without the traditional retail markup. They're 100% European linen pants, dresses and tops, but the pieces you'll reach for all summer long. Their lightweight effortless to style and start at just $32. They also have beautiful 14k gold jewelry that adds a subtle

finishing touch to any outfit. And it's not just clothing. Quince offers everything from home to travel to everyday essentials. Everything is affordable too. All of Quince's items are priced 50-80% less than similar brands. In fact, I have used my Quince suitcase at least four or five times this summer. It's simple to use and it's very durable. I also use their shirts and t-shirts a lot this summer. They're very breathable, easy to use and very comfortable. Make your summer wardrobe easier.

Go to Quince.com/cara for free shipping on your order and 365-day returns. Now available in Canada too. That's QUINCE.com/cara for free shipping and 365-day returns quince.com/cara.

building a great team is the fun part. But nowadays that search for great people is global.

“It's just how business is done these days. Pebble is a hiring platform built for modern teams.”

Their platform takes delays and guests work out of going global and they do it with embedded compliance and AI-driven workflows. With Pebble, you can hire an over 185 countries in minutes. And when it comes to compliance, Pebble has built a decade of expertise directly into the platform. Pebble brings everything, contracts, payroll, benefits, and compliance together in a single platform. Your global team can actually be set up for success with the support they need to

thrive wherever they are. Bottom line, anywhere is possible with Pebble. It's global hiring, simplified. And here's a special offer for our listeners. Pebble is normally $399 a month per employee. Already a no-brainer for what you get. But right now, there's a limited time offer on their site that makes it even easier to get started. Go to highpebble.ai before it's gone.

“That's hi-h-i-p-e-b-l.ai terms and conditions apply.”

Support for this show comes from Gusto. Be honest, there's probably one task on your list. You constantly push to next week because it's just so tedious. For a lot of business owners, like myself, that task is payroll. Gusto is here to take that entirely off your plate, so it becomes an easy part of your job instead of the dreaded one. Gusto is an online payroll and benefits software built for small businesses. It's all in one, remote friendly and incredibly

easy to use so you can pay higher on board and support your team from anywhere. Actually, podium media uses Gusto to operate. Automatic payroll tax filing simple direct deposits help benefits, commuter benefits, workers comp 401k, you name it, Gusto makes it simple and is options for nearly every budget. Unlimited payroll runs for one monthly price. That means no

“hidden fees and no surprises. You can save time with built in automated tools, offer letters,”

onboarding documents, direct deposit, and more. It's quick and simple to switch to Gusto, just transfer your existing data and get up and running fast. Try Gusto today at gustow.com/cara

and get three months free when you run your first payroll. That's three months of free payroll

at gustow.com/cara. Again, that's gustow.com/cara. So let's get to the tech infrastructure that supports the creation and dissemination of deepfigs. Analysis from media outlet indicator found that notify apps often use infrastructure from tech companies. VS explained the relationship between deepfake porn sites tech companies in the social media platforms and to what extent do these sites and notify apps rely on

big tech infrastructure. This was an issue when I was interviewing people who were creating all manner of nonsense around the insurrection on January 6th and companies moved quickly to take some of it down, but they were clearly the way people got to these different sites. That's a broad

question, Cara. So let me try and break it down into pieces. The first is, depending if you're

generating deepfakes at scale. So think the kind of thing Lori was talking about with Mr. Deepfake, he's trying to generate thousands, millions of deepfakes and so that requires computational hardware. What I call graphical computing units or GPUs, which today are pretty scarce and expensive. So and that scarcity and price doesn't look like it's coming down very much anytime in the new future. Those resources are heavily supported by cloud infrastructure providers. Again,

I'm not naming any specific companies, but there are a number of companies that offer cloud services. So part of the onus is placed on them. If somebody is trying to use their service to generate this kind of stuff at scale, if somebody's trying to generate just a handful of deepfakes, they're probably not going to figure it out. But if somebody is generating millions and millions of them or thousands and thousands of them, they may have some idea of what's going on.

But that's not easy either, because you know, there are many challenges for some of these firms in figuring out what's a deepfake and what's not, what's consensual, what's not. And so as the

first issue, you know, the bad guys are continuously evolving. So that means that the actors and

questions are changing. The technology used to generate this stuff is changing and it turns out having studied deepfake detectors quite a lot and created someone my own with my lab,

if I'm drinking this cup of coffee and there's a deepfake of me, with part of my face obscured as I'm drinking that cup of coffee, it's harder in many cases,

for many detectors to figure out that video clip of me saying something which I probably never said

is not true. Likewise, the definition, Kerry, I'm not speaking from a legal perspective, but for a normal person, what constitutes an act that sexual in nature can vary a lot. A sexualized act versus an actual act of, you know, some kind of sex, very different. Also, the relative positions in which a sexual act is carried out where certain body parts may be obscured, but other things may be clear. All that causes machine learning algorithms to detect deepfakes

“to have a lot of problems. So I think, you know, those are some of the challenges from a technical”

perspective in figuring these things out. So one analyst said that some deepfake sites now offer APIs to people creating non-consensual images and video generators, explain what that means. So when you go to a website, you're typing stuff in, a prompt or a query, or clicking on some buttons, and you get a response. An API is an application program interface and it replaces you with a piece of computer code. And that piece of computer code is interacting in a different kind of language

with the server behind the website you're going to. And so if somebody is using an API to generate deepfakes, that suggests that they are generating it at scale. That suggests that they have a piece of code that is shipping requests automatically through that piece of code to the server and that server is sending back whatever they requested. In this case, some kind of sexualized image.

“That suggests somebody who's trying to do it at scale, who is, you know, who probably has his”

or her own databases of people they want to propose in these situations that they will never

and to begin with. Indicators analysis also estimates that the new to fire economy may be worth up to $36 million a year, Lori, and you're reporting on Mr. Deepfakes. You noted the financial incentives for the community members posting in the forum. Tell us about the incentives and talk about the money making, Lori here, and for the platforms that host this content too. Although for people don't realize Google says it doesn't allow apps that contain sexual content, Apple says

the company's apps to a prohibits overtly sexual content, both companies disabled searches for new to fire in their app stores. We'll get to the work around in a second, but talk a little bit at the incentives, the financial incentives. It's interesting because when we were looking at this site, that wasn't a site that was a platform. It was a whole ecosystem that enabled the site to exist, and that included platform liability from a lot of these different companies. So for

example, you know, the incentive was you can go on, not only can you view, you know, thousands of videos of sexually explosive deepfakes if your favorite celebrity you could make your own, and this is why we believed Mr. Deepfakes was so dangerous. He was dangerous because he was creating an army of Mr. Deepfakes because now they're all these creators on there, which is kind of paid a play. And there's a freemium model, and then there's this freemium model,

where you can pay to have specialized deepfakes of public figures who didn't consent made. And so people were probably making more money than the guy behind Mr. Deepfakes off of this platform.

It was creating a whole ecosystem, but what else enabled that ecosystem? First, I would go from a

regulatory standpoint. The laws just hadn't caught up, and we saw this with non-censitral pornography oftentimes tech moves quickly when there are the correct guard rails, women, and young folks are the first ones to feel that pain. But also there were payment providers that were accepting payments on this platform. And so it was all of these things. And we talk about these small steps that actually helped enable this ecosystem to exist. And one of the coolest things we saw

“because I think so many folks would say, well, it's a game of whack-a-mole. You go after this guy,”

and then they're all these other ones that are going to pop up. But actually, what ended up happening is we did go after this guy. We found out who he was, and I think maybe most alarming was he wasn't a technical guy. This is a guy who was, I would say, in his 30s, worked at a pharmacist at a hospital helping people. Was newly married, had a young child, and he had been able to create a website that enabled a whole new wave of deepfake abuse,

because he had an interest in technology and deepfakes. And he saw an opening, because when Reddit banned, sexually explicit deepfakes, we were able to track where his username was talking

And so all of these things together created, I would say, an ecosystem that allowed it. And all

“of these things also, once people started chipping away at this, once the law started slowly,”

but surely catching up, once Google deranged the platform, there were all sorts of small steps that actually created a friction that made it harder to exist. So let's shift to the current laws and regulations surrounding deepfake porn, including the take-it-down act, the law that makes it a federal crime to publish non-consensual explicit deepfakes. The federal government began enforcing it in May. Carry in your pursuit of tech

platform accountability, you've come up against Section 230, as have many, the law that protects platforms for being held legally liable for content that users post. You mentioned product liability law as a way to stop deepfake porn and get justice for victims. So talk about using

“product liability law. Section 230 was intended to basically just immunize platforms for content”

that third parties created and posted. But when it comes to deepfakes, especially like in

the situation of GROC, we're saying that GROC was the one who created this material and also published it, and therefore Section 230 shouldn't apply at all. But even if it does, by suing them under product liability, we're suing them not for publishing the content, but for, you know, basically harming people through its defective design, defective manufacturing, these were all foreseeable uses. There's no reasonably safe use for a new-defying product.

And so that's just like a second way to overcome Section 230. Now, XAI, they haven't fully exposed what their defenses are going to be in our case, but when they were opposing the temporary

“restraining order, they were claiming that they wanted, I mean, so far, this would be the only AI”

product that is saying that Section 230 should apply to it and that they're not responsible for what people type in as prompts. But even more alarming is that they're claiming that they should have the first amendment protections, and that GROC should basically be protected under free speech, which is a pretty alarming idea, you know, that AI and a chatbot would have free speech rights. They don't, you know, so far, we're not at a point where we're giving constitutional rights

to a chatbot. But again, the arguments haven't yet hit the stage where they've been fully briefed yet. So Vias, talk about work arounds because a lot of these companies are very clever

in terms of work arounds. Foreign people have always been very clever. How easy is it to do work

arounds here from a technological point of view? You know, I was thinking about this when Laurie was speaking earlier, Tera. You know, she talked about a website where somebody goes in and says, "I want to create a deep fake of such and such person doing such and such thing." And so that's a classic example of a place where that website is using the APIs that you asked about earlier to access a cloud provider where this code running to generate the notification of the sort

desired by whoever asked the query. But now, think about it this way. Let's say the platform has the mechanism, let's say they have a perfect mechanism that doesn't exist today, but supposedly did to detect the deep fake video of some form of sexual activity. Then the work arounds for the porn guys that are some simple ones. So one of the classic methods to generate deep fakes, regardless of whether it's for good or bad, is something called stable diffusion. So unstable diffusion,

what you do is you let's say throw an image. I'll use an image rather than a video as my example, but you throw an image of somebody and that image gets converted to some very coarse representation. You know, think of this as something that's a very skeletal version of that image. And then what ends up happening is that skeletal version just goes through the notification process, but at the skeletal level. So that's skeletalized, unified image doesn't look anything like

what a real nude might look like. And so you can think of it as some kind of computationally weird representation of, you know, a person who's about to be sexually abused via a deep fake. And then they could do all this on platform on a cloud. And then without ever passing the source

and perform the last step back on their own servers, whether generates the final nude. So that's

an example of one way in which somebody with, you know, just limited computer science knowledge who's familiar with the code, who has access to all the code, can generate something while avoiding the guardrails of some of the platforms, even if those guardrails are closed to perfect. Support for the show comes from groans. Some are can disrupt our routines with busy weekends that are full of travel and events, and with everything going on, it can be hard to remember

to stick to your wellness habits. Well, groans is one of the wellness habits you don't have to think about. It can be the easiest tastiest part of your busy schedule. Groans are convenient

comprehensive formula in one delicious daily pack of gummy bears. This isn't a multivitamin

a greens gummy or a pre-biotic. It's all of these things and then some of the fraction of the price and it takes great. They're able to pack in 20 vitamins and minerals in a daily pack with 16 ingredients, all of which include nutrient-dense and whole foods. A greens pack also contains six grams of pre-biotic fiber to help you visualize it. That's equivalent to more than two cups of broccoli. Groans and greens are backed by over 35,000 research publications. You can't

fit the amount of nutrients. Groans does into one daily gummy, which is why it comes in a pack. Say 52% off with the code, [email protected]. That's code, cara, k-a-r-a-g-r-u-n-s dot c-o.

Support for this show comes from Harvey AI. The future of law is a genetic, not just tools that

“assist, but AI agents that navigate complex matters. That's why Harvey created agents that can”

do the work from end to end. They build a plan, pull from secure data sources, run sub-agents and parallel, and draft the work product ready for your review. So you delegate the work and on the judgment. Trusted by more than 60% of the Am law 100 in leading Fortune 500 legal teams, Harvey is the AI operating system designed specifically for legal work. Harvey, AI, tailored for law. Learn more at Harvey.ai. Support for this show comes from Saint Germain, Elder Flower Lecore. Nothing says

summer like a sunny backyard hang. Think light colored linens everywhere and a easy conversation on your lips. Your favorite or driven one hand and the bright refreshing taste of a Saint Germain Spritz in the other. Those are the details that stick with us. And the Saint Germain Spritz, an elevated take on the Hugo Spritz, is the simplest part. Just mix sparkling water, wine, and Saint Germain for your go-to summer cocktail. Visit Saint GermainLecore.com to shop and explore

recipes. Enjoy responsibly. The Defiance Act, which is aimed at allowing victims of non-consensual deepfags to superdamages passed in the Senate and is currently stalled in the house. Even though some states have criminalized deepfags enforcement remains low. Lori, what extent are these laws effective when it comes to stopping or at least curbing the spread? It's happened before with credit card companies and certain businesses, you know, things tend to slow things down. But in this

case, how important is it to have the take it down and the Defiance Act? I'll answer and I'm curious

“for Carrie's response too, but I think it's incredibly important. You know, the take it down act,”

you know, force tech companies to pay attention and quicker gives them 48 hours to take down non-consensual abuse. But oftentimes, when it comes to victims, it's really difficult to find recourse for this. You know, and so the Defiance Act would create civil penalties. I would enable them to actually have the ability to go after the folks who have done this and have civil recourse, which I think makes a big difference. And when we started out on our search for Mr. Deepfags,

which was, let's just be honest, really, a search for asking why is this allowed to happen. You know, there weren't a lot of state laws and I will say a lot of the state laws and a lot of the victims and survivors we spoke to. Unfortunately, the owners in many ways becomes on these survivors to speak out about it and help change laws. And they did in the state level, which, you know, the state laws changing all of them varying in scope, civil, criminal penalties. Actually,

we're where a lot of these different victims initially had recourse, which is also scary to watch as folks go after regulation at the state level, because oftentimes the federal government is just slow to move. I would say I hope that the Defiance Act passes. It can get through.

“I think this is a bipartisan issue because it's really a child's safety issue as well.”

Right, which is how they're saying until, by the way, a wired investigation found that

and multiple lawsuits and after Musk's XAI said it would add restrictions. In response to an NBC investigation, the XFD account post that we strictly prohibit users from generating non-consensual explicit deepfakes and from using our tools to address real people explain how some of these legal dynamics apply to XAI in that regard since you're in a lawsuit with them when it keeps showing up that they're doing the same thing. You would accuse them of. Well, I mean, that statement that they say that

it's against their regulations and they're basically passing the blame to users. There's nothing

in that statement that suggests that they are modifying their product so that deepfakes are no longer easy to create. And the problem with things like to take it down act, it's great. You know, suddenly now there's a 48-hour window of time that tech platforms are supposed to remove illegal content. But it has no teeth because victims can't enforce it. Victims can't sue the platform and say, hey, you know, it's now been 48 days and that content

is still up. So the problem with all regulations and all these options is that the victims cannot enforce them. And so when we're talking about who's the bad guy, VS has talked about the bad guys in these situations, in my view, the worst guys are the platforms. And so until we have regulations that put the victims in the driver's seats to not just, you know, get criminal justice against the

“individuals, but to actually get justice against the platforms, you know, I think the only”

recourse we have is to use our courts and make these companies pay. So every episode, we get a question from an outside expert. Here's yours and you're all going to answer it. So let's listen to it

first. Hi, I'm Georgia Wells, a tech reporter for the Wall Street Journal. I just finished a deep

dive into the issue of teens using deep fake nudes to harass each other. And so the big question I would ask is that now that the government has just started enforcing the take it down act, how might this change the harassment landscape for young people today? Thank you. Laura, you go first, then carry, then VS. Yeah, I do agree a little bit with Carrie on saying, you know, there's only so much you can actually kind of enforce when the onices is on the victim. I do think from a cultural standpoint,

it is very different than it was a couple of years ago of federal law, take it down act. Again, state laws are at least sending this message that the stakes will be high and that young boys, young teenagers can get into real trouble for doing this and can get charged. I mean,

“I think that at least from a perspective of schools being able to speak to students, parents”

being able to speak to their children about this and having some structure will be helpful. Again, though, I'm not sure the take it down act is going to be the thing that's going to actually change this. I also agree with Carrie on platform liability and creating higher stakes for the tech companies and the whole ecosystem that enables this to happen. Carrie, so I think they'll get the memo that this is illegal. I mean, having criminal laws was really, really effective

when we were dealing with the scourge of image asexual abuse over the last decade. We went from having three states to having 50 states with those laws and my firm saw such an enormous and quick plummet of offenders creating images like that. So that's great, but we still don't have an

apparatus that targets the platforms themselves and prevents them from basically monetizing this

kind of humiliation. Until we have laws that victims can enforce themselves, it's not really harming

“they who I consider the true offenders, which are the platforms themselves. All right, the yes?”

You know, I have very mixed feelings about criminalizing a 13-year-old. All right, but at the same time, we need to protect perhaps other 13-year-olds or 14-year-olds or 10-year-olds from the consequences that accrue to them because of some 13-year-old putting our deep fake porn of that kid. So I think, you know, a first step has got to be something around education. A lot of our teachers in schools are simply not teaching their kids about deepfakes and it's one thing if kids are using deepfakes

or gender of AI to, you know, create reports for their class projects. I mean, that brings up a whole bunch of other ethical issues. But it's another thing if they use deepfakes to explicitly cause damage to a kid and one of their classmates. So I think the ethics around this has got to be

others may not. And, you know, that education and the consequences of those actions

“in a school context are clear, I think it'll be challenging. You know, it's just to sit to respond,”

if it's helpful, just to respond to that. I think the frustration is so many times. I agree, how do we expect 13-year-old boys to know that this is okay when they're getting served up notification apps on social media? You know, I think it's, it is, it's an end. Like two things can be true. And so there's a huge gap between education and being able to educate folks on this and what the platforms are allowing and what they say they're allowing and what's coming through,

it creates a really difficult environment. You know, I worry generally about the broad statement

or the broad platform's moniker. I think it's a wide variety in what different platforms are doing.

Some platforms are going to a great amount of trouble spending a tremendous amount of money to try and perform the best innovation possible so they can put some of these problems to bed and others are doing much, much less. So there's like a huge range of what platforms are doing, some doing relatively little, some making every effort to try and get there. Now, I want to add that, you know, again, even if platforms were doing day very best, there's a famous saying

due to a statistician at the University of Wisconsin. All models are wrong, some are useful. All right. And so whatever models are being used by platforms to find these

“deep fakes, they're never going to be perfect. And so the question is, are they making the effort?”

Some are making what I would consider a very good faith effort? Some? Can you name the who is doing a good job? I'd prefer to stay out of that, but I don't want to talk about specific platforms, but yeah. So let's end by looking at what individuals can do as the fake technology continues to evolve. Carrying now that the Take It Down Act forces tech companies to remove deep fake news, how do you request to take down if you're targeted?

Well, part of what the Take Down Act required was that there be a flow for people to easily request contact removal. And so that kind of infrastructure, I really appreciated that being incorporated into the laws. So ideally, you know, a victim within the platform itself can now report it, you know, whether it's through a URL or if it's like an Instagram or Facebook, then the image itself would ideally have a method to request removal. When digital forensics

professor and Deepfake expert Hanie Farid was on the podcast in January, he expressed concerns that agentic AI will supercharge the creation of deepfakes in the coming months. The yes, how do

“we make sure the new laws and policies don't block positive uses of these tools? So I think again,”

if we don't want to block the technology, we want the technology to evolve because again, as Lori said much earlier in the podcast today, you know, Deepfake's do have the capacity to help do things like better understand disease. I'm not going to call this a Deepfake, but the janitor of AI is being used to create new kinds of proteins that will help cure diseases. janitor of AI is being used to create materials that will degrade and perform in ways that we

would like that would protect the environment and more. So I think what we have to do is to look at intent, you know, what is the intent of the individual who is creating a deepfake? And in the case of the 13-year-old boy or 14-year-old boy who's creating deepfakes of one of his classmates because he's pissed off about something. I think there there's intent, whereas if somebody, you know, there was this guy who created a deepfake of the pope for Francis in Puffy Robes and a very

dapper-looking version of him. You know, that I think is something that's laughable and I find that's, at least I find that somewhat entertaining. I doubt very much of the pope was, you know, horribly offended. And so, you know, I think we want to allow artistic freedom and expression, scientific inquiry for the same time regulate so that it's the uses that are banned or that lead to.

The individual intent one. Yeah. Lori, in the end, you tracked down Mr. Deepfakes. What ultimately

brought down the site? And where is he now? Great question. Where is he now? I would love to know. So eventually, we did track him down with the help of, you know, investigative journalist, the internet and some of our cybersecurity friends who did some pro bono work for us because they have kids and they viewed this as a real threat. And we were able to track him down and, you know,

carry who I've known because I've covered non-consensual pornography before I called her before going to his parents home to make sure that we did everything correct, just in case, you know,

“we didn't want to create any legal liabilities. But I will never forget him looking at me.”

He knew who I was with so much to stay. And I know this is a delusional part of me. But there was a part of me that thought maybe we would have tracked this guy down and we could show up and say, do you understand the real harm this site is causing? You're a father. And I remember he walked right into the hospital. He wouldn't say a word to me and the doors closed. And I remember thinking at that point, well, we're going to have to go talk about this to everyone who will listen to us.

And that has to change because we have to change culture as a part of this. This can't be where the story ends. He can't just take himself off social media, which he did and continue running this site. And so a lot of things happened, which I think we're really exciting. I mean, we, I think we're

one of the first to show up and threaten his anonymity. Say, you cannot do this type of abuse

and remain anonymous. So that was friction. That exposure was friction. A law started changing when in the UK, they started talking about different types of laws that would make it harder for folks to access Mr. Deepfakes. They caught a faxest to folks in the UK. That created friction. A very valuable server went down for them to take it down at past the CBC published their investigation

“as well. Mr. Deepfakes finally went offline for good. And I think it was such a great lesson in”

all of these players that actually impact how we build out a better future. And the headline is friction. It just created more and more friction. Now, what's happened to him, I think is actually is just as important of a headline, which is nothing. Really. I mean, he lost his job at the pharmacy, but we've talked to many folks, law enforcement. I actually spoke to a couple different folks in Canada from the law enforcement perspective. There's no open investigations into David

Joe. That's his name. And there's thoughts that maybe he's left the country. But that's it. And I think that in and of itself tells its own story. Absolutely. So last question, Deepfake for an obviously isn't going away. So what's the best possible outcome? Each of you, what is your most optimistic case for how it pans out in the next five years and how we get there?

Kerry, you go first and then VS and Lori, you get the last word. Kerry. Okay. Well, it should

probably come as no shock to you that my best case outcome is that the companies that unleashed these harms and profited significantly from them should have to pay and there should be discouragement and compensation for all the victims. And it should be financially penalizing for these companies. So that's would be your best. And is that possible outcome? Do you feel like that's a possible outcome? I mean, we're on that road with litigation. We'll

see how it works. I also, you know, I do condemn the platforms like the app store and Google Plays who for many, many years were offering and selling new-defying apps. And they just kind of were ostriches with their head of the sand pretending like they didn't think these things were harmful. I'm really sick and tired of platforms not doing anything until something becomes actually criminal. You know, they deny that it's harmful until there's an actual criminal law.

“Yes, you know, that's what I would like in place. Some kind of criminal law that says,”

well, look, here are the kinds of behaviors and even intended behaviors or intended outcomes that should be criminalized. Here are the penalties that you will face if you engage in these acts and this needs to be something that's at least in the US nationwide. But more importantly, many of these actors are going to be overseas. And so what we need is not just US law but law and various countries and some broad international agreements, perhaps enforced by an organization

like Interpol, which polices things like child trafficking already. They should be authorized to do and investigate things like this. That's a very good point. That's exactly where it goes. This is a broad. What happens is immediately. It goes a broad. Lori, last word. Look, I agree with what everyone here has said. So I will end with saying, you know, my gripe with Silicon Valley and I have a lot of gripes with Silicon Valley is they're sitting there talking about AGI and what

happens when AI becomes smarter than us and it doesn't matter. It really doesn't matter because

here's what's happening here on Earth. Over here in reality, we have children who are ending

their lives because of deep fake sex stores and people are sleeping on it. My hope is that I don't have to shout into the ether for years and say, this is real harm. This is real harm that we can get better education that we can change culture to understand that when new technology

that don't have the voice and we can make some changes from there. No, it doesn't take to

carry's point oftentimes it takes criminal law. If I'm putting sitting in my seat oftentimes it takes public pressure and investigations, I'd love to get to a world where Silicon Valley maybe has some different folks in it and this isn't the last thought as people are having dinner and talking about AGI after taking a cold plunge. That's it. That's a very good point, but you're going to wait a long time for that. Just a little bit. I've got manifesto. I'm

not a fan of anything right here. All right. Sure. Why not? Why not? Anyway, I really appreciate

“it all three of you. It's a really important talk and we're going to keep coming back to it”

again and again and hopefully see something happen by raising awareness of it for sure. Thank you so much all three of you. Thank you, Cara. Pleasure to be here. Thank you so much. Thanks. We've reached out to XAI and X for comment. They did not respond. Today's show was produced by Michelle Aloy, Catherine Millsop, Madeline LaPlante, Dooby, and Kaylin Lynch. Special thanks to Lisa Soap and Nico Robbins and Julia Sharp LeVine.

Our engineers are Fernando Aruda and Rick Kwan and our theme music is by Tracodemix.

“Nishot Kroa is Vox Media's executive producer of podcast. Go wherever you listen to podcast”

search for on with CaraSwisher and hit follow. Thanks for listening to on with CaraSwisher from podium media, New York Magazine, the Vox Media podcast network and us will be back on Monday with more. Support for the show comes from KPMG. In any organization, disruption is inevitable, but struggling through it doesn't have to be. The KPMG adaptability index is your blueprint for building capabilities to handle what comes next. It uses real data to look at how your

culture, strategy and partnerships all work together to help your business thrive.

“Stop reacting and start adapting. Visit kpmg.com/us/adaptability to explore the”

adaptability index and post surveys today. Support for the show comes from fetch pet insurance. Do you have a pet? Every six seconds,

a pet owner in the US gets hit with a vet bill of over a thousand dollars. And it's almost always

an unwelcome surprise. That's where fetch pet insurance comes in. fetch is the most complete pet insurance. Get paid back up to 90% of that bills. You can use any vet in the US and Canada. All that's are in network. Go to fetchpet.com/safe right now for your free quote. That's fetchpet.com/safe. We all have to drink water and staying hydrated is one of the simplest ways to feel your best. But come on, how is your relationship with water really? Are you getting enough from it? Is

it satisfying you? Or is drinking it becoming a chore? Here's a tip to spice things up. Hint water. With delectable flavors like watermelon, Georgia Peach, even lemon zest freeze, hint is water that will make you desire water. No sugar, no sweeteners, no calories. It just goes to show. Sometimes the right hydration partner changes everything. Try hint. Available at drink hint.com and in stores nationwide.