Lawfare Daily: Consent in the Age of AI

Running isn't enough anymore. With PWC's managed services, your operations don't just run, they evolve continuously,

powered by AI embedded directly into your workflows. So instead of maintaining yesterday's model, you're building tomorrow's at 40. PWC's managed services, we run your operations with tech and talent, so you can run faster, scale smarter, lead stronger.

And what do you do with that?

“The new Orde ball original HURBOO and what do you do with it?”

Let's talk about Karolina Herford. Now let's turn on the Orde ball. Is this media fake? Is different than is this media of me?

And that's where I don't think the existing kind of report and attestation piece solves

the kind of search problem or prevention of the content being made. It's the Loffer podcast. I'm Renee Doresto, Senior Editor at Loffer, and I'm joined by Kate Kahnick also, Senior Editor at Loffer. And our guest today is Alyssa Redmiles, an assistant professor of computer science at Georgetown University.

“So I think there's a lot of attention they would need to be paid to like what qualifies something as a sufficient seafood”

and how do we set people's expectations appropriately for how much that's going to protect them individually versus kind of mean something about mitigation in a specific product. Today we're talking about the people who create AI-generated sexual content and whether prominent technical proposals can actually prevent AI systems from generating exploitative content. So I want if you might want to start by telling us a little bit about the research that you've been doing. I know that most conversations about AI-generated imagery begin with a focus on detection,

but you know you've been arguing that that really misses some central facts. I'd love to hear you just start by telling the listeners how you think about this problem. So when it comes to AI-generation of non-consensual intimate imagery, sometimes called revenge porn, one of the main harms is on the use of someone's likeness without their consent to create sexual content of them. In particular as a computer scientist, I was looking at different kind of claims that companies were starting to make, for example, like Dali has a claim about using advanced techniques to prevent photorealistic generations of real people,

which is a website for hosting different kind of model objects that got found to have a lot that makes specific people, and started making some claims that you could opt into preventing models from being shared that could generate your likeness.

“And as a computer scientist that made me wonder, can we actually do that in practice and what are the flaws that are going to pop up if we try to automate that type of production?”

We actually just saw this from meta. We saw some interesting decisions about not only opting in, but in fact encouraging. I got a person notification from meta AI, telling me I could go create content of people just by adding their Instagram handle, and then it would go and it would generate content using the person's public Instagram account as the base layer, and it was a very interesting, it was a very interesting product decision I thought in light of the potential for abuse and curious what you thought about that call.

Yeah, I share our concerns about choice, particularly given that we've seen it with grock. There's been a lot of instances of a new notification, or at least sexualization of individuals by using a very similar importance, where you like at the individuals account, or it's just kind of a reply to content that they've shared, and so when I saw the meta release, I was kind of immediately thinking about that kind of situation. And was concerned that people may not expect that their content can be modified, and I think this goes beyond the notification, even for various job or reputation concerns, people may have, and actually in particular we're seeing people being asked to kind of set social media to public, right for various kind of immigration or border reasons, and so that may put people in a really challenging position.

The other thing I would say is, you know, even after, say, X, AI has mentioned trying to implement new technical controls, we're still seeing many cases of notification, and in a report that we'll have coming out soon with the Center for Democracy and Technology, we took a look at all of the kind of technical ways that foundation model providers are trying to prevent this generation, and we really don't have reliable means of doing that right now.

One of the things that I have followed in your work over the years is that you write a lot about how, sometimes abuse isn't necessarily focused on the substance of the content, but whether or not the person consented to its creation or distribution, right? And that might mean that there is a new image that the person actively made themselves and chose to upload voluntarily, or a person who is, as you know, not nude, but put into, you know, a potentially dicey image situation, and that did not consent to that, and doesn't have the same access to having it addressed, because the image generator will infact generate it.

“So can you tell us a little bit about how you see the consent problem and its differences from the content classification problem?”

I think the idea with consent that we've seen even before use of AI for non-consensual intimate imagery was so large, and we had done work with, as you said, victims who had not taken content, but maybe had it made of them using secret cameras, other approaches, as well as folks who had taken content of themselves either for recreation or for commercial purposes. And what we saw was for folks who are sharing this kind of content commercially. Some people viewed the resharing of their content or now the editing of that content as like a commercial threat, like theft, like basically you've taken something that I make commercially, and you're trying to make money off of it yourself.

Whereas others saw it as both theft and a kind of sexual harm, privacy harm, victimization in the same way that folks who were not commercial actors would have felt about their content, being disseminated or themselves being depicted. And I think that's one of the reasons why the idea of likeness or being identifiable as a real person and whether or not that person got the opportunity to consent is so compelling to me in this space. It's also compelling to me given kind of how we think about pornography in the past and people being able to provide consent forms and that being kind of a huge distinction or point of process at least in the U.S.

So I think for both of those reasons, they resonate with me.

So just to be clear, NCI and why it's not called revenge porn is because the idea any more, like that's kind of like it's like a, you know, not a politically correct term among those who kind of study it anymore. And I've always understood that to be kind of true because the idea of pornography is that one does it consentually and inherently like pornography is kind of thought of as consensual whereas NCI non consensual image intimate images is essentially like inherently the opposite of that.

“And so it does and also because revenge is not some often we find it in revenge situations, but revenge isn't the only way that like non consensual images show up intimate images show up.”

So I just cut is that correct is that kind of the definitions that we're operating under.

Yeah, absolutely. Yeah, so I want to kind of pull out a string here, which I think is super interesting from a legal perspective, which is this question of focusing on consent. A legal perspective, I think there is something really interesting about kind of this model that you bring up.

One is kind of a theft of property, so to speak, like a theft of something that you can commercialize or something like that like a content kind of theft.

And then there is, you know, this privacy based regime, so those are kind of like end up in different buckets and the law I would say. But in both, there is like an evidence issue, right, trying to show or demonstrate that consent has been given either explicitly or implicitly and kind of to track that. One of the ways I know platforms have dealt with this in the non consensual image context is to just believe the person who says this is my image. And I didn't consent to this, and so please take it down, is that as feasible going forward, did you do any kind of research on that, do you see just kind of a.

I believe it, like in terms of just how platform should react to this, like is that actually going to be an actively useful framework going forward. Yeah, that's a great question, so I think there are a few issues here, so one is once the content is created what we do about it, right, and so if that content is shared on a platform that has a reporting stream and people are able to report.

“And then certainly kind of attestation is one approach, I think the one concern we sometimes have with attestation is that there have been cases of like widespread sort of activist based.”

Reporting of say sex workers content for take down and so that becomes like a question of how you balance that, but typically there are other signals when that's happening where you may be able to kind of balance things out.

I think the problem though becomes that this content often spreads to platforms that are specifically for hosting non-consensual imagery or are just not interested in doing.

And take down, for example, thinking places like fortune or a bit more hidden. The other issue that can come up is people discovering this content has been made in the first place, and that's a big issue for the AI generation of this kind of imagery is how are you going to go into text that you're depicted.

“And that kind of comes back to this question of like can computational approaches do detection of you or any real person in generated media in a reliable way.”

And that's a different question than the one we usually ask which Renee brought up which is like is this media fake is different than is this media of me and that's where I don't think the existing kind of report and attestation piece solves the kind of search problem or prevention of the content being made. So technically speaking then when we think about the technology for detecting consent is that even it sounds even weird to say that or how do you credential an image is in a way that is like machine readable we can do this of course right this is this is come up a bit.

But we were seeing quite a lot of a lot of platforms struggling with labeling regimes with detection regimes and there's a lot of different things that they're being asked to build simultaneously. How should we think about essentially credentialing content as concentrated imagery that is not NCI.

“Great question. So I think that this in those kind of two components the first is how do I recognize that this image that has been generated or posted is an image of you and how do I do that in a way that is privacy preserving preferably.”

And then the second piece is how do I know whether that is an image that you consented to being created.

So for the first piece of like is this an image of you. One of the first things that may come to mind for computer scientists is facial recognition, which are right the technologies when you go through TSA now they often want you to stand in front of a camera. They're using what's called one to end facial recognition where they're searching for you in their database of faces and essentially what they're doing is they're comparing the image of your face that they just took doing a similarity score check against the other images in the database and presumably your face or name should come up as the most similar one.

When it comes to generated images a few things change the first thing is that the underlying distribution of pixels so all the little points that makes up the picture.

That distribution is different in a generated image than in a photograph and there actually has not been pretty much any research on how well facial recognition algorithms will identify individuals in generated photos. And the other thing that comes up is that facial recognition was originally created for these kinds of border type cases where you want to be like perfectly sure this is the person that you think it is when it comes to something like NCI I think the concept is a bit fuzzier right in terms of like.

Can a reasonable person recognize the individual the image and so how well something like facial recognition will map to human perception is an open question especially because it depends like. Is this you think it's you in the photo is it someone who knows you well in real life things to you in the photo is it's like your online fans think it's you all of those can add noise to this kind of machine process.

“So that's one aspect I think in terms of like attaching consent credentials to an image certainly there are different approaches like we've seen for AI labeling to do kind of cryptographic watermarking.”

That attaches content to an image and all of that relies on the platform where the content is being posted keeping that metadata and respecting it and so that is like an open question that one might have after the generation face. So I think this is really fascinating that you did all of this kind of research between. The various different types of ways that we move from something to the TSA way you say one to and like I think that like you know most people are so creeped out by things like TSA and it's incredible quick speed and this critical accuracy mostly because they think that like.

The TSA is running your face pass a database of all known human beings and the planet and it's not it's like the flight list like the manifest right like it's not in any way. That like kind of which I think is actually kind of a super important point to make and so I think that there is this really fascinating delta between our ability scientifically and through qualitative.

So like authenticity in both one sense is so crucial to certain types of like facial recognition and super types of like of AI generation stuff but it's actually not on point.

“To like kind of this problem with non-consensual intimate images because the damage is really just the idea that it's you right that's a reputational damage is a normative damage.”

It's kind of a breaking like you know being shamed or kind of you know embarrassed by something that you're caught undoing or that is revealed about you. How do you kind of capture that sociological component of this I mean you're a computer scientist so I'm just really curious you live in the mathematical proof the authenticity proofs of these things. So how does one measure how if you know if you're designing a study how does one measure that other component.

So one of the things that we do here is we often try to look at fields that measure human perception so take psychology for example.

And there's a well study task called the familiar facial recognition tasks that's used to diagnose things like problegia.

“So it's been studied quite a lot and basically the task is I'm going to show you an image.”

I'm going to ask you what is the name of the person in the image and you're going to reply either with the name or with identifying details like oh this is the person who played Wolverine. And so that's an example of a task that we use to see like can someone recognize and then we control for all sorts of confounds things like are you have the same gender and ethnicity as the person who we're asking you about are you familiar with them in real life or not, et cetera, et cetera. So we do that kind of work to capture human perception and compare it with computational metrics.

And that doesn't get at the deeper part of what you're talking about which is things like even if this is a cartoon of me is that something that would cause harm. And how do we think about that from like a recognition standpoint because certainly I can recognize a cartoon of Albert Einstein even if it's not photorealistic.

And so in those cases we do one of two things one is we create surveys that we call them yet surveys so we basically create a scenario for someone to imagine themselves in.

So for example we would say imagine that a stranger created a video showing you as a cartoon having sex. Would you find that acceptable or not acceptable and when we have like a rating scale right and serving methodologies like these kinds of scenarios because they kind of approximate people's feelings they not precisely real but they're they're closer. That's one option. The other option is we interview those who run like support organizations for victims survivors or work with them to speak with victims survivors themselves to understand case studies that are a bit more nuanced.

Like cases where there's cartoons cases where things are not quite as clear cut as maybe we think about in a kind of like photorealistic notification case. You've also done a lot of work looking at the motivations of the creators you're really fascinating paper where you talked about how. How people who create these images think about their role and what they're doing particularly in cases where they make them for themselves and don't share them and you've a lot of really interesting nuance dynamics to your thinking about the creation process.

I wonder if you could share a little bit about that with the audience. Absolutely and so we started studying communities of folks who use AI to create sexual content because we were curious the extent to which those communities were creating norms that sort of separated out. Non-consensual creation. We are also curious how they thought about issues like accidentally creating NCI for example if a particular model has been really heavily trained on a particular celebrity individual it's possible. You would write sort of a generic prompt and get back someone who looks a whole lot like a celebrity who you know and so we started going into these online communities some of which have hundreds of thousands of members.

“How are people getting into this? What are they making? Is it abusive or not? How are they doing governments? What are their norms?”

And that helps us both see the type of content being created. It helps us understand what are the pipelines for creation. Because I think we all talk a lot about like a new defecation website. I upload an image. I get back one that's undressed. But there's a whole wide world out there of folks using what we call openweight models things that have been uploaded on hugging face elsewhere and they are either modifying them or using them just as they are to create their own content either locally or in the cloud.

But what we actually find in these communities is actually serving as like upscaling centers so they're kind of training each other if like a model is too big to run on someone's computer then they're going to like compress it down so that they can run it.

“And in fact one of our participants said you know I'm actually not interested in making sexual content. I have this other content I want to make but the most active technical support I could find was in these communities for making sexual content.”

And of course not all of that content is abusive but of the 28 people we did interview across a couple of different communities three of them talked to us about making non-consensual intimate imagery and several other people talked about getting requests for making it that they turned down. And the moderators who we discussed with to greater or lesser degrees were trying to moderate this type of behavior but they're actually struggling a bit because people come to these communities for kind of an unsensored place for sexual expression something they didn't feel like was easy to find places to talk about sexual content creation or preferences.

And so when there are kind of rules put in place sometimes the moderators get pushed back that's like excuse me you're not supposed to kind of yuck my yam don't tell me what to do.

“And so they actually said that they found two things useful to justify why they had rules.”

The first was the terms of service for the platform that they were hosting the community on so the terms of service have some mention of non-consensual intimate imagery and because people so want to be in the community. If they bring out hey the community could go down if you share this or if you take these requests that really helped.

And the second was the new take it down act the idea that there might be legal consequences for sharing.

Really like motivated the moderators and was also a good justification to the community. That's interesting that you bring up these notions of you know perceptions of censorship versus permissiveness within the community. I know there's also an emerging legal debate over whether generative AI outputs receive first amendment protection either through the rights of users to see content or the rights of developers to.

You know build models and you know the sort of expressiveness of code is a long running debate.

But you know this is obviously unsettled law we're starting to see it play out in a few different cases there was the Garcia versus character technologies one.

“But right now I think there's an interesting question I am not a constitutional lawyer cake and like jumping in and pick this up but one thing that I think about a lot is.”

Does that conversation leave out the person who is being created when we talk about the prompters expressive interest the model companies editorial or design interest where does that leave the person whose likeness is being rendered without consent. Yeah I guess from my side this is something I think about a lot because then computer science the main one in the main privacy threats that people address with. Computer science research is something called membership inference and so the idea of this is like there are some images used in the training data can I attack the model and recreate the exact image from the training data or can I extract it.

And this is an important problem for various reasons but I often talk to colleagues about my frustration that is not the whole problem because my ability to reconstruct the exact source photo is maybe not my concern when it's a harm around my likeness the issue is can you reconstruct an identifiable version of me. And one of the challenges we've had is with membership inference you can define this mathematically because like that training data object existed at some point so we can write some math about what we're trying to do.

With the likeness we need some sort of mathematical definition of like what is like this and that's a much more as we said talked about a human perception kind of thing like can we achieve that I'm not terribly convinced we can and yet it's an important problem to work on to evaluate for etc so that's where I see this coming up. Okay, so I love this question. I think that this is where kind of a list of amazing research and kind of a lot of the word that you do Renee and some of the stuff that we've talked about like kind of offline and off the podcast about deepfakes and the implications and what the legal answers to Jenny is going to be.

I think that this is kind of where like this is going to maybe be the four words decided because I think that these are such obvious places where vulnerable individuals can really be exploited but also it is the place where people who have the most economic stakes can be exploited and the combination of those things motivates people.

Yeah, so I don't want to kind of go all the way back to kind of, you know, to bore everyone in this pod who did not turn in for cake on x history of IP law and internet history, but this is essentially what you saw in kind of in my, in my version of events that this is what you saw with the early internet that we're kind of speed running through as an issue legal issue now for Jenny, which is that you have property based rights being the most enforceable and the most economically viable rights that you can enforce.

You can enforce and courts and so you see them at the four and you see huge interest groups going to bat and taking down the technologies or trying to take out the technologies to do this and so obviously I'm thinking of because this was like happening when I was growing up in a teenager and so I was like on both the side of like this. This group of lawsuits ending right as I got to law school and being a person who like downloaded illegally tons of stuff from Napster obviously like the Napster because I appeared appear sharing kind of issues where you had huge interest groups like the RIA and the motion pictures association of America kind of decided to bring suit because their their actual content was getting ripped off.

That is a solely copyright kind of I like squarely an IP type of right right and that was the the question there for a lot of people was like do we change how we distribute IP rights and what we think of as IP and how we do this in the age of the internet. That's a little different when you have a generative AI which is both mathematically from like the authenticity point that like Elissa was talking about form ethnically new material that happens to have a likeness like it is pixel by pixel it completely different image than a photo of Renee you know if you had a generated image of Renee that looked exactly like Renee looks right now in her Georgetown office like you know and then you had this image of Renee.

“We asked the the machine to produce whatever model we wanted to use and if it was like almost one to one they would still be different images from a from a from it from like I guess like a granular mathematical perspective.”

But and this is kind of what I think is so interesting I was getting it before phenomenologically the picture plays the same role so sociologically it pays the same role it doesn't actually matter and so the effects of the picture the harm that it creates right doesn't end up totally matter and this is I think the thrust of essentially what the debate is going to be. What do we end up going down and I mean when we say like the law does the law end up going down this really particularized route of like mathematical authenticity or is it going to go to kind of done this humanized route of like how it categorizes the harm and like whether or not the generation of that harm exists through AI or something else like we are going to recognize this harm.

And that has different first amendment balances so if you tend to recognize generative AI as like where you were are proponent of recognizing Jenny eyes having its own first amendment rights because it is. Generating something completely new and technically authentically different you're just going to be able to block all a lot up not all of but potentially block a lot of the regulation and potentially litigation regulation and litigation that is going to kind of be filed for like against.

“Against some of these models and companies that are running the models for the outputs that they produce that are exactly the type of output that all this is work is kind of talking about.”

If you decide to take a more kind of harm based approach a more dignitary privacy based approach where it doesn't really matter like where the image comes from just that it exists and it can be traced to the generation from like either a model or a human.

Then you have like a whole different kettle of fish and kind of a whole different set of legal solutions that you're going to put in place there.

So I don't want to kind of take this over with a legal discussion but I do think that this ends up being kind of the pay out of a lot of Alyssa's work if we could you go to like put it to policy.

“And I think it's super interesting just because it reveals the limits of like what authenticity gets us and it reveals the power of things like like this right.”

Likeness being something that no one has been able to perfectly quantify or understand in either cognitive neuroscience or IP or whatever for years.

I mean like literally you know how trademark likeness is decided is through survey. It's through the exact type of research and kind of like qualitative and quantitative stuff that like Alyssa does or like you know social scientists do and so. And I just kind of I like bringing this up just because I think it it gives us kind of also context for like some of these ideas are new and new at scale and some of them are like taking a new twist on a really old discussion and a really classic discussion that we're kind of just getting to see a new balance of.

With PWC's managed services your operations don't just run they evolve continuously powered by AI embedded directly into your workflows.

So instead of maintaining yesterday's model you're building tomorrow's advantage. PWC's managed services we run your operations with tech and talent so you can run faster scale smarter lead stronger.

“I think this question of what should the regulated act be I'm kind of curious to hear your thoughts on this Alyssa is it is it the creation is it the distribution is it the implied threat maybe of.”

Putting a person into a very particular type of sexual scenario is it.

But I think from the ecosystem what where where do you see the the lines around policy on this front or maybe regulatory laws better term than policy for this one. So in this phase I'm often kind of torn between my background in computer security and focus on kind of anti censorship protection of people's communications.

And how we can best address harm so I think certainly regulation that addresses sort of the pipeline that funds this would be most effective if.

And in a way that is careful about addressing non-consensual intimate imagery specifically and that sexual content in general and I think that's all is kind of attention at least I've seen from doing work on the security side in this space is just. And then companies say react to a regulation by going really broad in their prohibitions sometimes we find that just kind of pushes everything to the corners of the internet where we have the least ability to kind of take control. And I just think about it in terms of a balance there and because this is dual use technology particularly that used to kind of create things I think one of the biggest areas for computer scientists to do research is had a way identify when something sort of purpose built for creation of non-consensual intimate imagery whether that's a website with very clear kind of keywords and functionality or a model that's very clearly kind of trained or fine tune on particular individuals and there.

“And there are folks doing research in both of those directions I think addressing those tools and both who owns and profits from them as well as who uses them is important.”

The other thing that I think about maybe more so from corporate responsibility I think I'll defer to you all on regulation is kind of we have seen the use of mainstream platforms like Google, Facebook, etc. as single sign on providers so you know you click and then you can easily sign into the platform for various notifications services and when that happens it gives this kind of like. Acceptability stamp to the platform even if it's not the intent of the single sign on providers. Same when we see those platforms like running ads or having applications and their app stores that can do this or are dedicated that way and so I think there's a lot of kind of corporate governance points in terms of just like keeping an eye on where your brand is popping up and what you're promoting.

We've seen lip service for doing that. We've also seen some failures after the lip service and I think a lot of those failures are because this is a dual use kind of thing.

“You have to really like test the actual application like you need to we usually like generate an image with a face covered.”

We tried to do some double checks but it's not identifiable and we use that to test what the system can do. And I think that often that's kind of not what's happening just like the form is being read by whoever is reading the form or whatever is reading the form and that's where we've seen a lot of failures. So I think whether that's a regulation thing that gets addressed or just sort of a corporate responsibility thing that would be important. Yeah, so I want to actually kind of cut in because I know Renee might have thoughts on the same question. And she's also done a ton of work, Renee you've also done like so much work on kind of how all these different systems are running kind of capturing the idea of how they each responded different types of prompts.

How grock is responding to prompts, how meta responding to these same types of prompts. I'm curious how your research you would describe it as different or the same as some of the stuff that Alyssa's done and also if it has pointed you towards a different policy kind of suggestions because I do think that Alyssa's point, which is like it kind of depends on like what it is exactly that the model or the actor or the whatever is doing like it is one thing if it is like.

And so kind of I'm just like wondering like both of you have had kind of these qualitative conversations and interviews with people and done a lot of the qualitative experiment to work yourself. So anyways, I just kind of will start with like every nay I'd love to hear that.

“Yeah, it's a real big challenge. I used to think that corporate responsibility was was something where you would just never see this from large foundation models, including grock because they would just prevent it.”

And then we had that debacle with donut glaze right and on on Elon's on Elon's platform where for I think probably most of the country was aware of what happened without but just to restate in case they're not it was basically that people began to realize that spicy grock would generate all sorts of images of people in very compromising situations and with x it's it's interesting because you could just ask grock to generate the image right under the person. So this was an extraordinary vector for harassment and many of those images got millions and millions of views actually sank clear and influencer who also had a child with Elon Musk is currently suing him.

“I don't think we have really seen that case move very much. I haven't seen any updates on it in a while, but you know she's sued because of what happened. I think he's fighting about venue at the moment.”

And this is where this beginning to realize that corporate responsibility isn't going to hold. You just see these these rather extraordinary things were like morals go out the window and I just want to say for for listeners that the donut glaze thing is you would essentially ask it was a way of routing around things were so I just want to say like because I do not think the most people are going to be for that are not very online or going to be familiar with this kind of absolutely horrid. Specifically for the generation of child sexual explicit material that like how bad this was you could instead of asking for a pornographic a specific type of like thing to be all over somebody you could ask for donut glaze to be all over someone which was the approximate look and style of like what that thing would be to create kind of a very explicit image that.

Could be taken into various contexts and just like you know having a child covered in this or anything else. So it was very it was a very kind of huge controversy in the world of people who follow this type of thing. Unfortunately, I don't think it actually kind of made the mainstream news at all. Both because it's incredibly explicit and because I don't know people tend to not care about these things of the margins and like Elon Musk has such a high threshold for the think like of all of the absurd things that he does it's like half of them don't even rank anymore.

One was interesting because it did lead to immediate regulatory action in Europe right so you had a lot of immediate calls for you know various types of the regulators in Europe requested specific data from X that sort of pushed it into the news a little bit more than I think it otherwise would have been in part because you began to see. So I think it's a very interesting thing to do is to make sure that I think that it's a great coverage of Elon saying this is censorship you know being asked to provide data on exploitation on his platform was censorship requests that you know people pointing out that maybe this is actually a terrible thing you know.

The censorship actually wrote an article about this for law fair for anybody who wants to track it down. We'll put it in the show notes because then you can kind of see the specifics of this particular story. But I think the challenge on the regulatory front is not wanting to cross bounds into free expression as we've talked about you know where where those lines. You know you mentioned the child exploitation content that's much more where I have done my work not in adults NCI so much as where there is no consent there is no consenting person in that in that exchange right the the child is either being exploited or an image.

That is potentially derived from models that have a perception or inadvertently retrained on that stuff that's where that's where you start to see that happen. So there are some very granular laws that come into play when the child content comes the focus of the thing that are a little bit different than the potential that it might be voluntary content creation and uploading. So I think that's where for me a little bit of my work has been focused much more on the former.

“I think I want to talk about the technological barriers to creation though because this was the other thing right if you have corporate responsibility and the corporate responsibility relates to them putting guardrails on their model.”

There is this dynamic where people will go and do things with open source models and now with cloud code it will very you know just tell you how to set up an open source model in your computer this used to be something where it was a little bit more gated by competence and now anybody can do it.

So I think this is kind of a landscape right and as you said we have so the tip of the iceberg is like the the notification websites and services I go to a website I upload an image I click some buttons I get something back.

“You think those are in some ways the easiest to some degree we take down the website it's pretty clear with the website is now is it going to pop back up a million times yes and so that gets to what underpins that website.”

And what underpins that website as far as we can tell from the research we have done in the communities that I mentioned where some people were building these kind of tools as well as looking at like how to guides that are hosted across the web. Is that they're using as you said an open wheat model they have kind of a system prompt and they've built this such that people can put in whatever they want but they've kind of specialized it to perform well at at notification. And so one of the things we find in the report that I mentioned we'll have coming out is there's no mention of open wheat model providers doing any kind of monitoring of downstream use of their services.

In fact if you look at like stability AI which builds stable diffusion which is one of the open wheat models that's used very heavily on the website called civil AI that I mentioned that has been used for modifications of people.

“You'll see that I believe they reported zero reports to neckmink during a particular one year period during the same period open AI for example had far far more reports.”

Yes and so you know as neckmink themselves sort of says like the number of reports maybe tells you more about how well you're monitoring than it does about how much abuse is happening.

And when researchers have done research they found several different open rate models being used quite prevalent and in ways that are observable because they're kind of publicly on the web.

“So that's kind of one piece of the puzzle the other piece of the puzzle that you're getting at Renee is can we do anything in the model itself like such that the model couldn't be used in this way.”

And what we find is there's like three different approaches you could take at the model level technically right now none of them work very well so one set of approaches people have proposed like what if I remove certain kinds of content from the training data will that prevent. That idea so let's say I take all the images of children out of the training data can the model now not make a child plus sexual content otherwise men is child sexual abuse material if you could do this perfectly. It might help in a closed weight model so in one of what that you can't modify but even if you could do it perfectly it wouldn't help in an open weight because you can just reintroduce the concept and we can't do it perfectly.

So in our own research we find it kind of makes it go from three props worth of difficult to 12 prompts worth of difficult in order to generate just because you still have a few examples.

There is a bunch of research on ways to kind of make this harder do anti tampering kind of prevent fine tuning none of it's ready for deployment we don't see any platforms using it and so most of the protections that we see from foundation model providers are for the products that they build themselves on their foundation models and they're really those like input output filters like let's try to check that your prompt is okay let's try to check that your output is okay. And as we've discussed those are circumventable they're also using AI models to do that filtering and AI models are unreliable so right now it's a very kind of gappy ecosystem.

The last thing I wanted to mention was we've talked about kind of like the you response in terms of regulating and I know there's been this proposed update to the that AI risk act in terms of kind of saying you can't have a model that can do this unless you've deployed certain safeguards and I think one of the things we found in our research is you hear a lot about the safeguards that. foundation model providers have implemented but we find very little transparency information on how well the safeguards are evaluated and even when we interview foundation model providers we find a lot of kind of nebulousness about the kind of threat model that they're trying to address so are they only worried about.

API users, et cetera all of that's a little fuzzy. The answer is your questions we've had about likenesses and sort of what fits the definition of NCI have been fuzzy. So I think there's a lot of attention. They would need to be paid to like what qualifies something as a sufficient safeguard and how do we set people's expectations appropriately for how much that's going to protect them individually versus kind of mean something about mitigation in a specific product. Yeah, so this is actually kind of getting it something that I want to ask both of you which is that like to kind of buck it a little bit the different ways that we've talked about this I didn't tell me if for both of you have this scenes right or the scenes wrong.

Like I feel like we're kind of talking around this idea that there's different ways of attacking this problem is kind of like the post talk kind of legal reaction litigation type of way to kind of create a post talk reaction and kind of a tort based like kind of solution set for an individual that is harmed by this type of stuff. And ideally kind of like an disincentive to do it in the future for other people for tort features who might do this type of thing again to other people. There's obviously we didn't talk about this criminal liability who created around that to a certain extent like the NCI in state kind of laws that have been created over the last few years.

And then there's I think you just said it really well, Alyssa there is like the and and Renee kind of gesture data when she was talking about like the model based versus the open source versus the guardrails versus the regulatory like world of things that we could open up. How do you get it this problem when one information is free you can kind of pass all of this along these models are free like is there it seems like whack a mole and steroids you just can't possibly kind of do this and then there is been you know for better or worse all of this call for transparency around these things which.

Let's out from a security standpoint a lot of the secrets that were like learning and refining to like make these products and models safer. So one of those speaking of the donor glaze kind of thing to go back to that was something that I had heard about a year and a half ago when I was interviewing people at open AI about like their how they're what their red teaming process look like.

“Is that popular knowledge of the time and I remember being like wow that's horrifying and also like oh that's not going to stay a secret just to open AI's red team for a very long time and of course it didn't.”

And so I'm kind of just like very interested from both of you going forward you have things like red teams which for those and familiar are red teams are essentially like.

Realty people who are either in specialty areas like you're either a nuclear development or poison control or you're in like you know some version of you know I don't know some version of harm prevention. And you get asked by the models to essentially run a bunch of bad actor questions the red hat like kind of idea like you put on a bad actor hat and you test the machine with as much stuff as you can throw on it and that is kind of. Carbrales and safety mechanisms are in theory brought in by the company like that is a corporate responsibility idea that we could put into into regulation and force them to have that and be responsive to it.

But that doesn't get rid of the whack a more problem.

And maybe you know keep going so like renew you asked this question of Alyssa but I'm want to ask it if you and then like Alyssa kind of want to see if you just agree with that framework that I presented from your research. And then you think that the where do you think the like the choke point of this is is it all of the above.

“I think there's different voluntary things that the different actors can do immediately red teaming by the way.”

It's not clear from for outsiders what the legal frameworks are right there's certain things that you know you you can't do and reasonably.

You can't do you know when met a makes an announcement about it's thing and you want to know if you can you know if if it will generate for under 18 accounts that's not something that you can really. Do ethically as an outside tester right so these questions around what is the formal. The legal framework for authorized AI safety research which should be for the people within the companies which should be outside of the outside of the companies for maybe.

“Specifically designated individuals I think it's very hard to know how to how to do that because just saying I was read teaming is not a legal defense.”

As far as the you know responsibilities in the chain you know so first of all I don't have very much faith at open source communities are not you know there there are these subsets that are actively there for just this sort of content right that is that is the motivation. So with that then I think you look at questions around distribution I've seen you know law enforcement go after certain types of monetization sites sometimes that's hard because they live outside of the country.

There's the hosting and cloud services have something of a role to play here but then people get very uncomfortable about the idea that they're scanning.

And again this doesn't this doesn't adequately address issues of consent and whether a person has that you know those images up in their drive because they just theirs they made them they're an adult and this is fine. So you do have a whole lot of different component parts here and a lot of it is going to be this patchwork of.

“There will be I think the main stream companies that will adhere to laws and then be good corporate citizens and then you'll have these sort of niche outsiders that are going to continue to to keep that that market active.”

Yeah, no, I do agree Renee and one thing that I might add and I think this relates to the kind of you know should we require red teaming as well as to the continued proliferation is.

Red teaming can vary in quality and it can vary in like the extent to which it covers the threat models right so like we part of why we do the research with perpetrators is so that we can make sure when we're doing tests that were representing. How those people are actually doing things sometimes that's not happening and so I think the question kind of becomes if we find that we can't currently safeguard a particular system or. You know, the odds of harm or you know, x out of y how do we sort of trade off what we want to release or allow kind of under like an EUA risk act type concept versus what we're able to protect like if we're only able to protect non person generations of video.

“That kind of change what we want to release and I don't think I alone have the answer, but it becomes kind of a societal question and so I think the thing I worry about.”

As we're often looking for like a technical bandage where they're just me not be one or there isn't one yet and that becomes kind of a question of like do we try to not let new stuff out is that feasible do we try to put more resources into.

To turn messaging primary prevention doing like social norms shaping those are things that I worry about.

I'd love to maybe close with something like do you think there is one intervention that is technically possible now but largely neglected by companies that you would like to see implemented. What is the implemented more widely is at deterrence messaging so this is something there's like three studies out there looking at how deterrence messaging can which are basically messages that try to say hey. The request you have appears to be for content that might be illegal to share let's say I'm going to take it down act.

I've been used for child sexual abuse material before there's been research on using them for non-consensual intimate energy. The research shows a lot of promise when we're deployed however none of the seven foundation model providers we looked at another the one saying interviewed.

“We're doing deterrence messaging for NCI to had it on the road map but nobody else did and I think there's a lot of research to be done in terms of not just kind of deterrence messages for generation but also like.”

For bystanders how do we think about that within least communities where people are making sexual content that's not abusive how do we think about it. It's kind of social norm shaping efforts can be very promising yeah and I should maybe be clear when I say deterrence messaging I mean when you attempt to generate content on a particular say foundation models website or what have you not necessarily when you're sharing content in encrypted messaging I would not prefer. So I think what this conversation is really made clear is that AI generated sexual abuse is not just a problem of bad content slipping through imperfect filters.

It's really a problem of systems that are built without consent at the center trying to attack it on now legal rules that often intervene after the harm is done and then safety regimes that put an awful lot of burden on the target as opposed to the the creators and times. So I think the you've really brought out a lot of the very nuanced challenging issues here I think maybe some people who are new to the conversation have always thought that oh well those just ban it and then it'll stop. So I appreciate you walking through the nuance with us I think the questions are not just whether a model can generate this material but who is responsible for thinking about potential guardrails what kind of research is necessary to test the risk and then.

Thank you to you both for a great conversation.

“The Law Care podcast is produced by the Law Care Institute if you want to support the show and listen ad free you can become a Law Care material supporter at lawcaremedia.org/support.”

Support us also fit access to special events and other bonus content you don't share anywhere else.

If you enjoy the podcast please rate and move your us wherever you listen it really does help and be sure to check out our other shows including rational security allies the aftermath and escalation.

“For latest Law Care presents podcast series about the war in Ukraine you can also find all of our written work at Lawcaremedia.org.”

The podcast is edited by Jen Potter, our theme song is from Alabama music and as always thanks for listening.

Do your current managed services really help run your operations or are they just running in circles running isn't enough anymore.

“With PWC's managed services your operations don't just run they evolve continuously powered by AI embedded directly into your workflows.”

So instead of maintaining yesterday's model you're building tomorrow's advantage PWC's managed services we run your operations with tech and talent so you can run faster scale smarter lead stronger.