Siliconsciousness: The Epidemic in Deepfake Financial Scams and Potential Cures

The video is also used with Shopify. It can be made to help a real help. Start your tests today for only one of your promo. On shopify.de/record.

The world's best world world is in the world's best world. The world's best world is in the world's best world. The legendary checkout of Shopify, is just the shop of your website, and over everything. That's the music for your ears. The video is also used with Shopify. It can help a real help. Start your tests today for only one of your promo. On shopify.de/record.

“To stay up to date on all the news that you need to know, there's no better place than right here on the DSR network.”

And there's no better way to enjoy the DSR network than by becoming a member.

Members enjoying ad-free listening experience, access to our discord community, exclusive content, early episode access, and more.

Use code DSR26 for 25% off discount on sign up at the DSR network.com. That's code DSR26 at the DSR network.com/by. Thank you, and enjoy the show. >> Welcome to Silicon Consciousness. The DSR network podcast focusing on the artificial intelligence revolution, politics, and policy. [MUSIC]

“>> Hello, and welcome to DSR's Silicon Consciousness. I'm David Rothkuff, your host, and this week.”

As everybody, we'll talk about an important issue in the world of AI and related technologies. This week, we're going to look at a topic of recent study called Deepfake Financial Fraud with its authors. Ania Schiffrin is a senior lecturer in the discipline of international and public affairs and faculty co-director of the technology, policy, and innovation concentration at Columbia's School of International Public Affairs. And Alice Marwick is the director of research at data and society welcome both of you.

>> Thanks for having us. >> So let's start, where we should start, right? Let's start with the premise.

“When you talk about Deepfake Financial Fraud, what do you mean and why should we care?”

>> Thanks, David. Yeah, I think that the problem has grown enormously in the last few years. And even since we started writing, and I feel like probably every single person listening right now knows exactly what we're talking about, which is people are getting scammed out of money and savings constantly. And one of the ways that this is happening is that because of AI and image generation, it's become really, really easy to make deepfakes that trick people into parting with their money.

And everywhere I go, and I'm sure this is the same with both of you. I get in a conversation with someone who has been stammed or knows someone who's been stammed. A few examples, my mom who's 89, she gets these messages on a Facebook of some handsome guy saying, "Hi, I'm a vet. I'm lonely. Would you like to meet?" In fact, that's a fraudster. There are superstar musicians, golf players, athletes who are getting impersonated all over the world.

But even people you would never think, like economics, professors in Malaysia, or mining executives in Australia.

And even though it sounds shocking, all over the world, people are handing over money to these fraudsters. So Alice has been working on this for a while, and I got interested in it for a whole lot of reasons. I heard Craig Silverman speak, and I met somebody who's been faked. And so with my students and co-authors, we started looking at what are countries actually doing about this?

And so that's really what we've been trying to do. And the paper just came out a few days ago.

So we're really excited to be here talking about it. Thank you. Well, thank you. Alice, you feel in where you will, but I'm interested in why anybody would do a deep fake of a Malaysian economics professor.

“Well, I can't speak specifically to that. I'm sure on you can, but the thing about AI is it allows people to personalize scams, right?”

So there's some scams that have a pretty broad audience.

So there's a deep fake video of Elon Musk saying, "Invest in my cryptocurrency, you'll make millions of dollars." And that's a fake video. Elon had nothing to do with it, and people are sending their hard earned money overseas to some scam artists. But then there's other situations where people might be scammed by a deep fake that pretends to be their boss, or even somebody in their family, or somebody they're close with,

or somebody they respect to sort of a niche famous person, right? Like not as famous as Elon Musk.

And what AI does is it allows scammers to do this much more cheaply, and to a much broader audience. So they can sort of say, we want to target this kind of person, and they can design a deep fake scheme that will appeal to those types of people, like golf fans, or, I guess, fans of economics professors. Well, to talk about a niche audience.

“What you get at though is I think one of the most important issues here for people who are just grappling with this idea to understand.”

And that is, you know, typically when we hear the words deep fake, we think that's a video of a person or a video of a famous person, and we don't necessarily think deep fake could be a video. It could be a picture, it could be a document, it could be an audio file, and it can all be wrapped in AI in such a way that it can go and find out that you're a golf fan. And, and tailor this in such a way that it's hard for you not to think that it's real. And I say this as somebody who has a relative who got a phone call that sounded pretty legit about their taxes, and that they had to send in several thousand dollars.

And, um, sent in the money, and then was talking to somebody else and, you know, hit their forehead with the, the butt of their palm and went, "Oh, I, I've been scammed." Well, one thing that we've really learned, what it's so interesting, because while we were doing the research and talking to people all over the world, everybody had their scam story, including people that are scam experts and AI experts, people that are really being in our world. And, one, you know, one of the things they said, well, a few things, one is, as you point out, the Alice points out, the personalization is really key. So, I have a friend who got a tent, which he thought was an email from her boss at midnight saying, "Could you go buy someone so a gift certificate?"

“And my friend thought, "Oh, my boss is being so annoying, and almost did it until she realized." And then I did another friend who had her coat on, leaving the office, and she just sort of handed over the money, because she was in a rush.”

And we, you know, somebody of very famous AI experts also had, like, missed a plane, and there had been a day to breed, so the message was super personalized, and it got him as well. And I think afterwards we'll probably talk a little bit about, you know, how these things are made and marketed and what we can do about it. But I'm sure Alice is something to add as well on this personalization point. Yeah, I mean, I think we often think of people being scammed as people who are gullible or people who are uneducated, but if you look at the statistics, people across the socioeconomic spectrum, all ages, all genders, education levels, are just as likely to be scammed.

And one of the reasons for that is that scammers take advantage of emotional upheaval. So they often target people who are vulnerable, like maybe they're moving or they're looking for a job or they're an immigrant, or even just they're going through a divorce or they're lonely, right, they're looking for people to talk to. And what the scammers trying to do is to get people to stop thinking with their logical mind and to let emotions sort of take precedent. So they do things like your friend David, where they're like, oh well, you owe all this money in taxes, and if you don't pay us immediately, you're going to be arrested or the scam text that we all get from the DMV saying that if you don't pay to get your card register, it's going to get towed.

So they're when they're designing scams, they're really doing it with an eye to human psychology and AI tools are really good at personalizing tax, at personalizing photos and videos, at, you know, a scammer can take everything that you've replied to them, put it into an LLM and say, how do I respond to this person, like how do I convince them to invest in my cryptocurrency scheme?

“And so you're not just working against a scammer, you're working against the entire power of an LLM, which has been trained on persuasive texts from all around the world and all over the internet.”

Alice, that is so scary. And it's reminding me of the time I came home in my poor old husband, his computer had been frozen. He got a call saying, you know, we're Microsoft sent us money and thank God he didn't.

And Nusa, one of our student researchers, was saying she's been finding new data last week on how young people, kids, are actually more susceptible.

“And I was just talking to a colleague in Indonesia. I mean, there's like borderline about how video games, you know, they get sort of tricked into buying more flowers and more stuff until, by the end, they've racked up enormous amounts of debt.”

I don't know, David, if it would be maybe a few more questions, I'd love to talk about sort of the whole thing from soup to nuts, and where I feel there are intervention points that we need to do.

But I'm not sure if you were Alice had anything to say before I talk about that.

No, no, let's talk about it from soup to nuts, and let's talk start with who are these people, how are they operating? And ultimately, when we get from soup to nuts, I hope, when we end up with nuts, what we're talking about is, how, not just what people can do to deal with this, but what governments can do to deal with it. Yeah, so as part of our research, we helped convene a meeting in Cualo Lumpur with the Cazano Research Institute, and then another one with check my ads and the Miguel sector for technology media and democracy, and that was held in Ottawa.

So it was really interesting talking to people from all over the world about how they see the problem. So we, you know, China, Taiwan, Malaysia, Indonesia, Australia, Singapore, and one of the, actually the way that Chinese think about this and others too, which I thought was so helpful, was it's sort of a pipeline. So it starts with the people that are generating the scams, and then they're finding ways to market the scams. And unfortunately, we know from excellent reporting and, you know, research that that Facebook is now made made in 2024 made projected 10% of their advertising revenue from scam adverts.

So that's a huge place in the pipeline, and if the ad they use AI to sort of vet the ads, and if the ads look like they're fake, they charge more.

“So that's one place, right, you created, then you have to sell it, and that's often marketing on Facebook.”

And then you have to lure people in, and then you have to exactly what Alice is saying, sort of trick them into parting with their money, and that often means a bank transfer, and then the bank accounts get cut up into little bits. It could mean going to an ATM, I've a friend whose mom could barely send an email, somehow a scam or stay on the phone with her all day long in Connecticut, and got her to go to a crypto ATM and take out crypto in Taiwan or in Brooklyn, people will literally come to your house and take a shoe box of money from you.

And then of course, it goes straight to the kingpins, so it could be one of these Chinese crime families, the organized crime corruption reporting project didn't incredible investigative series, where they found the scammers in the compounds were calling people from Georgia, but the kingpins were actually all in Israel. So the money goes to the kingpins, and then they have to wander it, and that's where you get into your classic, you know, casino's bank accounts, the UK and the US just put sanctions on the prince group, bank accounts, I think, in Singapore, China arrested some of these guys.

So there's points along the way, and countries are doing different things, I'll say two things, and I'll let Alice come in again, but some places like Malaysia, Mexico are trying to limit the number of SIM cards you can buy, and that's sort of, you know, starting at the very bottom of it, other people are trying to sanction the money laundering aspect of it.

And that is in any given situation, who is the person who could most easily deal with the problem.

“You know, in my family, if somebody was hotter, somebody was called my dad said, whoever can like take a sweater, put a sweater on or take a sweater off, right, the least, the least trouble can solve the problem for, you know, the most people, right.”

So Catherine Charkin and why you is written a paper on liability in the AI age. She says, you know, if there's a product that's defective, Amazon actually is more information than anyone else. They're the ones that are vetting the advertisers, they're the ones that know when something is getting returned. One of the arguments we make in our report is actually in this case, the cheapest cost of order, maybe not forever, but now is actually meta. They're making all the money of these adverts and they should be doing far more to limit it.

So, you know, there's lots of other things the governments are trying and people are doing, but clearly that is one point in this pipeline where intervention, you know, more could be done and should be done.

“Yeah, I want to circle back David to your question about sort of who's behind this and what are they doing. So when we think about the impact of AI, we've seen that it's having enormous impacts on industries all over the world, right.”

It's increasing productivity, efficiency, it's changing the way people work, and that is true for criminal organizations as well. We've often thought of scams and hoaxes and fraud as something that's kind of done by a single person. Right, there's all these Netflix documentaries about the scammers and the hoaxers, but in reality, these are criminal enterprises that function like almost like companies. They have compounds which are basically call centers where people are human traffic and they are on the phone or on like on a phone, you know, texting with people being on Facebook all day long.

They're reporting to be all kinds of different folks and the those people I think are some of the most victimized in the entire pipeline that on you describe because they are often scammed into working for these criminal syndicates through job search scams.

Right, so when we think about the influence of AI on the scam ecosystem, it's not just about these kind of like novel things like a deep fake or a voice clone.

Also just about automation at scale and the fact that scams can now reach out to so many more people to the point where all three of us on this call have multiple stories of people we know who've been scammed, right? This is becoming an everyday problem for people all over the world. You've been doing this for the whole time, you're just talking about the story and then you're just talking about the story.

You're just talking about the story that you're just talking about. You're just talking about the story that you're just talking about.

You're just talking about the story that you're just talking about. You're just talking about the story that you're just talking about. This podcast is underwritten and part by the U.S. Embassy of the United Arab Emirates. Its editorial content is completely independent and the views expressed are exclusively those of participating experts. It is presented live without editing. For further information about the UAE's efforts in the areas of artificial intelligence and technology, go to the website of the Embassy at www.ue-emBC.org and search for UAE-US Tech Cooperation.

“We thank them for their support. We thank everybody who's supporting this podcast for their support and we look forward to it developing and growing over time because the issue is so important.”

Although I'm from New Jersey and when somebody says the word crypto to me, that translates in my brain to scam automatically.

And sort of the regulatory loopholes and the stakes and gaps and boys. Clearly, if met as making a ton of money off of something that you can identify as a scam, they could identify as a scam.

That's a big gap, but you know, we talk about AI every week and have for several years now on this podcast, and we talk about regulation everywhere, and frankly, most senior government officials don't understand AI. These areas change very rapidly, and then their big forces of food that are saying right at the moment, don't regulate. In fact, if you are, you know, talking about regulating crypto, you can end up with millions and millions of dollars being put it to a candidate opposing you by the crypto industry.

So maybe you could talk on you and start and then Alice Bob was, you know, just how bad off we are right now in terms of the regulatory landscape and the ignorance of senior officials and regulators.

Well, I'm a great believer that you don't have to be an expert in something in order to regulate it. You know, we don't understand necessarily like how our planes get up in the air, but we're still able to regulate them right. And so I think what a lot of the regulators have done, particularly in Europe is to sort of say, you know, what there's a duty of care, and if you're a big company, you know, whether you're a bank or whether you're a social media platform, you have a responsibility to know what the risks are and to mitigate against them.

“So that that I think is sort of the, you know, the minimum, even if you're not a huge deep fake AI expert, and this of course is what Europe has tried to do, and the UK has tried to do with things like, you know, miss and disinformation.”

And Alice is written a lot about this whole concept of duty of care, so she'll be able to talk about it. We just got back from presenting the report in Brussels and empowers. We had seven meetings over two days with different parts of the European Union, the European Parliament, the French regulator.

And it's very clear that they're super worried about this and they're, they were asking a lot of questions about what other countries have done.

So I think that even if in the US we're a little bit stalled right now, and I'm so glad you brought up this whole problem of AI money going into local elections, because it's it's really a terrible distortion of our political process and even my local, you know,

“for it, congressional races is seeing this. So, but so I think, so I think I think that scams may turn out to be one of these areas where we could actually get some agreement across the aisle a bit like kids, right?”

And you've seen that kids safety is something that's sort of mobilized the British government, they mobilized the Australian government, and I think the scams problem, like it also defect pornography. I think it's going to also turn out to be something where governments can act. And as I mentioned, there's a lot of things they could do. I mean, one, one other thing that we've seen smaller jurisdictions are often able to get people more together. So Taiwan has actually done quite a lot on data sharing. Singapore has a multi stakeholder initiative where they're bringing, you know, the banks, the platforms, individuals, consumers, associations together.

So I think just because our country is so horribly paralyzed and gridlocked, I do think around the world responsible policy makers will act. And I go back a lot to what Craig Silverman said, which is it's about building trust. And if you've a situation where people are going to scammed and nothing's being done about it, then they literally don't trust the government and that can lead to, you know, terrible fragmentation.

“Yeah, so we are in this extreme anti-regulatory moment in the United States, or I would argue that we're regulating in favor of the AI industry, right?”

Like Trump is doing a lot of regulation to try to promote the American tech stack and as a sort of tool of American hegemony. But what's interesting about this. Well, I would add just there, by the way, he's actually pro scammer. So it's it goes a little further than that because yeah, I mean, it's definitely it's definitely a grift, right?

But what I wanted to point out was that you don't need to regulate AI to have a material impact on scams. A lot of the use cases for AI in scams are very, very hard to regulate, right? Like translation, a lot of scammers use AI to translate. You can't really say you can't translate this kind of text.

“You can't necessarily regulate text that's super persuasive. If you want to make an Instagram account with a bunch of pictures of the same synthetic beautiful woman, that would also be a really difficult use case to regulate.”

And the only sort of instances we saw of that globally was, you know, people who want labeling requirements on synthetic content, which only works if everyone does it, and even if every major AI company did it, there would still be lots of open models that don't. It's never going to be a good solution to this. So what on you and I were able to do was look at other points of intervention, like social media platforms, like banking intermediaries, like global regulation and cooperation.

These are all places that don't require us to regulate AI companies or AI technology, but they allow us to have material impacts on the way scams operate and how they reach people.

Okay, we only have a few minutes left here. So let me ask you sequentially two different questions and I'll start with you on you. One other element of this that would seem to me to be important that I have heard people have problems with is enforcement, you know, the kind of. To to per before we went on the air, Alice and I were talking about ghost clusters and and and the question here is who you're going to call right. Because you call the local police and they're like, huh, well, we don't do that. You call the FBI and they're the FBI and so you know, I mean, are they going to get, how do you, how do you when you're caught in a scam.

So I'm going to go with it and and are enforcement agencies up to far on all this. I can take that first.

We've thought about this a lot because a very small percentage of people report being scammed.

It's the kind of crime where people who are victims often feel ashamed of it, so they tend to radically under report. And there have been studies showing that even people who do report, they might call their bank. They might call the local police, which as you point it out, are not equipped to do a cyber crime. The FBI will listen to you talk about a fraud case, but they can only go after it if it's in the like hundreds of thousands of dollars. If it's like your average person losing five hundred dollars or a thousand dollars, the FBI can't do anything.

The ARP actually has a great scam hotline. But one of the things that we identified as a problem is there is no centralized place to report scams and it makes it really hard for like. State attorneys generals, for example, to like keep up with what scams are trending in their state or for people to work across jurisdictions or across locations because the scam ecosystem is global. Okay, so we do have a limited amount of time, so I'm going to go to you with a separate question here. And this is probably one that you've gotten most frequently from people you know in your normal life when you tell them what you're doing.

And that is so what do I do about this? How do I immunize myself and against these kind of scams? Absolutely, and I'll just add by the way that Taiwan has a dashboard where they track all of this stuff so it can be done.

“Yeah, so I think the main thing I tell people is number one, if you get a call, remember we have lousy customer service in this world.”

So if you get a call from anybody pretending, you know, saying they're your bank or they're like it's just not conceivable. Nobody cares about customer service anymore. So the audited it's your bank or the IRS or Microsoft calling because they care about you are zero. That's the first thing. So if you get a call from an airline saying, we have a delicious meeting meal waiting for you. You know it's a scam because no airline has it right.

Because nobody has that right. That's the first thing. And then second thing is just don't click on anything really because there's so much that including last couple weeks and when I know I'm going to party invitation. So so that would be the second thing. It's just don't click on anything. And the third thing would be don't send anybody any money.

“And I feel like if you follow those three rules, that will help you a lot. That's what I think.”

By the way, I got one of those party invitations and it looked like a normal party invitation. One of those envelopes and it came from somebody I sort of know who has never invited me to a party.

And I wrote the guy note and I said, did you just invite me to a party and he said, no, I got hacked and it got sent out. Don't click on it.

“So, you know, there are these little things. Like if somebody you don't know is invited you to a party.”

Don't click on it to RSVP.

Say, you can send them an email or you can call them to verify what's going on. Right.

Well, look, this has been terrific. I encourage people to go and take a look at your work.

“And you know, it's kind of things going to come up again. And I hope that the two of you will come back and join us again for more conversation on this important subject for now.”

Thank you both. And thank you everybody for listening and we've got more silicon consciousness every week. We've got more of our AI energy and climate podcasts every week. And of course, all of our podcasts on politics, national security foreign policy.

“Are there for you and you can subscribe on YouTube where you can subscribe to the podcast on the DSR network.”

So, until next time, thanks everybody. Bye-bye. This was Siliconjustness, a production of the DSR network.