Meta’s Product Director Explains How Its Ad Algorithm Works

I'm Connor McDonald.

I'm here with Cody Ploughker, Connor Rowling,

and we've got a very special guest, Sue Kesh, Miriyala, who's a director, PM at Meta, specifically for the core delivery product. So we go super deep on Andromeda, Jim Latis. We talk about value expression and what that means

for marketers today. Thanks for joining. We'll jump right into it. We are often accused of going really deep to a nerdy degree talking about ad delivery.

But there is a man that exists that would allow us to go even deeper and even earlier as it relates to ad delivery. So we've got Sue Kesh, Miriyala here, the PM director at Meta, leading core delivery products. Thanks for being on.

Excited to be here, 100% so I thought we'd start super high level. We have a bunch of performance marketers and advertisers listening. Why don't we just start very high level

how the current delivery system works?

Everybody's heard bit estimated action rate, ad quality, maybe you could just give us a broad walk through of how the bidding system and delivery system have Meta has worked as advertisers know it over the last few years.

“I think the ad delivery system consists of first,”

we are trying to optimize for what outcomes that people want. So if you really think about it, there's a bid. But that bid consists of a set of value. And what the value is consisting of is what outcomes do you want? What's the probability we think those outcomes are going to have?

And this is the output, if you think about it. And then there's another big component of this. And that consists of how much we think is the advertiser, an individual advertiser's value. And then there's another piece of it,

which is what we think is the user value of it. So fundamentally, like I think we want to maximize the entire ecosystem. So the ecosystem consists of what advertisers want and how much value we think we'll get from showing a particular ad impression to a user.

But there's also a piece of, like, where's the ecosystem? And that is a user quality bid, and we could talk about that. I'm both of those together combined, like, would you talk of the bid? And so that is like the outcome, but how does that happen?

So one is, first you got to tell me the way I think about it

is how do we calculate this? First, and most importantly, you got to tell me who you want to reach. And that's like if you think about it, the broadest audience classification, people in a certain geography, people who made purchases, you can have a semantic or language

around which audience. Most of them, you want to focus on which region, or what are the kind of-- that's targeting. And then after you tell me who you want to reach,

“and you've got to tell me what outcomes do you want?”

And those are optimization goals we could go deep, you guys are. Nerds about that, so we could talk about it. But it's really around what is the value, what do you want? And I think we could talk about conversions. You want one to get a sale, or you could talk about value of a sale,

like the margin. But then those are on the DTC side. There's also different folks that want leads, different folks that want you to install their app, or get value of an app, or there's a whole slew of things

that we kind of want to optimize for. And then if you give me those two, then you got to tell me how much you're going to pay for that, like how much we're willing to value that action, that comes through your bid.

And then, relatedly, it's the budget piece, right? Once you've told me the bid, you've got to tell me how much, because obviously, the more you want something, like you're going to get, there's a limited set of audiences, and we're going to generate, we have to keep going down there.

And then the last piece, I would say, is that quality factors are thingy, which we would calculate. Internally, to make sure that we understand how valuable that actually is. 100%. So before we peel back the layers,

any further, I want to kick it to code quickly, because code you corner and I are, we're on the brand side. So where we are not nearly as familiar with the sort of delivery mechanisms, like I mentioned, we've heard all these terms. But, Cody, what's your understanding,

and with all of your performance marketing background, and all the growth that you helped lead at Jones Road, like what was your thought process around acquiring customers, and how it relates to the delivery system? I feel like it's changed over the years,

and I know it's changed like the last few years. I'm excited to get into it and hear more. I'm like, you know, there's been so much chatter lately in the last like two years about. In drama, it has changed everything, you've probably seen all the tweets

in podcast, but drama. And then there's obviously Jetman Linus as well. So I'm excited where, you know, we're just like, armchair quarterbacks, we're podcastingers. We're trying to put things together and read the documentation,

but excited to have you break it down. But yeah, I didn't think my understanding is like, creative has just become so much more important. Obviously, we've talked about a ton, but like manual targeting is way less important.

And then there, you know, so creative is more important.

“I think the delivery changes, especially as, you know,”

social platforms have changed. And follow our base feeds are less important. It's just like the types of people of content people are interacting with, like has gotten way more important. You need way more creative than before.

allowing that to happen like in drama or things like that. So I just think the role of what we've had to do from the brand side is obviously shifted from, let me go in and select to the target and make all of these manual changes.

And yeah, let me kind of be a little bit more hands off, but focus on what I can control to actually die delivery, which is not all on target this person. But it's actually, hey, like, I'm going to make this creative. I'm going to make these lane pages.

Is that, is that right, and fair? Yeah, I think you got a lot of it. And I would like really try to like break it down and maybe like structure it, maybe slightly differently.

“But I think that's a nice way of saying you were wrong, right?”

No, no, you were right. I think, you know, I think the way I would think about it is at the broadest level. We really invested in like, if you think about an ad product and you could think about it as a product space

and then like, what happens behind the hood? Like, how do we deliver the outcome? And so one of the ways we've been really investing over the last, you know, several years is the value expression piece.

And that's like, how do you talk about the language of what you value?

And I think, ultimately, the closer I think in general,

we get towards the exact value on how an individual advertiser thinks about outcomes from meta, like a sale, or then the closer we can get to be like, hey, you don't have to like press all the knobs. Because we know what you want, we're aligned.

And so you could give us more freedom across our different optimization approaches to be able to deliver that value. But in a world where we don't have that value expression, then you need to go back and like set very manual targeting and it really kind of break that down.

And not just targeting, but across the kind of its spectrum of control. So the way you think about it is like value expression and control. The more richer your value expression is,

the more it's aligned towards how you actually, like, ascertain value, the less control, you know, I think you need, because then we can then optimize across all of our systems to be able to deliver that. But the more the, obviously, the less, you know,

“we're aligned, you have to then do this mapping”

where you have to look at, hey, this is what meta says, this is what I say, there's like a gap here and how do I kind of tweak the knobs to get there. So that's like one big area we can talk about that across optimization, like attribution,

there's some pretty, really cool launches. We've products, we've built over many years in large investment there, like incremental activity. - Yeah, stuff on, try about that. - And then behind the scene, and then just like,

but, you know, and drama to jam a lot of some, it can go deeper into there. Those are like ways, the ones we kind of align on like, hey, we're talking the same language, we're pretty close. Then we can invest and that's a lot of the advances in AI

to be able to like deliver that over a broad audience.

Like ultimately, like if we have, say, a very large audience

with a lot of different ads that could compete for it, we need to do much more sophisticated job on ranking, real time to be able to like do better predictions and across a larger potential sets of opportunities. And that's where a lot of our ranking systems

have been really advanced as well too. So they really go hand in hand, but I would say that, at least that's the framing that's like helpful to break those down between the two. - That's perfect.

“Why don't we go deeper on this idea of value expression?”

From the advertisers side, for a long period of time, we didn't have a lot of control over, like what we were ultimately optimizing for, and it's really interesting to see meta, because we're really clear at the meta performance marketing summit

over the summer on you are developing the tools to best align and advertisers interest with platforms delivering. So maybe you could give us a couple examples of what that looks like now that listeners might recognize.

- Yeah, I mean, the biggest one, I would say, is value optimization. So if I can like take a little bit longer,

I'll tell you like, originally that was one of the first product

I built like, you know, we're done launching a long time ago. At the originally, we would get like sale conversions, like it would just sell conversions, but then we realized eight conversions are great, but like, you know, somebody buy something for a dollar,

or something somebody wants for a hundred dollars, you value the one that's a hundred dollars more, even though they get you more of the dollar ones. And so we built value optimization. You know, that's been an evolution where we really built that up.

But then we are building like value itself, isn't like, it's just revenue, but there's other things like margin. You also, how you attribute that. And so there's a lot of, so value up positions, a big product. Value rules, like where you could look in an audience

and say, hey, look, you know, I value this kind of audience there. So those are all different functions of, I would say value expression, we'll go looking on it. - Everybody's talking about AI, but few operators are actually using it to speed up their marketing

and make better decisions. At Ridge, we use runeth. It's an AI brain for the whole marketing team. It watches every creative you've ever run, knows what's inside of each one,

and connected to it actually drove revenue. You can plug it into your Clevio, Shopify, North Beam,

So instead of asking what work glass quarter,

“anyone on your team can ask for 10 evergreen concepts,”

worth testing before Q4, and get them back within minutes. We use it, hexclad, use it, Jones Road uses it. It's the best AI brain for e-com brands on the planet. If you don't use it, you're going to fall behind. Go to runeth.com and book a strategy session,

mention marketing operator sent you and you'll get $1,000 in free credits. $100% and would you include some of the custom attribution that you guys are supporting now?

We run a lot of our business using first party MTA data.

And yeah, we've often found it beneficial to just allow me to understand the results that we're really measuring and looking for. Absolutely, I mean, we have incremental attribution, which is really tilted towards delivering conversions

or sales that are, you know, that you're crediting. Oh, that's for you. Like, hey, this wouldn't happen without meta, according to your systems. And then, custom attribution is even that step further, where it's like, hey, we have a very specific way

that we think about attributing sales from meta. And we're like, tell that to us, so then we could learn it and deliver that. Yeah. So, Connor, what has your guys' approach been?

Between incremental attribution, custom attribution, value rules were brought up. Like, how is hexclad leveraging some of these tools to death? Yeah. Why are you very hug after this?

'Cause we've put so much spend into value. Oh, yes, we think of value. And it's like, we are huge on value. And like, it makes sense for our business. If you were to go look in our CDP and you were to look at our LTV,

it's 100% correlated.

The most value we can extract on the first time order,

backs into the best 12, 24 lifetime value overall time. So, for our business, where our hero product is a big set, you know, we have people on our funnel for a very long time that we're educating on this. We've put a ton of add dollars into value optimized.

I was looking last night in like 50% of our ad account right now is optimizing against value. And it's just so aligned with our business goals.

“And I think that's what's exciting about, you know,”

I heard all the stuff you just said. It's like the custom attribution, which we use, the incremental attribution, which we use, lowest cost, we use value, we use like, we have, you know, this hero funnel that we use a lot of value optimized around.

But we are just, our business is becoming more and more nuanced in robust, and now we have funnels that it doesn't make sense to top to my survival. So we're able to just like align these different campaign objectives with the goal of that funnel.

And now we have a very nuanced robust ad account. We're certain things for using values, certain things lowest cost and for us value has been huge. Like we've gotten so much scale of it. We're acquiring exactly the type of customer

that we want because of it. And then we're able to point and shoot, you know, when we roll out a new product, it's like, all right, maybe value's not what we want for that. And now we're able to balance things, like,

hey, sure the value optimized stuff isn't going to be best

for like first order velocity, but it is going to extract

exactly the type of a customer you want for that product. And then we have these other funnels where it's all about volume, all about lowest cost and not as worried about, you know, extracting as much value upfront. So it's just like way, like, if I were to look at the ad account

three years ago, way less, you know, campaign optimization objectives live and it's exciting to see that we're able to like point and shoot our business, where it makes sense with different things. And, you know, values definitely been the leader there

with just like what we build our acquisition funnels around. - That makes my heart happy. - Great, awesome, awesome. - One of the things Conor Touched on, which I'd be curious, your perspective on is not only just

something like value optimization driving the type of order that you're looking for, yeah. But using them in conjunction with one another, and a discussion we've often had is like, can optimizing for lowest cost as well as value,

as well as layering in some of these other sort of value expressions allow us to reach new customers. Is that something you guys, yeah, what's your perspective here?

“- No, I mean, I think like, no, that's great.”

I think there's a great question. I think something you said is something I will say, which is ultimately like every business is nuanced and it's complicated. And I think the greatest thing is to be able to,

you know, you are the expert at hexa, I don't know how the hexa is great, great-pant. (laughs) But you are, you are the expert in how you value the, you know, how you see the whole thing.

'Cause you know, ultimately like I met a, we see like a part of the picture. There's a much bigger part of the picture you see. And so it's really about, you know, making sure like how our products fit with how you value in which map

and finding that mapping is the key. - My question was like, how, what Connor was describing was using different value rules in order to reach new customer, oh, I think got it. - Which is highly related to metas prioritization

of like explicitly trying to react. So again, there's a great segue into that. - But more like, is there more incremental reach instead of just being completely on VO, but like having like VO and CO side by side

because you might be reaching different audiences. Like does metano like, oh, hey, Connor's, Connor's a high value customer who's kind of by the 12 piece set and Cody's only got a buy one thing.

like existing customer, it's the, the optimizations isn't necessarily like, I can't make a definitive statement that like CO plus VO is gonna let you reach a new set of audiences versus just VO because it depends.

It depends on like the audience depends on how well your audience, you know, is educated about your product.

And so ultimately the way we think about it is,

you know, we have a prediction system and that prediction system is focused on trying to find like whatever you tell to say, you're really pointing at it in a very like myopic way in a way where, you know, we, if you tell us like,

we don't really care about new customer, because we'll all try to predict is the cheapest way, or the most valuable way to get your next sale.

“I think we don't have a concept as much about,”

they like, once you give me an audience, to be like, oh, there's a new customer, we're gonna prioritize like a new customers, which is an existing customer. It's really dependent on each impression.

What's the next best one? And it just happens to be someone who's purchased before or not, that, you know, is irrelevant. It's really just dependent on what's the, the next best. Does that make sense?

- In Cody, you've obviously followed closely like the new releases around optimization events and things like that. What are some of the ways that you've leveraged them and maybe what are the options that you're most excited about?

- Of specifically like new customer stuff or anything. New customer stuff, value optimization, incremental attribution, I might be leading the witness here a little bit, but I know some have hair deeply about. - Yeah, so we were part of the like new customer

investigation where we met with like Meta's product team. - Oh great. - And I think it's just like share of stuff, 'cause like the challenge that we found is practically, you're in half the guy was like, you know,

we were just, we were spending more and not as we're reaching new customers, even though we felt and had data like our exclusions weren't working very well. You know, we started using like a third party tool when it helped a little bit, but like we would try to exclude

and we would still reach the same people. And there's probably a lot of creative stuff and other things we could do, but we, you know, really tried to push Meta and Meta seemed to take it very seriously.

I know it's a big focus of how can, if brands care about new customers, how can we provide stuff for them? So yeah, we've been like, I don't know,

my toxic tray is like I always like to test things early,

so like Jones wrote a product disruptors, so like any alpha, any beta, it's like, yes, we'll try it, so there's obviously like, you can tell us about the customer life cycle strategy and yeah, some exclusions thing,

but for us that's just like really what we care about, it's not that we don't want any spend and existing customers, but we had times where it was much higher than we wanted. And I think maybe that's, you know,

maybe a flaw in our business or whatever it was, but like for us to grow and hit our targets, we really need new customers, incremental attribution was great for us, like really both on that,

obviously we talked a ton about in mentality and we care a lot about that, so I think that was a big one. We were starting as I was leaving you on a sort of test, like on the channel optimization as well,

'cause we have our own stores and we can pass that through. So there's a lot, I think it's, it's, you know, it used to be very simple,

“it used to be the power five and like this is all you have to do.”

And now it's like there's no one playbook, it seems like, but it's like you have to figure out what works for your business and what works for you guys and you guys is gonna be like very different than what works for us. All right, this ad is technically a job posting

for Neonpixel, but stay with me here. Neonpixel is hiring performance marketers. People who actually get incrementality, they understand hold out the understand triangulation and really that's the entire pitch,

because when you sign with Neonpixel, these are the marketers that are running your CTV strategy. We've been with Neonpixel at Hexclive for three years now and they really do work like an extension of our team. Live on real living room TV in days,

helping with creative strategy, same day end card swabs and measurement, read in our stack, not just some black box dashboard that we don't really understand have insights into. And that's ultimately what a managed service looks like

when it's staffed by real operators and real marketers. So whether you want to work on the Sharpest CTV accounts in DTC or you want that team running yours, go to Neonpixel.co/careers to apply. - So I'm curious from Metaside,

it does feel like there's this shift in perspective to Cody's point, power five, relatively simple, talking about consolidation. There weren't a lot of optimization targets. Was it two or three years ago where you guys realize,

hey, we don't have maybe the full context or we can't perfectly align every business's goals with meta ads in the way that we do now. So we are going to roll out these new optimizations. I'm just curious when that shift in perspective change

because it feels like all of a sudden to Cody's point on the channel optimization, new customer optimization value optimization. Like now I've suddenly have infinitely more tools. I'm just curious at what point did that shift

and now that we're like reaping the benefits of it.

- We've always thought about like,

hey, we need to align the customer with where we are.

“I think what's expanded is we've invested a lot”

over the last couple years. So anytime you see like a new product, we're likely working, I think I had the first meeting

It was called something else internally, maybe two and a half years before released. It just takes an incredibly long time to be able to build these new models, understand how they work, get the signal.

And so it's something where I think we just got the 23, we just released that bunch of stuff. If you imagine, there's been some changes in 21 that we were really working towards for a couple years as well, like with ATT and kind of rebuilding our systems,

post ATT, which is a challenging time. And so I think it's just a function of like, you know, we spend a lot of time working on that and then we had like space and bandwidth to continue like our kind of forward progress.

- Yes, so a lot of brands here are really focused on lifetime value, or at least the brands that are listening to this show. I think there are some brands like HECS Cloudware.

We get most of that value in the first drawer,

I think Ridge is very similar. You guys are very first drawer

“and you have to be first drawer of profit”

or in that same boat. We also, it's a little bit easier because we know that the higher the first drawer or AOV, the longer the lifetime value, we've tons of data that supports this.

Now some brands are like very, very different. They're getting, you know, two, three, five X, LTV. Like, five X growth on that first drawer or AOV in a year or in two years. And like, they need to be optimizing for a longer window

of value that's outside that first order. So what's going on behind the scenes? Because we've talked about this with some folks that matter that you guys are building LTV products and like, what does that look like at meta

and like, what do you expect to roll out with like optimizing for LTV? - No, LTV has been like a Nordstar for a very long time. It's a very difficult problem to get at. We are in the process of, you know,

offline beta testing LTV products and we're hopefully gonna be able to like ship something that I think is useful for folks.

Take us back, I think LTV they're like always been the Nordstar

but the difficulty in getting to LTV is that the first of all, it's like a harder prediction that the further you get from like, you know, something that's very quick and like, you know, say click that's super easy to predict relatively

to an LTV that there's a lot of space between them. And also the LTV definition for different advertisers is different. I remember like, you know, many years ago I was in a meeting with the CM over a very big company

and they're like, we want LTV and it was like, go out there and write the formula. - Yeah. - And it was difficult because it's like, oh well, I'm not sure exactly.

And so that's where it becomes like a challenge but yeah, we don't say that. - You're saying like some people are saying like, when they say LTV that mean lifetime revenue and then some people actually mean LTV

that like taking out costs and like an actual like lifetime margin. - Let me give you a blank three months and be with me in 12. - Exactly.

- Right. - What do I mean 12 months? - Yeah. - Like, you know, it's on advertisers or like, you know, after 30 days, all the extra value

is actually from us because we're doing a great job upselling or we're a great job like remarketing. And so really that definition is very nuanced. And so like, and then the ultimate for a machine learning system is you need more data.

“You need to be able to have like a standard definition.”

And so that's like the hard difficulty of balancing like the customization versus the what? - And what I've heard as well especially for like a consumer brand is like there's no good expression of value

and an event to go back. It's like, it's one thing for like for an app or something like that where it's like, hey, this person logged in this many times wherever they made it this page

that you can pass it signal back but it's like we're obviously, you know, selling something offline in the physical world. Like, how like what is the attribute that's able to then correlate?

And like, oh, this person did this and the unboxing and they took the product this many times. Like, there's no way to optimize for that as well. - Exactly. - So, you know, realises that whenever we release a product

we always are like comparing it to the next best alternative

that we have. So, we're constantly also battling our own selves. And so like when we improve value optimization for seven day, our, say LTV product has a generate more LTV than our seven day value

which is already fairly good at generating LTV. So, it's also like we need to make that benchmark higher to be able to build a product. But we are excited, we made, you know, we have some great products in the hopper

and you know, takes years, very excited about it. And just in terms of like a little bit to Cody's point, how a brand will ultimately express what they think of as LTV. Is that something that you guys will ultimately define

and then like sort of extract from the brand's data

“or is it something that we will be able to explicitly state?”

- Look, we haven't like finalized the product but I can, you know, I'll just share my intuition. My intuition is that it will be a lot better for the, the app does it to tell us where how they value LTV and for us to intuit it

because first of all, it's the calculation. You then need to align with it, goes back to value expression. Like if I think LTV's X and you know, you think it's Y, you're right, you're the customer

but then I need to know, right? And so ultimately it's better for that. - And by the way, just to like back up your point about this being Northscarf for a while when Joe and Throde first joined the Strupper

And it's like a like a lunch, Jake Bailey,

who leads the Struptors, like that was one of his main things he was excited talking about LTV and stuff. - Yeah, so four years ago. So it seems like it's been in the works. - Yeah, we're excited about it.

It's our Northscarf, but you know, you got, you were taking steps to get there. - Most brands treat customer support as a cost but with AI, every support conversation is now a chance to increase retention,

recover revenue and grow your LTV. Rich panel doesn't just use AI to answer tickets. They allow operators to run revenue experiments inside your support. You can test different windback offers on refund requests,

subscription save offers during cancellations and product recommendations that turn chats into sales. Plus they guarantee you're going to save money. 50% of your support volume automated in 30 days or you're running back, that's pretty good offer if you ask me.

So at richpanel.com/demo, they'll build an AI support team live for your business and show you how it can drive incremental revenue for you.

“- I think we've covered value expression really thoroughly.”

One thing we mentioned earlier and this was like the buzzword of 2026 in my opinion is in drama though. But you can throw in jam, you can throw in lattice. You just talked about ad ranking

and the predictive mechanics of that. Can you talk briefly about what the new system is and maybe how it has improved over how ad ranking worked three, four, five years ago? - Yeah, let me get it.

So we talk about value expression. So now imagine a world where all advertisers are the value is aligned. Like the way you define values, the way we think about value. That now it becomes a problem of a big machine learning system.

How do we predict that value as quickly and as effectively as possible? And in the past, I mean, I joined as a long time ago because I thought this was like one of the greatest applications of AI and we really seem that come to pass.

The way I'll break it down one by one that we can get into it.

So and drama is the first step

of time determined which ad we show is retrieval. It's like the first stage where we're saying, all the universe of ads, which are the ads that we could show for this particular ad impression. And that's a complicated problem

because at any given time, if we are building a system that is value aligned, there's more opportunities for us to look at. And so we need to retrieve more ads. And so the previous version of our retrieval,

we're also time limited. If think about it, we have limited amount of compute. We have limited amount of times because when people are schooling down their feed, we have to predict which ad to show.

And so given those constraints, we would retrieve a certain number of ads to say, hey, this is where we think we should look. And then there's these later stages of ranking where then you can refine that prediction.

So it's not like one system where you look at it

and you predict it, you say, first system looks at what

are the universe of ads we could look at. Some of it is easy, hey, when this country are you not in this country, but with automation, you would have a lot of ads. And so what endrometer does is a step change

in the ability for us to look at that initial step.

“And I think the, you know, it's about 10,000 more ads”

can be ranked within drama because of advances in AI. And it did previously. - Can I pause for a question tonight? - Sure, yeah. - So, hey, yeah, way more ads

available to be shown for that impression, right? - Yeah, what is the impact on the advertiser? Is there more competition, like, do they ads have to be better? 'Cause now you're competing with way more.

Do you need more ads to try to get them out? Like, what's a practical takeaway to like, a D to C founder who's watching this? - I think the, the practical takeaway is that you're competing, and it's a good way

because you're able to, like, say in the previous world, right? We're like, you know, we, the, the concept called truncation. Like, we'll like, look at the top hundred. Yeah, I'm making up a number. And then we'll like the, the rest of the 900,

we'll just not even look at because we have a limited time. We kind of, you know, kind of look at, we don't have time to look at the whole universe. With Endometer, we're like, okay, now we could like, look at the entire universe.

“And so for a founder, I think you're competing”

in, for more audiences, so it's a good thing because we're able to find a better match. It's all about personalization. And so I think that's a positive thing. It's all happening under the hood.

So there's no, there's no like, I would say, impacted like, accrues to you. If you have a, you know, a value line campaign that's liquid that we could look across a big audience for, you're gonna actually, your ad is gonna be considered

for everyone. So I think that the main founder, the main concept is, if you're gonna like really restrict your ad account on the targeting side on the audience, so we can't reach everyone, you know,

there's others that, what the same product, what the same thing, if you have a more value line

competing in a lot more options. - Yeah, so it's almost like the bars higher, like you're able to show the user like a better ad for them or more relevant ad. - Yeah, absolutely, right?

So imagine like, I was trying to pick like, you know, I'm looking to take a simple example. If I had like, you wanna pick a pair, you wanna pick a new shirt to wear. If you had too sure, it's like, it might be pretty good.

It might not be, it might be the two best shirts you have, but if you have a thousand shirts that you could choose from, the probability that like the 998 shirts, like you might find one that's like better

than the first two that are obvious, you know, it's higher.

And so you can offer better personalization for the user, which is great because they're getting more relevant ads. And then for the advertising, it's great because you know, you're also, you know, like able to like be considered for a lot more folks.

Amazing, okay. So we talked in drama to, do you feel like, if we could unpack a little bit more like gym and lattices or approach in that,

“I think it just comes down to that refining process?”

Yeah, we could talk a little bit more about how that works. - Yeah, so, you know, the first day just adds to like, you know, think of that a retrieval, being able to retrieve all the ads for each user. And that each user each given time, right?

That's basically the, kind of, the next step is, you know, you have, you know, 10,000 more ads to rank from, how do you rank that better? And so, what gym does is, it's a foundation model that gives us more information.

So the concepts of, and so what we can do as the later point is that we can understand ads better and look through the universe. And again, generate more personalized retrieval. So, know when you have 10,000 ads,

if you don't evolve your later stage ranking system, you're not gonna be able to better serve. So they really work in conjunction with each other.

So we can, the first step allows us to look at more ads

and the second step allows us to better understand them. And that's what gym does is the big, finder small, so we can better understand, not just like a single adding person, but also there's a bunch of sequences,

like so we can start predicting, you know, not just a single, what's gonna happen in this particular instance, but like the sequence of actions that might happen. And so it's really a cool way to, you know, again, better personalized what we deliver.

Can you say more on the sequences and sequence when my understanding of it is like, the system is better able to understand where somebody is and their buyer journey and show them not only the shirt, but actually like the ad with the shirt

that's gonna match where they are and their browsing journey. - Yeah, and so I think the concept of sequence learning is that like, you know, you want to simplify the problem in general

like when you do a big ranking system, you want to be as simple as possible. So then you can like scale. And what the jam model does is allows us to, when you do sequentially not just predict like a single action,

but like try to understand like, what are the sets of actions that may happen that will deliver? And that allows us to deliver more, you know, valuable coverage.

“So that's what it does is like better understand the user journey”

to be able to then predict, you know, which ad will do better. - And just to clarify that point further,

because I've always thought of sequencing.

And again, this is a more or less learning. We do the podcast for free consulting, and this is extremely valuable. - Sorry. - People are the most important part of your business

and finding real a players is hard and incredibly time-consuming. That's why I want to tell you about propel. propel is honestly the best recruitment agency that I have personally worked with.

They are marketing and operations specialist. They source talent from US, Latin, and beyond for top-peak-com brands like Jones Road, like many others that you know and love. At Jones Road, we've hired over 12 people through them.

We've hired video editors, graphic designers, we have multiple retention managers, creative strategists from them. What I love is not only they easy to work with, they're quick, but they work on performance.

So you only pay a one-offie. Only once you hire somebody, you don't have to pay, you know, any retainer, no-one going fees. Plus every hire comes with a three-month guarantee. So I highly recommend them.

They've worked really great for us. We've had a really great team that they've helped us source. I had to link in the show notes and submit the form to get 10% off your first hire.

- But the idea of sequencing sort of like, I'm loading up hundreds of ads live in my ad account. And Meta now has a better ability than ever. To not only know like, who's in market for a ridge wallet, but the fact that they've maybe already seen one ad,

they understand what it is. And now it's actually more about a more bottom of funnel. Maybe we're talking about social validation or the lifetime warranty. Is that something that's happening under the hood a little bit

in terms of best sequencing out the ads

“and the messaging that will lead to the conversion I'm looking for?”

- I think it's trying to understand where you are in the, not just in that particular ad campaign, but across all of your ad campaigns. And so it's really trying to frame up like where you are across the whole journey versus,

I think like you know, you hear this concept with full funnel, like I may be a misunderstanding, so you guys don't tell me. But like, you know where you're like,

not really like trying to get some outcomes. So it's not at that level where it's like, trying to understand like a very long usually, because it's because but it is trying to understand like where you are in the journey across all your campaigns

and the stuff that you've done. - 100%. Okay, so before we started recording, you said that you spend quite a bit of time on X Twitter, which I'm not sure, like I'm sorry.

- And thread. - For our behavior and threads, yeah, of course.

So this is where we're always talking about the latest

in the meta algorithm and ad delivery. What would you say is something that marketers get wrong most often? What are some things? - Or are you also saying you listen to us?

So what do we get wrong most often? - Oh yeah, yeah, yeah, yeah, yeah. - No, I think you guys are pretty good. (laughing)

“I think there's an over attribution of the volatility.”

You know, I think fundamentally when you make harder predictions, you're gonna be wrong more often, but over time, it's the right thing to do. And I think that's one too, I also think there's a lot of fluctuations in ad demand, like an any given time,

like different advertisers come on. There's a lot of seasonality. And I think people take those changes and then they attribute them to things that they've done. And so there's this like link on causality.

Like hey, I did this thing and then this happened and therefore those are related. And a reality could be, but there's, there's just like a lot of volatility across the board, there's seasonality.

And I think the biggest thing I would say that people kind of get wrong is is over attribution of something that happens to some action that they took. - You know, I learned something earlier.

I always thought that CPM worked on like a dial.

I thought that you guys had the good dial that you would just turn up on CPM.

“And they give us torrents actually a lever.”

(laughing) If there was somebody back there with the rich ad account, just kind of, no, I'm kidding. But we talked about this one often. And we just talked about it recently.

Cody's a part of a new brand called Lynx and brand new ad account. And I went through this experience earlier at Ridge, we launched a new brand as well. And when we first launched ads,

I felt we had like an almost an artificially high CPM. And this has been an unclean conversation on Twitter recently. Maybe you could shed some light on whether we're just making it up.

Or if there's something about the ad delivery system and the auction that just kind of has to calibrate who you're serving ads to and therefore we started this higher CPM and it normalizes over time. - We don't have a CPM lever.

Let me, I think that is a dial. That's what I do full time. I just don't like what Connor said on the podcast. No, maybe I could de-miss if I like how the prices are set. I think that might be a better.

'Cause we're talking about conversions, right? We're talking about value optimization. We're talking about very deep optimization.

“But then you kind of narrow back to like prices, right?”

'Cause it's CPM is a reflection of prices. We run an ad auction and there's, I'm gonna go back to Econ 1 and 100, if you guys take that. But there's a supply and there's a supply of ads that we have. You know, based on users, based on where they are,

how much people use our apps. And that supply is determined by humans. And I think our supply is growing. We have people very engaged and you can see the results, like people like our apps.

And so our supply, you know, we have a healthy amount and it's growing. But that's controlled by how many people kind of use our apps and how engaged they are and partially by the quality of the ads.

And then we have demand. And demand is like all the advertisers there. And so we talked about like, you know, we have different DTC brands, but we also have a lot of app advertisers.

We have lead gen advertisers, you know, people who want leads. We have folks that are like movie trailers and like brand advertisers, you know, co-gate, PNG. And so we have a healthy broad base of demand.

And you know, my goal is to generate, you know, to build products that work for everyone, right? And so, you know, DTC is a huge, you know, we love our advertisers there. But we also have, you know, on the channel,

we have retail advertisers. And we want to build great products for everybody.

And that ultimately is the demand that we have.

And the way our auction works is we're predicting like how much value we're generating for each advertiser. And, and ultimately, if I'm doing my job right, we're able to build great products for every advertiser. And that sits in an auction.

And the more demand we have in an auction, the more liquid that demand is, the more people are competing for the supply of ads that we have. And the way our, you know, the way our auction works is,

we want to allocate the ad impression to the place that we think is going to generate the most value.

values consist of advertiser value, like how much we'll get, but also the ecosystem value, right? The use value, and we didn't talk about that too much, but I think that's a really big component. And so, we're trying to, what we're trying to do is maximize value.

And the pricing is set by like,

we always want to say about how much competition there is

for that, for that ad impression.

“And so, ultimately, that's how prices are set.”

And then it gets filtered back down to like a CPM. You could see that as like a CPM. But we're, and so ultimately, you know, we're not really, you know, the thing I focus on and I think most focus, you know, on my team,

are generating the most amount of value for everyone. And the prices are just like an outcome of this market mechanism of how much supply we have, and how much demand is in our system at any given time. And together, you know, you can, if you remember,

you can't want to want like, you know, where supply is, where demand is, that is the market price. And that's kind of, and it's more, we'll more complicated, but that's how it's CPMs are set.

So at any given time, like, I definitely think like,

our predictions get better and a more direct message question. When you have a new brand, like we have predictions. And, you know, the more data we have, especially with the sequence only, we can learn across a broader range of data.

I think we get better predicting. The newer brands, like, you know, we're learning. And so we have less data. So that might be a function. But, you know, we've seen, you know,

I've seen advertising with like new brands that also do well. So it's really just a function of that market mechanism. Versus, like, hey, there's like, you know, let's monetize it, let me, you know, uh-huh. We talk a lot about CAC on this show.

You can't control what meta charges you, but you can control what happens after someone lands on your site. That's where our sponsor after self comes in. Helping you make every visitor worth more from cart offers and one click post purchase upsells to subscription upgrades.

And here's the best part. Rock, thanks, turns your confirmation page into another revenue stream with cross brand upsells. With earned and save, those earnings also offset your after self subscription dollar for dollar.

And you get to keep your entire payout. Head to aftersell.com/earn-save to see what after self could unlock for your business. And how you could get the entire software for zero dollars a month. And I mean, just to the point of a new advertiser code,

you just went through this experience. But it's not as if the CPM's remain cost prohibitive. It did feel as if it were learning phase over time that then normalize. Yeah, and it feels like to me without like knowing

what's behind the scenes, it's like as we introduce more positive signal into the account that we think is going to convert better or be like a strong signal to meta, um, it helps that prediction. For example, like a lot of people say a white listing or partnership ads can can help obviously their signal that it uses from the creator

and, you know, those have really helped us, um, we verify our business manager and not brought them down quite a bit. Um, we haven't done it at a ton, but like I've heard people like they've had their founders go on podcasts or stuff like that just like organic traffic, right?

Like people tagging us on social, like I have no idea if any of these are actually important, but it's like my philosophy so far has been like, let's just introduce more positive signal into our account and to our business and it's, it's not going to be a negative thing and we have seen CPMs come down with that.

“Yeah, if I can at, yes, I think, yeah, I think, uh, the thing I would say”

is, um, obviously the more, um, data we have, the better prediction so we're not kind of, I think about it where less, we're not, we're not showing too many false positives, we're like, oh, with this person might like it, they don't, and then, you know, our predictions get tighter. So that's absolutely so signal set up, sending data is really valuable.

Using some of our best, um, products is, it's also good. On the flip side, I also think, you know, over time, there's also this organic conversion, right? So like, what we deliver is we can deliver a user to, you know, for DTC via a website to a website.

Ultimately, if you're doing a great job of having, you know, a great ecosystem

where people trust your product already, people look at creators and they're like, oh, I trust this person, that conversion happens. That's the credit for the advert, you know, that's the business is credit. And what we do ultimately is like, if there's a lot of, like, say, like, intent, that's already sitting in the ecosystem, that's great.

That means, you know, we're generating value, that's, like, users like it. It's easier to sell, and so we have to, you know, go back to conversions. We can just, you know, give you more conversions for the same number of dollars. And so, uh, there's that other piece of it, too, that people sometimes undervalue, which is like, if you have a great product and people are buying it,

it's easier for us to, like, buying people to, to, to go buy it. Yeah, and we've seen the even, like, when we launched, uh, TBS. So, first time, like, met up performance significantly increased. We, we, we care about, like, uh, with, like, new visit percentage rate and, like,

“that improved, like, I think it's just, like, very positive signal.”

So, so we've seen that, and even from, like, brands I've talked to, like, the only few brands I talked to have that have not had the CPM measure, like,

Like, kind of started there, um, so that's what we've seen.

Yeah, people are familiar with a brand.

“And I think about, like, when you're, school, like, a theater,”

you're like, you, you, you, you maybe kind of interested, but you recognize the brand, or you've heard good things or something. Like, you're going to click on the ad more likely. And if you click on the ad more likely, and it's a good, great product, you're going to, like, purchase the more likely.

And I think it isn't the, correct me wrong. I think it isn't the documentation of maybe gem or something like that, where organic is actually, like, a signal in the algorithm. I've seen that, like, in the delivery system. Yeah, I mean, like, when you look at user journey, we look at all the signals available to us

to be able to predict that, right?

I'll tell you everything that's, you know, we're, you know, able to access, um, with a, within a pot policy. So, in that, and that, an example that Cody gave. He just went through, like, a laundry list of things that he's been doing. And he's saying, all right, I'm doing an accident.

I've seen my CPM come down. And that situation is he being, like, let's say they, they unlock, they, like, roll out a new white listing out. And it has a great CPA way better than the account wide. Is he now getting awarded better CPMs to the same audience?

Because they're, you at your system saying, oh, winks is now providing more value to this audience. Because they're providing more value. They're going to lower CPM or are they reaching a different audience. And that audience is just a cheaper audience to reach than maybe the one that he was

paying for a month prior, like, what's happening in that situation? It's, it's hard to determine, like, again, going back to it. It's, it's really, like, there's not a system here. Like, oh, we're going to find a more expensive audience. And we're going to find cheaper audience.

It's a more, um, simple function, right? Like, what's happening is, uh, we're able to find more sets of folks. So we made predictions. So if you remember what I was saying, we were trying to find the based on the optimization, the next cheapest of a valid position.

We're trying to find the cheapest roll-ass, which is the most revenue divided by the cheapest, uh, you know, price and the cheapest customer, we want to do is be able to find the most amount of value. Right. And so what, you know, what code you're doing with all the work, I think creates, I think, more signal for us to predict. But also more, you know, the audience, the users, uh, more demand, more intent for that.

And so that's what's happening. And so it's less about, like, steering the system towards, like, this audience, which is that audience, I think there's, like, a misconception there. It's more about, there is just generally more positive signals. And so we can predict better for a broader set of audience.

And there's more, you know, and it's really credit to the businesses that you're building a great product that people want to buy. Um, and I think the way our CPNs look is we're not like, again, not a CPNs up, but it's like, hey, going back to the market mechanism, would it say, I want to generate more value? Um, I think it's easier, uh, I think we, we get better predictions. We're like,

oh, more people want this product. So we think we could generate more value. Um, and it's, and it's, and it's cheaper to do that because we're right. And then, you know, once we're right, we, you know, we charge whatever again, based on, like, the demand is over that system feedback cycle. Yeah, exactly. Exactly. And so the system, it's great.

Like, I love it when people are like, hey, I only, you know, where a few people tell me this, but they're like, I, you know, it only cost me, you know, X. And I get 10 X the revenue. That's great. I have no desire to avoid to, like, generate, you know, like, like, uh, you know,

“the, I think, uh, that's a great place to be. Um, I want to do that for everyone, right?”

What I want to do is ultimately, if I'm doing a great job, then there would be more, more advertisers

that would see that big value. Is that makes sense? Yeah. I think I just want to be clarify that it's not like, there's a CPN. The CPN is the outcome of the other pieces of Mark, uh, supply and demand and demand being an outcome of like how great you're at it and how good the user value is versus like, you know, what the CPN is. You guys heard of here. Very clear. No CPN dial. Yeah. That's why we're here in person investigating. But if you, is so if you have, like,

let's say you, you're an advertiser and you have 10 ads live versus 500 ads live. If you have 500 ads live, then theoretically your chances of reaching more audiences are way higher than if you only have 10 ads. And if you're reaching more audiences, then chances are that you're going to be reaching audiences that are maybe less expensive to reach than the audience you're hitting with 10 ads. Like, is that like going back to like, and drama in creative diversity, like, would you say

that that does contribute to getting better CPNs? If you have more ads peeking a more people. And if you have, if that's the case, then inherently you're, you're reaching different sub-pockets of audiences in meta. And like, would that help drive your CPN down? You know, the difference between hitting your numbers and missing them, clear signal on what's actually driving growth. It can get really, really noisy. There's so much noise. You got platform data. You got blended data. MMM,

all the acronyms, MTA, experiments, all of it, all pointing in different directions. The more

“you're spending, the faster you move, the more the bad signal can cost you. That's why we use”

house and we've been using it for years. That's why the other marketing operators do as well.

a lot on a podcast. What is the true impact of your advertising dollars on your business? We have causal MMM for channel level budget calls, causal attribution down to the ad level, and architect their AI agent tells you exactly where your next dollar should go. And the results speak for themselves. Stockx saw a 41% lift in IRO is using house and you're not stuck with a help desk. You get an embedded measurement strategy to actually help strategic better decisions. Their whole team is great. We've

worked with a lot of them. They are world-class air. Go to house.io/operators. H-a-u-s.io/operators and start backing your budget calls with real causal data. I think creative is a big lever. I think having a diverse creative. Again, let's go back to what we were saying. Like I think on the

when you're value-optimized, you could then be more liquid. But creative is always something

the advertiser will own. And so having more creative, more diverse creative is generally a

“more effective way for us to be able to deliver. And so the way you should think about it is”

if you have more effective creative, say everything is the same. But you're creative, you have more creatives to choose from. Our system would be able to better find the right creative for right user at the right time. One of the things that stood out to me from the Meta Performance Marketing Summit was how Andromeda and the new ad delivery system had led to an increase in relevancy scores. And is that just another way of saying like we're creating a better

user experience, user value, and therefore theoretically, you know, the advertisers that have more diverse creative. And I don't want this to focus on creative. But is that relevancy score sort of a metric that we could be thinking of as user value from the way that you're describing it?

No, I would think about it as advertiser value. And so ultimately like, you know, for any given,

you know, the best possible outcome is I show one ad impression and it make the sale. Right? But, you know, that rarely happens. Like we generally have to, you know, show multiple ads. The same person, but also adds to different people. You know, advertisers are still learning this concept, though, that it takes more than a single ad in a single campaign to drive budget. So yeah, I mean, yeah, good point. Yeah, I would love to one shot it. Yeah. But we're not, we're not there yet.

And so ultimately what, you know, all these ranking systems do is allow us to better predict, right? And so the relevancy score is we're able to do those predictions more accurately. So we can drive more clicks for the same ad impression. We can get more clicks out of it. And for,

“you know, every click with generate more conversions out of it. So I really think about it as,”

you know, we can basically use less ad impressions to generate more value. Yeah. And, and I do

think there's a, the second effect of generating more, you know, user value because they're seeing more relevant in ads. But I think it's primarily like a positive to the, to the business side, to the advertiser side where they can get, you know, more relevant ad delivery. One of them, you know, you can get the same outcomes with less impressions. Talk about value expression. We've talked about the new delivery system with Andromeda, Jim and Latis. And, you know, for a long time,

and we've spoke about this at length on the podcast. For a long time marketers would sort of like toil away within the ad account, making all these optimizations. And frankly, you could create a lot of value that way. I'd love for you to speak through your perspective and met as perspective on what the marketers role is today with these new tools. You know, I think the marketers will is huge,

“but it's shifted. I think in the past, it was like you said, manually tuning a lot of the novels,”

making sure the output is matching how you see value. I think in this new world, it's really about understanding your business. Like, what is it really driving with the incremental, like revenue, the incremental value to your business, making sure that is aligned with how meta is pointed. So you got a steerer system towards that alignment, the incremental tuition, custom attribution value, optimization value rules, LTV, and just really mapping that together,

making sure that signal is strong, making sure there's a line. And then there's all a huge piece on the creative side as well, making sure there's now that you have a huge audience to go after, making sure that creative is relevant, creative is fresh. And so it's really around like moving away from manual tuning of knobs to like telling us the broad picture, making sure we have the right assets, and making sure that we're delivering what you, what we want, which one? 100%. And I think this

is a really exciting feature. Connor, I'd love to hear you speak about with this idea, which I think we've all been aligned around for a few months now. Like, where are you in marketers at Hexcloud spending your time if you're moving a little bit out of the ad account in the day-to-day tuning to something that best aligns, you know, the Hexcloud business with the meta-ide platform? Well, I think what's cool about what everything we're talking about is that like all the same

lovers from five and private and longer 10 years ago are still the same lovers today that really moved the needle. It's great creative, it's great destinations, and it's great offers.

big creative swing that you're taking? I think that at least what I've seen, that's still in the

humans' hands, like creating those big swings, those net new productions, and think the creative gen stuff is more relevant for like iterating on stuff that, you know, a human power team has made. So, I mean, that's where we're focused on. It's like, what are the products we want to be building funnels around? What are the best ways to explain these products in creative? What's the

best destination to send someone to? And like, what's the offer on that? Those are still the same

“lovers that were focused on, and that's what's allowed us to scale. I would side of just like the”

hero funnel that we had, you know, two, three years ago. Now, I still think there are some things that we're doing to make sure that we are aligning with what meta-watch from a signal perspective. You know, we obviously have like, capy set up, you know, we're using event match tools, so we're giving like the best event match in meta, like an L of R type tool, and then as far as how we organize our ad account, I do still think there's a lot of importance there. Like, I'm not going to go

“in, you know, we talked about this last week or Japanese product, right? These are 130 dollar products.”

I'm not going to go and drop those ads under my value optimized 12p set that has a million dollars

put through, because I just don't think that that's really setting that up to go find that new customer for this new product funnel, the best, you know, where that ad account or that that campaign's trained on a $700 set buyer, so I don't really want to drop ads promoting a $130 product in there, so I still think there's, there's some like outfit in that with how you decide to set up the account and the campaigns to make sure that you're passing the right signal back and not confusing

buyers, but I still think that like 90% of where our energy is focused is on the ads, the landers, the offers, and that's the same thing we were focused on in 2020, that's the same thing we're focused on in 2015, it's just that the technology's better, so now we're getting even more, you know, you talked about like jam and lattice and all these behind the scenes, things like that's nothing that the marker has to do, right? We just need to be giving it really good inputs and now the

“the ad tax even better, so yeah, that's how we're thinking about things. Again, I think there is a lot”

of importance with how you structure them, and we've spent a lot of time, do you do like separate campaigns, separate ad accounts, separate pixels, I do think there's a lot of ways you can go about that, and that's important, but yeah, those are those I think are focused on. And you know, it's working because the cash is a happy customer. Exactly, right? He's got that hex head lever going for a day. All right, before we jump your code, anything you want to add on this last point, that's a great one.

Fantastic, we'll end there. The cash thinks again for coming on.