Haus CMO Reveals the Incrementality Index Behind Smarter ROAS Calls

We've got one of my favorite people, great guest today, Olivia Corey, chief marketing and

“strategy officer from our favorite geolift measurement toolhouse, Olivia, how are you?”

>> I am doing great, I can't believe Cody is not here, but I don't know, maybe I'm a little bit sick of Cody at this point, so this is for the best, I'm excited to mix it up with YouTube. >> Totally, yeah, I was bummed, Cody's sick, he's been burning it at both ends, you know? I heard he's been going to Lowe's, he's been doing chores around the house, he's been spending

a lot of time on X, tweeting about AI, unemployment, I guess it's like really weighing on him to the point where he's missing podcast episodes, but I think we're going to be able to kind of scrape it together here without him.

>> Are you guys worried about how much he's taking on, like I just, I think it's in, I'm wondering if

you ever think about this, whenever my house chapter ends, I'm going to like lay on a beach for at least a month and do nothing, and he is, he's like, you know, 47 consulting calls, he's doing landing page services, he's starting new brands, like I get it, but like, no break in between. >> I would be on your team and I would go much longer break.

>> I like visualize it during yoga, I like visualize the beach that I'm going to be laying on for at least one month.

“>> I think he's planting a lot of seeds though, you know, he's like, what do I want to do?”

He's like, do I want to consult, do I want to make landing pages? Do I want to start brands? He's like, you know, he's dipping his figure in a bunch of different things, but I agree, I'd be, I'd be hanging for a little bit on the beach waiting for that idea to strike, and then I'd be that at once that idea hits, it's all in it.

>> One funny thing you guys will appreciate is, do you know that Taylor holiday hates voice notes like with a burning passion? I didn't know that. >> Okay, so Cody and I are just yapping voice noting Taylor just to bother him. So that's what Cody's been doing in his unemployment is, yapping it Taylor. >> Yeah, yeah, yeah, yeah, yeah. No, I know Cody's got a bunch of cool stuff going on. I like the idea, I like teasing him about

him just like, him just being spending a bunch of time on X, talking with Claude using Fable. He told me he was going to lose more, let's really tell me the other day. So I'm like, he's got a little bit of time. He's doing more things in here. >> Yeah, typically does. But you know, I want to say one other thing, Olivia, you recently became CMO of House, and I'm just happy to have

another CMO on the podcast finally. You know, we've got Connor as head of growth, we've got Cody

assumed CEO position. Like I haven't had like a CMO to really, you know, talk about the role with in a while. >> Connor Rowling, your friend Cam just became a CMO. So we've been, we've been DMing on how we'll be sharing, sharing more stories. But yes, I want to, maybe for a future episode, I would love for you guys to put yourself in my shoes and say like, what would you do as CMO of House? Like one of the big open questions is like our page. Should we be doing page? Should we

be getting into the meta ads game? We haven't done much here. But I mean, we have a lot of creator content, including you guys. Should we be pulling clips from the marketing operators and boosting

“them? >> There is a path to that. I think this is a good segue into what I wanted to ask about”

next. Because it feels like the best like V2B SaaS marketers right now are doing things like great customer advisory boards, great. You've got the, what's the Hampton's day trip with shoplift and a couple other ones are like doing a helicopter into like a Hampton's House. Like it's getting, there's more dinners. It seems like that. It feels like V2B marketing has gotten really, really fun recently. You guys are rolling out a top marketers sort of accolade that you're

going to be giving out, which I think is actually like the best form of marketing you guys could be doing. Could you talk a bit about that? >> Yes, so I am so pumped about this. We are launching the top marketers of 2020/6 program. I'm spending a lot of time here. Think Forbes 30 under 30, but more objective and you don't end up in jail. No, no, no, no. To be, it all seriousness. Like I think it's, it's no surprise kind of based on what we do and like kind of like the level

of sophistication these brands are at when they, when they adopt incrementality. I think it's like no surprise that we work with some of the best marketers in the world. Like we are constantly told that the community around house is best in class. It's part of the reason people love working with us. In so many of these customers are like not on the podcast circuit. They can't be for whatever reason. They're restrictions. And so we want to celebrate the idea here is like, what if we

celebrate and spotlight the marketers who are driving real incremental outcomes in their business? And so trying to make this like more objective and not just a standard industry list where it's about who you know. And so really kind of celebrating marketers who are doing cool stuff. I think we'll have some like featured categories around AI, creative testing, velocity.

like fill out, you know, a whole application. But what's really cool is my judge who I'm bringing

“in to help me is Connor Dolt. He is my top marketer of 2026. And I told him I'm going to embarrass him”

and he was so mad at me. But he because he's he's going to be a judge with me. He can't, he can't be included on this list. But I just want to shout out to Connor because I have learned so much from him. Like the thing he is just so uniquely skilled at is he is able to kind of we between the big creative idea, the big swing and the really petty performance marketing finance world just in and out so seamlessly that I have just been so impressed and like obviously

groan's had an amazing outcome. He'll tell you he didn't have much to do with it. But but I disagree.

So he's going to help me judge here. And this one's fun. We have our customer advisory board meeting in person every February. It's looking like it's going to be at Pebble Beach in February of 2027. And a subset of the winners of this top marketers program are going to come to our cap event and hang out with with the customer advisory board. And there will be stuff for non golfers too. My founder is a big golfer. And so I told him I need to do this for him. But there will be a

non golf track. And so I'm just I'm pumped. We're, by the time this episode publishes, there will be a landing page. We're going to nominate, of course, some of our customers. But this is also open to the public. Anybody can nominate, like you can apply, you can nominate yourself.

You can nominate someone on your team. And we're excited to see what we get and just celebrate

“some of the amazing work that's been happening. Awesome. I think it sounds like a super fun program.”

I've got two points here. One, I would love for there to be a, not saying I'll be on the list. But for the lucky 26 who make the top 26 physical trophies, I think we need way more physical trophies in our lives. Like we all got him growing up playing, you know, AYSO soccer or whatever. I would love anybody to possess a 26 marketer of the year, like Oscar equivalent. That's one. Do you think we could get that done? Conner told he, when we were jamming on this idea, he said

there's a reason paper played awards have stood the test of time. And so his idea was a paper played award. But maybe we can upgrade that to a physical trophy. Okay. Yeah, I'll talk with him too. I'll give him a pitch. Okay. And then just thrown out a couple other names. Conner Dalt is a good one. I guess he's technically this call. If I'd Conner Rowling, pretty good. Conner Martin. Oh, my name and the hat. Come on. I've got Conner attia. Founder of

Cantara. I've got Conner Gallicor. I've got Conner Sunderland. I've got Conner Day. And I've got Conner Gross. Oh, some of my just really, I've got no bias here. I've got no agenda. I just think they're really your favorite artists should be considered. Yes, it's a casual. The CFO of Grings, it was also a Conner. I don't know if you knew Conner Statsney. I just recently met him. Yeah, they had two Conner's that explains the great outcome. That is wild. We could have a Conner track at our cab event.

“Yeah. Yeah. Who do you think should be on this list? Like, who do you look up to in the space?”

And, and I think one thing that that tends to happen is you can flate the person with the brand they're at. And I think Conner Dalt is a good example of, like, again, I think I think both there, like the both the team and the product, created that outcome. But who do you guys admire as marketers in the space, not just like the brands you admire, but the people you're learning from? I think Maddie Martin from cadence right now is just on an absolute tear. He's one of those

guys that's, I think, like transitioning from doing it all himself to, like, slowly hiring, like, leads for his team. But, like, they've been on an absolute tear right now. And he's been the one that has really been, like, not only driving that strategy, but I think also pushing a lot of the buttons, which is pretty impressive over the last couple of years. And now he's finally, like, like, he, he, what's absolutely in today is like, hey, we, we just hired a creator strategist,

hey, we just hired a DR, like, video editor. So, like, I've been very impressed with with him and what they're doing in spite of not having, like, a super large team yet. He's great. He was on the pod grade episode. Um, I go, uh, I'm a big fan of Nectar match, I've talked about that for a while. Scott McCloud is technically chief of staff for a while, extremely strong marketer, way more under the radar. And then we've had Jared Brody on the show, who's a VP over there.

They're great. I also love, um, the back to the B2B point, which becomes a little bit more meta,

but, um, Robba, who have never, like, spoken to, but, like, crushed it from a crushed

it, triple well. He seems to have been highly influential in the way that some of these, like, e-commerce ass vendors are, like, deciding to bring things to market. Um, so I, I, I, I'd consider him. Keegan takes from David, the VP of marketing, X McKenzie just coming in there, figuring it out on the

Keegan's the man, we're, we're an aspen with him, uh, really good. Okay, this is cool. Well, when, uh,

“when we, when we launched this, you can formally submit, we'll review the applications, and, uh,”

and we're going to do a, like, a special kind of open house episode, uh, to reveal the winners. Motion just dropped their 20, 26 creative benchmarks report, and it's been getting shared everywhere. Slack channels, LinkedIn, Twitter, sharing it in our private group chats. And it's great because everybody's been asking the same four questions forever. What is normal? How many ads should be actually be shipping? What is a healthy hit rate? And which formats really win? The report

analyzes over 575,000 creatives from 6,000 advertisers in over a billion dollars an ad spend to answer these exact questions. And the report has some really interesting findings. Like, the fact that only 4 to 8% of ads actually become winners in over half of ads actually lose. In promotion customers, this report is especially helpful. You can upload it into your motion dashboard with their runeth AI chat and compare it directly against your vertical benchmarks.

Hit the link in the show notes. I promise you won't regret it. And as always, go to motionap.com

until the marketing operator sent you. It's funny having you on Olivia because we talk about almost every episode is an income mentality episode, but this one's a special one. I finally get to be here and participate in the conversation. Yeah, yeah, 100%. Um, okay. So I want to talk about like the way people are using income mentality today. I've kind of got like one of the shifts that we've had in strategy over the last couple of years, but maybe just you could start us off with, you see

more than anybody else, a number of brands, not only like implementing income mentality and measurement strategies today, but also how that shifted over the last couple of years. So maybe you could give us just like a brief state of the market. One of the questions I had was, if you've seen any interesting changes and how people are implementing these strategies. So some sort of like wide scope as far as like where does the industry sit right now, as well as like if there's any

interesting trends that you're seeing over the last few months? I talk about our house journey

in two chapters. Like the first journey or the first the first chapter of house was like convincing

people that incrementality mattered. This was, you know, 2021, 2022. That term was very niche. Like it was reserved for the Airbnb's of the world, the Ubers of the World. And Zack and I just like weren't even sure about the tamp here of like a do enough marketers kind of like care about this to go evangelize it in their organization. And we're through that. I mean, that is like very, very clear that you can tell that, you know, everybody's running incrementality tests right now. And we've entered

chapter two, which is like, how do you create an operating system around incrementality testing such that you are using it to improve business outcomes? And we just we just published a case study about this with Gamebridge this week around like, how do you string a series of tests together

“to actually drive growth into your business? That's what we're all after, right? When you talk to a”

lot of brands, nobody's really interested about the idea of like just cutting a bunch of channels and getting more efficient, although, you know, if your goal is, you know, like a PE exit, like, yes, like we've had a few of those. But for the most part, people want to see, they want to unlock growth, like they want to drive growth with this stuff. And so that is what we've been working through with our customers is like, how do you actually use incrementality testing to unlock

growth in your business? And that is, that is like, why we have a services component to our business. It's hard. It's like, really hard. What do you do in the number of disagrees with something else you're seeing internally or you need to turn ads off in part of the country for some period of time to get a precise read? Like, it is, it is certainly not for the faint of heart. And so that's why we have this, like, kind of software plus services strategy to like make sure you're actually

getting value and you're getting ROI. And you can point to these tests and say, this impacted my business in a positive way. And, and so that's, that's the big shift that, that I've seen at least is, you know, they're, they're thinking about it more as like a kind of culture of continuous experimentation than a, than a report card, if you love. One of the big shifts that we've had over the last, it was this time last year where, because it comes from two things. And I hear

“this all the time when speaking to people one, we get a number of tests that, I think it's really”

hard to, you know, perfectly measure incrementality with the whole that in terms of like getting a direct, I rose. We run, and I talk about this all the time, like relatively short tests. And for that reason alone, we're just in like a miss sum of the value that's being created. So like trying to like scorecard and like grade some of these channels on like a three week test basis is like

if you, if you, you know, widen the aperture on our business where like spending more money we're growing profitably over your, like, like fundamentally everything's happening the way that we want it to. But the way that we test often leads to like some relatively conservative

readings on the impact of some of our channels. So that's one point that I'll make here. The second one

is when we would prioritize like, okay, let's just get, what is the incremental rowas of meta, we would just be getting it for a snapshot of time. So not only is it just for this three week period, maybe you're getting a one week post-treatment window, but it's also just February 2025 and like that's not super applicable moving forward. So where we've shifted and I'd love your perspective is treating incrementality testing more as AB testing and we have direct actions that we can take

from that. And what I mean is to say, the, the cleanest example, when we have many of them now, because it's almost exclusively how we're running our, our measurement program is going into BFCM, the three weeks beforehand, we launch incrementality tests on our top two channels. And our

goal is really to just get the read of which one is driving the best incremental dollars at its current

scale. So when we need to spend more money as we do going into BFCM, we have really up-to-date data as to where our dollars might best be spent. That doesn't tell me the I-row as of YouTube or the I-row as of meta, but I feel very confident that we are now making a better data driven decision around where is my next dollar best spent driving incremental results. So it's like a shift that we've had

“and I think that's a, I think it kind of maps to what you're describing where we started out”

and we went channel by channel and we measured and now we're thinking about it in a more like progressive way. And I think of it almost like, yeah, directionally, we want to be taking action from these things. Do you think that is more of a ridge-specific thing or are you seeing that across the board? I love this. By the way, I'm really glad we're having this conversation because a lot of our customers listen to this podcast and I think there will be kind of good tangible advice for folks who are

listening here. I heard you talk about this on a pod and I was talking to the team, do you think, I love it, by the way, the like AB, I think that's very actionable. There is a very clear action after that test on this thing is better than this. That's roll it out as the winner. Do you think that you're there because you've run? I don't like, you're one of our highest velocity testers. Like, do you think you're at this stage because of the maturity of your program

or like, if you could go back, would you have started here? Like, I know you've started with channel baseline. Like, give the audience maybe some advice here on like, do you think this is just a function of like, you've answered a lot of core kind of baseline questions and now you're moving into optimization or you actually think this is maybe where some of these brands should start. Yeah, when we started, we went channel by channel. We measured the incremental rows of each of those

for these snapshots of time. And there was value in that. Like, we found, you know, non-branded search wasn't as incremental as we thought it would be. And that's very, I think, particular to our business. More people search for ritual every month than they do for men's wallet. Like, there's just not a lot of in-market demand. And I think we have to be very careful about our non-branded search, not capturing existing intent. So anyway, so we go channel by channel. That's kind of how we roll

“out those tests. And I think that was important. And we got a lot of value. And then I guess what I'm”

describing now, our strategy shift does come from maturity. I guess what I would say that in hindsight, you know, I would love to just be able to show them be like, hey, we need to drive a two-x incremental rows. Like, that's where we're profitable. That's where we're going to hit our goals. Let me just go channel by channel and figure out whether we can do that. And I just think it's a little in hindsight. I think it's a little bit more complex in that. So I do think the channel by channel

is a good testing strategy. And a great way to kind of get the ball rolling and how to leverage geolift studies in how you want to approach measurement. But at the same time, like, I have found it a little bit more nuanced or tricky to get like a true readout given the snapshot and time component given. The snapshot and time component in terms of like the time of year, again, February for us is way different than June. And then also the fact that we are measuring some of these channels

on like relatively short timeframes when, you know, 40% of our customers every day say they've heard about us for over six months. So I'm like, it's just like a lot of nuances there that I

“thought would be able to better account for. And I think that's part of what drives this strategy”

towards this more like AB testing directional approach. Can I add a note on that? Because I think a lot of brands like come in and they immediately start running hold out tests on their hero channel. Like right, like straight up hold out, right? Like like classic AB, no spends spend. And they're trying to measure the incrementality of meta as a whole. I don't think that's a good

first use of your tests. Like if you got to become a to a size where you're on boarding

with a house, you know, presumably you're you're probably like a healthy eight figure brand,

think you should save those tests for a new channel expansion. But I think I think the move

is to spend your first like six months with house actually optimizing meta with with hold out

tests using the like directional approach that we're talking about. Not like a true channel hold out, but like how do we unlock more incrementality out of meta? So like let's start testing you content optimize stuff. Let's start testing like maybe running holdouts on more upper funnel

“brand forward video creative. Like I think you should actually start with those like directional”

tests inside of your hero channels. Because how many how many times have we talked about people expanding channels to early or like we don't think they're getting all the juice out of meta that they can be getting. Like I think you you think you actually should start there to optimize meta because it's really hard to to test all those things without auto to a like house. And then go into your like channel expansion like YouTube what's spent 10,000 a day versus

no spend at all and like actually measure the the true incrementality of the channel as a whole. But if I could go back and start over I would I would have waited like half a year or a year to do a lot of those channel holdouts because now that we've done all this like marginal frontier testing to figure out like of you content super incremental for us like all these like kind of smaller things in meta or super incremental. We've a way more optimized meta account now

“sitting here today than we did a year and a half ago. But we didn't start there and I think a lot”

of brands should figure out how to use house to maximize their hero channels and then go into like the YouTube holdouts of the CTV holdouts of the TikTok holdouts and kind of like do that channel wide holdout and then like and then you're back into the like once our YouTube's incremental now you're back and like the marginal frontier like directional testing again. So I think that that's one thing I would change about how we did it at Hacks Cloud is actually starting with the

like the art three styles where's the next dollar best spent because I think those are some of the those are the best wins and learnings we've gotten in the last year and a half and like I look

at our account and we're spending millions on view content now we never would have done that if we didn't

have that that like directional data telling us to do that. What I could also hear is that you're almost best starting with the channel holdout of meta the same way that I just described we started at Ridge then you're doing all of these tests around tactics and figuring out view content or our video versus image you guys have a great great case study on the different incremental impact of ad formats etc etc. And then you can go back and say okay now with all these new

learnings we've got this new implementation let's test meta again and you could measure that improvement but even then that to me now feels like a directional approach where like what I would care about is improving meta and at no point in my like am I hitting my two X incremental row as target

“because I think that I think it's hard to do but do think you even need to start with the channel”

holdout like if you're if you're doing 30 million a year and like meta got you there it's like

all right we're 85% meta you know 10% non-brand search and 5% branded search like you know meta got you there it's like it's not you know do you even need to do that channel holdout can you go right into the let's do like 30% more purchase conversions band versus 30% more of you content span versus VA you spend like can you just go right into that more directional testing do you think Olivia I'll tell you why teams will often just come back to this is when you launch new

channels you need a reference point to compare against like if you're you want let's say you're launching YouTube you're launching some of these new channels the next the next question is how does this compare to meta and so what we found is it like they end up actually coming back to the baseline meta test because it is such a big it is such a big pocket of spend and they just want to understand for these new channels how does it compare relative to to my investment in meta and

and then they understand how to reallocate so and then there's there's another benefit which is that getting like a channel level read can help tune your MMM so we've seen a lot of like I remember Cody Cody did not start with meta and then when he started working with with us on MMM we were like hey we're actually blind to like the biggest channel in your business and we just need to get a hold out on this because we need this for our MMM so that's those are two reasons why

fine folks are are coming back to the channel level read is because they just like they need a point of comparison for the other channels but I'm I'm with you guys the optimization tests have been and we saw this also with Jones Road is like the the how do I improve meta like the optimization test whether it's spend level whether it's tactics whether it's incremental attribution or view content or some of the mid funnel stuff like those are where the winds are happening

but it it sort of like Connor McDonald you said this when we when we kicked off the conversation of your your your your geo holdouts in your channel level reads are a bit conservative like I do wonder if and this is we see this all the time with Bummer News like a lot of teams

and that is I'm wondering if like it's just a function of like where you guys sit in your org

where you're I mean you guys are you know you're not found or level but you are very entrenched in these businesses you have nothing to prove at this point I think like you've earned you've earned your stripes like for a team where you have a perhaps a CMO who's new do you do you like that what you just described of like the absolute Iroas just feels conservative for all these reasons mostly it's a two week test and we know there's probably compounding effects over time like

do you have any advice for for for marketers like maybe in the position where they have someone to answer to and they're running these tests and they don't look as rosy or as like you know optimistic is as they as they want it to and and and and for that reason like would you recommend they just go straight to optimization and not even worry about the channel level measurement just like any advice for for teams on this quick gut check for the operators listening if you're spending on TV

or CTV today can you actually say what it's driving incrementally this is exactly why we work and have worked with neon pixel and hexclad for the last three years we've grown with them a lot over time in CTV has become one of our top growth channels they help us treat premium living room TV like a real performance channel it has smarter household targeting it has suppression of people who already know us it is a very robust analytics back end so we feel really good about the measurement

and ultimately and most importantly it is a strategy that is built around incremental growth not just claimed attribution we look at incrementality is the true north star on measuring channels and they're really measurement that agnostic they don't force us into their own black box dashboard they work inside the measurement systems we already trust and help us understand what

“TV is actually doing if you want to check it out go to neonpixel.co and ask them to design a controlled”

CTV test on your numbers I think that no matter like where you start whether you're trying to

get that like um the clear i row is we we always say like raw i row is um or you're you're

opting just for the straight like directional sort of progressive approach from a leadership perspective it's best to just commit to this becoming an ongoing system of measurement and even if you're just getting that that channel level rose which like you bring up a good point where it's like you want to run channel level rose like just to calibrate your mm it's like there's never there are a million reasons why you would want to do that um at no point would I say that should be treated

as a scorecard and that in and of itself someone should then decide I'm gonna go spend less on meta because it looks bad or my team is not good so now I need to like go figure something else out like that's not really I don't think it's it's very rarely like the right way to interpret those

“results so no matter like what your actual testing methodology is um the best thing to do is for the”

whole org to be aligned and like this just being a progressive form of measurement um and that is why if I were the CEO of house I'd be going on like podcast and I'd be talking about this sort of thing so that like the industry can kind of come around on it because it comes up all the time um

and I think there's a lot I'm sure you see it I'm sure you've had this conversation a million times

someone onboard someone test house someone gets one bad readout and then they're like what the hell am I supposed to do from here and I think that comes down to like an education piece and a leadership piece of again just committing to that progressive approach I was doing some user research um on a totally different product that we're building and one of my questions in that research was like what are the biggest fires in your organization in your role and someone said

actually the biggest fire is when we get a bad house result and that just like broke my heart like oh gosh like this is supposed to provide clarity and it's supposed to help and when it when when when these teams experience bomber news it's like a lot of my job is like coaching on how to think about it and kind of rolling we were talking about this when we were together like a lot of a lot of teams with the right mindset understand the way in which uh quote

unquote bad test result is actually a great learning and I want I just I want uh it's almost maybe a problem at the executive level like I want these teams to have the trust and the buy-in to be able to celebrate the failures as much as as the wins uh because it's it's really

“important and it's it's uh it I think one thing you said Connor that that's helpful is to think”

in terms of relatives versus absolutes is we've we've come around on this is like the absolute incremental return is probably much less important than the relative performance of these channels of these tactics over time and as long as we are moving dollars and reallocating from

and you see that like Aaron Aaron Zaga from Newton uh I just recorded a podcast with him

“he his bias to action was just like it was so impressive how he didn't he never like second”

guest or result he just made a call and moved on and went to the next thing and it just no kind of like you know sacred cows and that like oh this this channel you know there's there's no like kind of channel manager mindset where you have folks who have like specific kind of responsibilities and I've seen the magic that happens when you do kind of just like look across the portfolio and you manage that way and I you know I want I want to make sure that that teams don't get like stressed out

about the quote unquote bad result totally and the relative versus absolute is exactly what I was trying to get out when I say we've moved to almost an AB testing framework where it's like I just care about the results relative to one another and then the other thing is I just want to be making

slightly better decisions as often as possible like that's the goal of the whole program it's never like

it's never a score card it's like can we operate can we operationalize this can we get as many readouts as we can across the dimensions of the business that we really care about and therefore can we make slightly better decisions and just understanding that like that progressive approach is going to push us in the right direction and it's not perfect by any means and it doesn't if we're if we're you know moving away from this idea of absolute we're also never getting a clear readout

at least it ridge unlike what is meta doing every day it's just a matter of like do we feel that our media mix is slightly better than it would be otherwise um and anyway that's like again

“last year I think we'd really kind of turn the corner on that line of thinking well there really”

is no like bad result I think we're we're saying like quote unquote bad result but as long as you

as long as you're allocating dollars better based on that result like is there really a bad result quote unquote like you know the example the most recent example we have related to what you just said Olivia as we were doing more non-purchase conversion testing on meta and we were doing the same three cell setup we did when we when we tested against you content and we saw that these these two new objectives did not they were not more incremental than are then spending more up

into purchase conversion campaign like you could argue that like service all about to bad result I would have loved to have unlocked you know traffic and reach and video views and all that improving that putting more spend there is more incremental than more send into purchase conversion campaigns but at the end of the day we still learned a ton and now we turn those off and we're putting more of our meta add dollars into a better spot because of it so is it a bad result if you if you

action it quickly I think it's only a bad result if you like sit there and pontificate on it for a month and a half and you don't go and optimize your mix based on the data that you got so it's like what is a bad result well the bad result is only a bad result if you're not going in you know

“it's like if you want to see a road test but it takes you four months to implement that the”

winning variant on your website like that's a bad outcome even though you found you found a winning variant it's the same thing with kind of sitting on your test results whether not the new thing you tried was quote unquote better or worse than than the historical thing um I want to ask you a question to them curious how you educate customers on this we talked about like the doing the true channel holdout on meta and if you're a brand that's been spending on meta for two years three

years has gotten to got you to to where you are that's not actually a true holdout test right because even the cells that you're with holding spend from those demographics have been exposed to meta ads for however long you haven't been running holdout tests so technically even though those cells are not being exposed during the duration of a test they were exposed beforehand right and for a long time beforehand so it's not like those cells those demographics haven't seen

your meta ads they just haven't seen your meta ads while you're running this test so again that's that just goes back to what we were saying earlier like even that is not a true net read of the channel right and reality you'd have to run a holdout test for like a year or longer maybe to get a true net read so do you guys educate your your customers on that when they're running some of these early tasks and it's like oh wow meta only got us a one five X I row as I was like yeah but

the cells you weren't putting you were not exposing spend to like they were before this test ran so how do you how do y'all think about that and like just making sure your customers know and like are aware of that kind of phenomena yeah that's a great question I've heard you guys talk about this before so you have let's say you know you're spending nationally you've been spending on meta for a decade and then we're running a holdout test and you're saying that holdout group

that we're we're turning off ads has been getting ads for 10 years so like they're not actually it truly held out most brands treat customer support as a cost but with AI every support conversation is now a chance to increase retention recover revenue and grow your LTV rich

inside your support you can test different windback offers on refund requests subscription

“save offers during cancellations and product recommendations that turn chats into sales plus”

they guarantee you're going to save money 50% of your support volume automated in 30 days or your money back that's pretty good offer if you ask me so at a rich panel dot com slash demo they'll build an AI support team live for your business and show you how it can drive incremental revenue for you the reason that's okay is because that dynamic is happening in both the holdout and the treatment where like those people who have been exposed to ads for a decade they exist in both the control

group and the treatment group so what it's doing is like it's inflating the baseline of both groups and so then what you're measuring is the incremental effect of this new treatment group

getting ads for a short term like for these you know three weeks or or whatnot so the long term

effects of the users who have been exposed to ads for a very long time that is present that is happening but that's happening in both the holdout and the treatment so it would just kind of like lift up the baseline of both groups but what you're not answering I'm an again what you're not answering in that question is like what are the long term effects of ads and this is why I've been talking a lot about like if the hypothesis is that these ad effects compound over time then that is

a reason to run a longer test and so that's that's that's that's come up with like you know the more upper funnel channels like a CTV it's again it's just kind of a lining the test design to the hypothesis like if you believe that there's something happening around compounding effects

“over time then you need a longer test to measure that so that's that's how I would think about it”

and that's been our approach that has clouded with a lot of the testing we've done on like YouTube and CTV like we are running we were exposing these cells to holdouts for like two and a half three months sometimes with an observation window that can last three to four weeks and it's kind of painful to let those tests run that long at times like I wish we

at times I'm jealous of the brands that have you know like a $60 first order AOV and they're like

sales cycles a lot shorter than ours but at the same time it's given us so much confident confidence to scale up YouTube in CTV and some of these other channels that I just don't think there is I think there's a lot of ways to kind of sus out incrementality without a tool like house using you know like GA4 or a server side analytics tool or north beam I think there are also a lot of tools a lot of channels that you just literally cannot do it without a tool like house I think

“CTV is a great example and now we're it's CTV is one of our fastest growing channels in the”

last three years no way we would have been able to have that level of confidence without like a like a three month holdout test with like an observation window that ran through a sale like we just have to do it that way but not that's not everything like some things we don't have to do with that long but like CTV in YouTube channel holdout test like the classic example of you gotta run those things for a long time and just be patient if you're gonna get a lot better

more actual readouts if you can be Connor McDonald you because you you mentioned you guys run two week tests how do you think about the reality that different channels have different payback windows so for like a YouTube for example are you comparing your two week YouTube readout to a two week meta readout because this really trips up customers of like yes they're on an even playing field in terms of length of test but it might be the reality that just certain channels have different

dynamics in terms of payback window we run we mostly run two and three week tests we are just about to launch a test today that will be six weeks so you know we're getting there we're coming around to a month plus long test we're looking at those we're looking this is a this isn't example just of like how we ran the test we ran meta and YouTube we measured both of these we looked at them directional to one another we were not um thinking about like well what does this mean

over the next six months or something like that like what is like the long tail of YouTube value um that might be because we are optimizing for relatively short term outcome for we said we really want to figure out how do we best deploy the next dollar for the week of BFCM and then it sees longer tests like the six week tests we go live with our sweep six today just super exciting we have Tony Hawk that's a part of it which we're super stoked on and so we're going to do a

full six week test on some of the upper funnel channels around the branded Tony Hawk content and the point being like what I would like at the end of this is a data point that we can point to to say like is this type of partnership is this type of content is this media buying strategy like does this have a larger impact over a longer period of time so it really just depends for us on how we want to be looking at it we have examples of both we do air on the side to talk about all the time we

knowingly are making some compromises when we're not considering the total long term impact

of some of these different channels but there are some exceptions to that as well. What I appreciate about your approach sometimes is like you don't overthink it progress over perfection like you could find places to poke holes in all of these approaches so you know sometimes it's just progress. Can we talk real quick about like the one of the big challenges is on operationalizing like a system around incrementality we've been we've been thinking a lot

about this at house we've been adapting our product roadmap accordingly of like geotast moment in time snapshot some of the challenges that that you you've voiced. How do we better connect

“these tests in the measurement back into kind of like buying decisions that you need to make?”

So I've been thinking about this as the macro and the micro decisions that need to be made so at the macro level this is like channel budget allocation like where am I going to allocate budget at the channel level maybe this is weekly or monthly? A lot of what we've been doing is like building this into the MM where you can do this it's like kind of clunky to do that only with experiments where it's like I have this meta read and I have this other read how much should I move from one

channel to the other? And so the MM is all for that where there's like a nice like UI scenario planner

where you can like ask what if questions what if I reallocate a million dollars from meta into YouTube.

The MM also handles non geosecmanable channels which is tricky like affiliates and podcasts and whatnot and then you get these like really nice clean return curves of like exactly how much budget should

“I move whereas that was just kind of fast and loose before and then on the micro level this is new”

the next question from there is okay then what campaigns do I put the do I do I add that budget to like when within YouTube like where do I add that budget and so this is why we have built a pixel it is now live you can you can add the house pixel and what we're doing is where where the the value so much of the value of attribution is like the real time stream of data like we need to know what's

going on very quickly you know at a granular level even more than daily hourly and so by having that

real time stream event event data we are we this is this has been like two years in the making we've built a machine learning model where we're trying to like de bias the attribution data with our experiment database so they have like you know thousands of experiments we're de biasing that attribution data based on the cross-customer database and then your own experiments where you run them and that is our way of saying like okay so what campaign do I put this into

we just historically like you kind of use a different system to figure out how to do that but a lot of our customers have said like well I want to make that decision also like rooted in incrementality

“and so that is like together I think the combination of the more macro and the micro is what's”

helping these teams kind of operationalize my next question for you is like show to human be doing that like so like that just think about the you got experiments you got MMM you got attribution we are attempting to solve the triangulation problem of saying like here this is all kind of calibrated in in causal truths by your experiment data but like do you want to log into different views and try to like kind of you know do you want to

do that as a human of like triangulation or do you think a system should be doing it for you agent enter a genetic media decisioning thoughts we look at house is like this outer ring a measurement and then like after you you get the the read out on this outer ring a measurement you're trying to create a relationship between that most outer ring and and a more inner ring that you're using on a more day-to-day basis so like the easiest example that we talk a lot

about on the podcast is like creating these these multipliers between different measurement channels so it's like hey we know that for the duration of this test that we ran on view content we like the iro as read out because it was more efficient than the same amount of spend in a purchase conversion campaign great and then we're saying okay like now how do we take that data point and in like to your point Olivia operationalize it in our daily weekly monthly decision making

well let's create a multiplier we know what the iro as was during this moment in time we can also go into whatever MTA you use and say well what was the row as or crack or whatever like in platform MTA match you care about and now we create a relationship where we can say okay if if we're at this in north beam we know that that maps out at least in this moment in time to this iro as we were comfortable with and now that becomes the the number that we're moving

to kind of recalibrate what that like row as or crack number should be in north beam and you could

do it with I mean I think there's a that's just one example I think you create a multiplier with a lot of different data sources um now should a person be doing that or should a row of be doing that I think like yeah over I don't think we're there yet I think it's very much the same as trying to take like a American UGC out and say hey make this Italian and like get it in the attic out like it's um I think we will get there I just think there's a ton of just so much context right that you

would just I think the the role will turn into great how do we take this context and like give it to this agentic media buying like how do we make them know that up point two one day click row as is actually winning an incremental according to our holdout test and then how can we like set up rules to make that machine move budgets up or down based on that but then times what hundreds of tests and

“like hundreds of like nuanced pieces of context I like I think that's what the complication comes in”

but I think we will get there yeah and I think ultimately it will be better than a human buying

that media and like doing that triangulation themselves but that's how we're doing it internally I think we're fairly far away from being like yeah like what's this like an agent buy our media based on all these rules but I do think that's where we're headed and I don't know when we'll get there whether it's in a year or five or ten but I do think we will get there yeah the two big qualms that I've heard when we when we talk to brands about this is it needs to understand

all of my business context like cool house you have a lot of our data but like hey we're looking at post purchase data we're looking at our promo calendar we're looking at our product launch schedule and my response to that is like cool like we can take that like that's that's part of our MMM already like we're already ingesting a lot of those is covarious covarious so like that feels solvable in terms of like how do you get the system enough business context such that

it has everything it needs and then the other big qualm and objection I hear is like it needs

“to be explainable like you need to be able like it needs there's almost like a sense of safety”

of like if I'm going to hand over the keys to the system I need to understand why it is making these decisions but that's also like that is sort of the the transparency around the experiments is what's helping us like it's sort of what's helped us with the MMM is like you can see a beaker and you can see exactly where an experiment is informing the call so like those feel solvable short-term counter McDonald like I guess I just I'm going to I'm going to keep pushing

on like the why why is this why why can't this be done by a system people are the most important

part of your business and finding real a players is hard and incredibly time-consuming that's why I want to tell you about propel propel is honestly the best recruitment agency that I have personally work with they are marketing and operations specialist say source talent from US, Latin and beyond for top pcom brands like Jones Road like many others that you know and love at Jones Road we've hired over 12 people through them we've hired video editors graphic designers we have multiple

retention managers creative strategists from them what I love is not only they easy to work with their quick but they work on performance so you only pay a one-offie only once you hire somebody you don't have to pay you know any retainer no-on going fees plus every hire comes with a three-month guarantee so I highly recommend them they've worked really great for us we've a really great team that they've helped us source I had to link in the show notes and submit the form to get 10% off

“your first hire I really like to phrase debuyasing I think that's really good and that's like”

that's a great way to try debuyasing the MTA data is how I interpreted that we're like with the pixel we're seeing a certain result like a rose and we're debuyasing it based on all this additional information we have via incrementality test that is theoretically what my team is trying to do all the time where we say okay we're getting this result from this meta campaign and it's a view content and the roses really low but based on all these previous tests we're going to

debuyase and say we're comfortable spending on this campaign but would you say that's the same Connor brought up developing incrementality factors it's really just like is debuyasing like more granular incrementality factors and it's automated and it's the next so the one of the learnings we had from incrementality factors is they work well but they're very blunt instruments so just like a lot very blunt so the model takes into account the the model that's doing the

debuyasing it's automatically applying these incrementality factors but there's a lot more that goes into the factor than just your single experiment the most recent one it's all of your experiments that you've run over time you know recent ones are weighted more heavily but also moments of

insights it's also it's also your investment levels so there's just like a lot more that the

model is tuning on then one single like static experiment result but yeah you nailed it that's exactly what I meant when I said the attribution product is like debuyasing is we are doing kind of like a souped up version of incrementality factors in an automated automated way and this makes total sense to me and like I said it's what my team is trying to do all the time and you can imagine it you can imagine how far we are from being perfect in that like

like do a crazy degree I mean I just I've mentioned early we run all sorts of tests like we see significantly different incrementality factors between men and women within our wedding bands line we see very different incrementality factors between view content and purchase optimized within meta we see a difference between video and image so like all of a sudden if we are running you know a video view content adds to women on wedding bands like we're supposed to be like

considering all this context when we decide are we spending more dollars on this campaign or not we are not even close to being perfect with that like and I know that for a fact like

“the only thing that I'm somewhat comforted by is I know that we are at the very least”

working with more data than the average brand and I think we are considering some of it when making these decisions and therefore we're making slightly better decisions but if you told me and said hey we're going to consider all these tests that you've run all these different dimensions and attributes of the ads and some of the other things that you mentioned in terms of abysing there isn't a doubt in my mind that a system could do that better than a person that's

like my general answer here what I will say just to hit a couple of the other ones that we that we hit on you know this idea of like geolift calibrating mMM and then doing more granular optimization with mt8 like makes total sense to me in like a scientific way in economics way where it's like yeah like if you do the model like that all makes sense and then it does it feels like historically

there's always been like more nuances in terms of context and things like that so I guess that's where

I would start from an economics perspective economics treated self like a science I got an economics degree and it's like yeah you don't have rational actors working in it like the models never play out the way that you anticipate in the way that like a physics problem might so I've always felt like we've had much we've been working in a scientific way but with much sort of blurrier lines um if we can begin to you know account for some of the additional context

and the devising I think that goes a really long way and then um yeah the other thing that I would say my team is doing often is looking at things like inventory like our promo calendar um like you know there's this there's something and this is maybe the one point actually I'll say that quickly the promo calendar um the business objectives we can instill that and I say that with

like meta we talked about this at the meta performance marketing summit we'll always set up our

ad accounts in our campaigns in a way that aligns with the way that we want to sell our inventory

“like that's what we're doing as a business and I imagine house and anybody trying to approach”

a genetic media buying will have to accommodate some level of like guide that marketers have to put in place because we can't just operate this like metatry to this way this like economic experiment or it's like hey just give us your ads give us dollars and we're gonna spit them out on the other end like that never work so it's like just a matter of sort of containing some more of these constraints I have no issue with that that seems like it's it's pretty clearly going

to be adopted from like a technology perspective and then the only one that like maybe I have some sticking point over is like long term business objectives where we might have some sort of very specific um goal for a very certain type of product to the degree that like we're actually going to make a decision around it that is um in conflict with the way that we otherwise would but like at this point like I'm really getting down to like a very fine detail so my answer to

your question is like it feels like we're not that far away from a really large percentage of what we consider marketing or media buying or media allocation being done far better by a machine and you nailed it with like what your team is trying to do it's it's about the signal right it's I think it's why you might not trust like a meta agent to do budget allocation is because you're like well it's it's using it's data it's using like platform data and I don't care about platform data

“so this is I think like if you trust this signal that's powering like the foundation that's”

powering it then then this becomes conceivable and if you think about like financial portfolio management like this is already happening like why doesn't this exist in marketing and I think you said like rat like you know sometimes we're not all rational actors like there are these things that are

billboards on you know the the 405 and but you can we this is and this is what we've done as we like

work that into the system of like what are the exceptions that we need to work into this system that are specific to your business and that nuance but it's in we're we're spitting out and we're like we're not claiming to automate media buying there's so many so many components of this the the part we're starting with is budget decisioning and we're spitting out these recommendations and we have brands now like approving rejecting and then there's the next step of like autopilot

like do you want us to go implement this for you based on APIs into the ad platforms and we're we're not there yet still like more of a conversation before the implementation happens automatically but we're starting to see like that like a probably handful of brands accepting these recommendations but it took a while to get there in terms of trust and just like does this have enough context

“to make the right to make the right call but it's pretty cool that it's it's happening I think”

it's it's moving in that direction and I just don't hear a lot of good reasons why this can't be automated. Does the model still work if you have like a handful of channels that you're running a bunch of holdout tests on but then you still have some over here that you have and like let's say you've ran a bunch of meta tasks a bunch of like intro channel meta optimization tasks like static versus images you contend versus all these things and then like you've done some

testing on other channels but let's say maybe you've never been a holdout on like Pinterest TikTok

YouTube but you have on like CTV like does does the does the agentic recommendations still work if you've only ran holdouts on some of the channels but not all of them and is that where like the business context and all the other data sources comes in it's like all right we don't we haven't ran a holdout test on CTV but we do have your survey data we do have the in-platform report of data we do have like UTM data from the short link here overlaying on the creative like how

how does that work and a tool like if the one you're describing it yeah we you know we to your point like we have the pixel data we have the we have a lot of this data and then we have the cross customer experiment database as well that's informing the initial kind of like we call it like cold start estimate for a channel that you haven't tested and then this is what I this is this is my favorite feature of both our MMM and in what's going to be built into architect this is the

name of our agentic decisioning system but we show you where we have an experiment to inform the recommendation and we also show you where we don't and then there's like a confidence interval associated and so there's this nice flywheel where it's like if you don't trust that recommendation

“like we get it maybe you should run a test first you know because they were like here's what's”

informing this recommendation and if you don't feel like there's like robust enough evidence then that's a really nice flywheel back into your testing roadmap of like okay we're going to test this next but but the the incrementality index that like collection of experiments across customers is also helping to inform some of this as well okay so your aggregate data all the all the

other tests you ran on CTV for all the other brands can also help the brand that's never ran

that CTV holdout test exactly very cool before Q4 hits the smartest e-commerce operators aren't just asking how do we spend more they're asking how do we make every visitor more profitable that is where after self comes in after cell helps e-commerce brands increase revenue per visitor contribution margin and incremental profit by optimizing the highest intent moments cart check out post purchase and even the thank you page brands like hexclad rich and pillar increasing their AOV across

“every cohort using these zero risk upsels we've tested and they have no negative impact on conversion”

it's just more free money in your pocket and with rocked thanks you can offset your after cell subscription no redesigns no risky funnels just better economics want to see how much money you can make with after cell go to after cell dot com slash rocket revenue forecast put in your numbers and see how much more profit your funnel will make before bfcm starts we started our like I don't know it was probably four or five years ago at this point we're rich wanted to approach

like better forms of measurement because we're running a really low M.E.R. and we were getting to the point where it's like if we can just find the wasted spend here it's going to be like extremely additive to the business and we started with MMM which is what you're describing which without theoretically without any geolift tests the MMM will try to figure out the the the rows of each channel but like what our experience was was that we just had these really large

confidence intervals and they could run these weird experiments where you scale up you scale back down really quickly and it's going to try to tease out like what was the effect on revenue and therefore what's the effective incremental rows of this channel we just found we had these really

better informs what CTV might be doing and then also running geolift tests geolift tests can then calibrate that model indirectly inform it as like what is this doing for your brand and

“yeah I think that's an exciting place we always joke um we don't use an MMM currently we use”

my mental model which is just like we're we're taking in I'm taking in the data and we're making some gut decisions over here um and that's really just because we've had a we've I think we have an odd business with these four different product lines and and channels within each of them

so it's always we haven't had a great tool for like the bespoke nature of our business

and I think we're also getting to a place where MMM is five years ago I think we're like kind of one dimensional it's like hey we're gonna take in revenue we're gonna take in channels we're gonna model them out that way and I was like our business just says a dollar spent on medic can be spent in like eight different ways I'm like I just don't think we're gonna capture the nuance that we need to and it feels like um the tools today are just becoming more bespoke or customizable

we can set it up in a way that better aligns with like the dimensions that we care about within our business I understand why MMM's get a bad rap like I one of my big learnings over four years of house is like geo experiments are the best we have there is a counterfactual there is a holdout

and there's still noise like marketing data is so noisy it's so messy like you know the amount

of conversions relative to the amount you're spending is just it's like very hard to find signal

“and so why would you expect an MMM to be able to to do that like so that's what just I'm like”

I don't need like there's based on what I've seen and confidence intervals around an experiment I don't understand how you could not have just gigantic confidence intervals around a model that is not informed by an experiment because there's just so many this is what our scientists will say is multi-cliniarity there's just like so many confounding variables in any given moment like you're dialing up all of your channels at the same time you're starting a sale like how could

a model actually disentangle which of those things is causing the sales so yeah MMM's it's tough

but the you know of course the best we can do right now is we can debias with with experiments

I like to debias in turmoil I hadn't heard that and I'm like that's a really that's a really good example of what everybody's trying to do all the time and like we're all trying to connect these dots and just doing that in a more automated way makes perfect sense to me you know the difference between hitting your numbers and missing them clear signal on what's actually driving growth it can get really really noisy there's so much noise you got platform data you got blended

data MMM all the acronyms MTA experiments all of it all pointing in different directions

“the more you're spending the faster you move the more the bad signal can cost you that's why”

we use house and we've been using it for years that's why the other marketing operators do as well they're the best tool on plan earth for measuring what we call incrementality which we talk about a lot on a podcast what is the true impact of your advertising dollars on your business we have causal MMM for channel level budget calls causal attribution down to the add level and architect their AI agent tells you exactly where your next dollar should go and the results speak for themselves

stockpicks all 41% left in iroce using house and you're not stuck with a help desk you get an embedded measurement strategy to actual help strategic better decisions their whole team is great we've worked with a lot of them they are world class there go to house dot io slash operators h a u s dot io slash operators and start backing your budget calls with real causal data I've got some over underrated questions for you yeah I've heard you do this twice now and you

need to make it a regular segment it's my favorite it's my favorite part of the episode yeah I'm glad you like it yeah the conner and i tried it a couple of weeks ago it's just this feels good it's great way to get through some stuff we'll uh all right this can earlier the word folks I yes when you were talking about waving at the end of zoom calls I was thinking about how some of my like I joke like my real life friends who are not in our industry they're not like

tech business people they make fun of me for that they also make fun of me for using the word folks I'm like but I don't know what to say instead when we're like you say people like we're seeing that folks are implementing this like I think people sounds sort of rude so I don't really have a good alternative but they're like you've changed folks does not strike me as corporate at all like I think like I said I think it's I think it's I think it strikes me as like sort of Midwest

and charming I'm like yeah that's cool folks are doing it I'll go underrated on folks like that's all folks it sounds kind of it sounds kind of cheesy it's pretty cheesy what do you use it that that's an extremely underrated form factor of folks that you just used to live yeah how

it guys hey guys see guys what's up guys gendered terms konuş just sending it yeah

slack but a slack spot is gonna slap your hand I was actually asking a friend this the other day I'm like is guys as guys gotten to the point where it's not even gendered anymore I was asking some

“some girlfriends of mine and I like no I think I think guys is like a fair a fair a fair way to address”

a group of people in a non-gendered way it's like this it's like a separate meeting of guys do you do you agree or disagree I think we're I think we're reaching that point pretty quickly that not a lot of good alternatives we have folks and we have y'all there used to be a slack spot like a corporate slack spot that would alert you like when you it would say like this is a you know it would it would kind of like hand slap you when you use the word guys so I've been I've been trained

not to use it yeah I was wondering that if I should not if I should not use it I got the I got the go ahead from from some girlfriends of mine but mate now I'm getting now I'm getting a signal pushing me into another direction you're making me question my decisions guys guys guys over underrated I think they maybe it might be overrated I don't know maybe you know y'all do you see y'all I try not to but sometimes it slips out you also have dude

and that's I think it's not fair to women like your version of um in like is dude

like I hear Sean Sean and in the pot is always saying like dude dude dude in it's like

it just comes off his way cooler than an um or a like but I can't say dude so why not you can say dude yeah I feel like dude it's kind of like guy it's like it's like it's not even gender anymore like you can just do couples from from the original of the word completely right exactly like the same way guys kind of is decoupled it's like you're dressing a

“group of people guys or like you can address anyone is dude I think I think you should start using”

dude Olivia all right I got I got two more ones at one to real one uh and I put this on the list actually because I didn't realize you as we're doing it Taylor holiday talks a lot about using um the the collection of geolift studies to like pre-in form or like quote unquote like assume what the incrementality factor might be for a new brand I want to say over underrated on that because I wasn't sure what your perspective was going to be the sounds like you

are on board with that I think it's extremely helpful for the cold start problem where you don't have an experiment of your own like what we've seen is these brands really are different and the same channel the same tactic might you're seeing this with incremental attribution right now where some brands are like this is absolutely crushing and others are like this tanked my business so we do see like that in the hierarchy the brand specific results are extremely important

but in the absence of that for cold start it's been extremely useful so this has been a huge unlock for us on our on our just across our entire suite of products so underrated underrated yeah like that can I ask a clarifying question on that Olivia are you doing are you controlling for like

AOV bands to make sure that like I met a hold out for a $175 at first order AOV is not being applied

to like a $30 food product or whatever yeah I mean this is it's why it took so long to build it's why we have our best people working on it this thing is so we've been working on it for two years so that's I think it's it's just it's been very it's been very complicated to make sure what we do is we we take it's called the incrementality index so we take like the prediction for any given channel and then when a customer runs a test we look at how the test result

compared to our prediction and we call that a win rate and to get the win rate up to where it needs to be took a lot of tuning based on all of those factors so it's almost like your version of a of like an MMM's out of sample kind of right your base is saying like did the actual hold out we ran get close to what we predicted it would be based on all of our aggregate data the same way and MMM's like hey we're gonna we're gonna stop pumping in the data from you know June one and we're

gonna see based on how much you actually spend if our model predicted what we know to be your actual revenue it's kind of a it's kind of a similar philosophy yeah yep exactly and then you just you have to be really careful with like test labeling like a lot of customers run spend up tests

“and you have to treat those very like when you're just testing the marginal efficiency of spend”

like that needs to be labeled correctly when you set up an experiment so it's like a lot of cleaning around like making sure that doesn't contaminate our channel level reads for what did you say about naming conventions being really really really important I wrote that down during Meta Performance Summit is the founder you did awesome quote on stage about the importance of naming conventions and I totally agree they're the heart and soul of any brand heart and soul all right I got the last one

we'll close out on this over underrated on the measurement strategy team at house and I'll be the

yesterday for like an hour and a half it felt like going in depth on what kind of test we

“wanted to run for our upcoming sweepstakes so I think we're probably unanimous underrated on on the”

measurement strategy team I've worked with a lot of good teams in my career like I have we I used to look back on the Netflix growth team is like it just does not get better than this and this is one of the most impressive teams and they make each other better there's like this spirit you know where when you you like don't want to let each other down and so they I mean when we hired Nick

“I thought we were like you know we have Nick and Alyssa were like our founding team members and”

I just thought they were amazing and then like they hired people even better than them it's just been

like so so impressive Noah we just promoted Noah he's crushing he'll still be around but they yeah this this obviously this team is like very near and dear to my heart and yeah very very underrated I was a say properly rated because I feel like everyone looks at the house team

is like they're like an A1CS team I was joking I have never cried in a work setting I can't

imagine crying in a work setting except when Nick Doran told me he was leaving to to start his brand

“I like I was like I gotta get off this Zoom call the tears that's why that's why despite being like”

it's an open secret that it's like just an extremely talented team it's still underrated Olivia's crying when people leave I mean it's like he can't that's not properly rated yeah no I just wanted it I'll end by just saying we talk a lot about this internally and I don't know if I've ever properly expressed to you like thank you for how much you've done for this community like I

feel like when I talk about how that first chapter is is making sure you know people know

that incrementality matters like so much of this is what you do on the pod you know we every week so thanks for everything that you've done and and for for pushing the industry forward well appreciate that it is a symbiotic relationship though because without incrementality we have had like 20 episodes so we're about a hundred something more than that because of you guys so yeah now goes both ways