Big-Swing vs. Incremental: Our Ecommerce CRO Testing Framework

So it's a it's a tough first episode back after some time in the woods. How you guys doing doing fantastic

Dude, I'm good. How are you doing? You went and viral yesterday. You went you went pretty viral yesterday on Twitter for your for your chair set up There were there were some there were some haters out there There's some haters out there don't let the looks to see you are one of our creators strategist page has one of these in the Office in L.A. and I tried it last time I was out there and I had I was at least 50% more dialed in on this ergonomic chair than I was sitting laid back slouched in my in my

dust chair. So I'm locked in now first first episode for those listening I mean, maybe we can describe it. I'm gonna struggle to describe it But you've got it there's like a there's like a knee section almost like, you know, if you're sitting at a pew That's it reminds me of like a Catholic sort of Pew situation and then no backing and then it sort of wraps around and then There's something to support your butt. So it's like it's forcing you to sit upright and you're kind of like suspended on top of your

“I think the big thing is like yes, you're sitting upright but like it leans you forward a little bit”

So I just like I'm just like but not in a weird way. I'm not like right now I'm on a right now and it's got a little rocker feature So if I want to rock a little bit, I can get a little motion going and and it's unreal just to go look at Go look at Cody's tweet and and you'll see what the setup is. It's pretty great. I highly recommend You know what to put the year tweet plus my my quote replying the

In the show notes, but I if you guys have ever seen Silicon Valley. There was like I gave who's that like really knowing guy who had like the pants chair And then that's whatever reminds me of it's just like it's probably a great thing It's probably really comfortable But looks a little silly. I'm so in for you guys to come visit me in Denver So when you get out here actually you know at Connor next time in L.A. Which is next week

You've got to come to an office work day and we'll pop you and page his chair for 30 minutes and you tell me how you're feeling after that

Yeah, you know it's shout out page it's amazing office fodder like it really gets the people talking

You know it's making it on the podcast. She's selling cherries to you There's probably this is going to be big for this weird chair business Eric Van holes replied and said someone in his network runs one of these brands that sells his ergonomic chair So I'm going to hit her up and try to get my affiliate link on because Cody's right that that tweet has shown It was my my best distributed tweet in years. So I need to get the affiliate link on

And by the way, just listeners who you guys know the only reason we're talking about this is because Connor wanted to talk about David and other kind of We're like we've talked about David protein the last and like protein in general probably lasts like 10 episodes

“Like I think we got to cool it on the protein”

It's true, but thanks for squeezing it in there. We've got the streak the streak is technically still alive Yeah, it's like it's like our snap streak. It's the David protein mentions mentioned streak. So Appreciate that Cody for keep that going. All right, we're talking we got a bunch of stuff to talk about today We want to start off with some some CRO testing. I think all of us have been doing some interesting things on the CRO side of the world Connor. I want to start with you

You I was not here last week. I think you guys talked about it a little bit You just launched a new theme test. So I'm curious like what was the hypothesis with that? What did you end up doing and and what results are you saying so far and when you say theme test is this like Entirely new web build when you say theme? Yeah, yeah, yeah, yeah Yeah, so let's get into it. Cody and I just touched on it briefly and I said I'd come back with some more notes

And I've got those you said if it did well

“So if we're talking about it, it means it's doing well. It's been live for less than 24 hours”

So I'm not calling it one way or another. That seems early enough to call it. I did it. Yeah, it's funny

No, actually for the first like hour we had this weird tracking issues of the results were actually really bad

But what was funny in that was I was talking with our VP of Ecom and she said it started out and it was like negative and she goes She was I feel good about it cuz she goes all all bad test turn good and all good test turn bad So she was excited to like we've really got like a 10-foyle hat on over here at Ridge Yeah, I thought that was really funny too. So we've just got to go and live now. We'll talk about it I cannot say one way or another

Whether it will be a win or not, but I've kind of got like how we are thinking about winning in the scenario so I can get to that in just a second But I'd love you guys is Thoughts here. This is these are the things I was prioritizing one. It's a complete Theme rebuild. So totally fresh. We worked with van on it So Xavier who did the hexclad site, he did the simple modern site

They're great to work with total pros. I'm so full You know bottoms up rebuild of the theme and we had a number of things that we wanted to do. I'll start with Expanding category. So we've talked about this quite a bit We've dramatically expanded categories prior to within those categories one of our main focuses was just like how are we providing the navigation tools and sort of like the site infrastructure to support

Collections filters different Collections page tooling so we can navigate between the like hundreds of wallets use that we have at this point So that was a really big focus for us two. We've got in category education bundling So once you're not navigating between what you want, how are we just how do we have the site? You know

Functions to like educate someone about what they need and bundle the products that they need to so what I put in that bucket is like being really thoughtful about product cards Once you get to the PDP however you navigating two and from kids Sort of things like that and then three and then I've got a list of like small sort of like tertiary goals that we have three is just improved subjective design choices I spoke to um Who is the guy that you use Cody in Canada or have used Andrew? Yeah, um, I spoke with him and he was like look

He told me Years ago he was like look your sights good. He's like you good. He's don't really have a spy a speed issue like the sights good He goes really if you're a founder just wants to redesign the site

“He's like that's totally valid and that's why we're redesign like half of sites and I'm like yeah, that's that's totally a”

Valid reason for us to include here. So that's the third one that I have is just like we've had more or less the same theme that we've been building on top of for 10 years

So let's just sort of blank slate it make the design choices that we want and sort of just feel better from a subjective perspective about you know The web choices that we're making. So those are the three big ones that I had for goals of the the site redesign Do any of the stick out to you guys or If you had to redesign your site today and I know both of you guys are thinking about this all the time like what are some of the things that you list as sort of top priorities I mean your last one was like the core I wouldn't say it was like purely

Like when we did our redesign because we did the same thing in 2022 with the anger where we'd like did a full on Site redesign totally different experience and just split tests the entire theme against the original one The core reason for that was we just felt like our our website was not indicative of our Of our branding and like the brand ethos which is just like sexy badass cookware That was like the reason we designed it and then like that was that was number one

And then once we tested that and found out that that was converting better than we went into the other reasons that That you talked about first which was where we probably should have started which is like Product merchandising we were adding new categories like quarter over quarter over quarter and they were not Discoverable enough so then we went into like the the navigation Optimization the just like category discoverability stuff

So yeah, I resonate with with all those I've got I've got a few others here

“I'll jam through them quickly. You guys let me know if anything stands out one. Oh cleaning up old code easy one and I actually think this is something that I think of as like a really long-term benefit”

We've just made but not always the best technical decisions over the last you know

10 to 13 years so this was our chance to sort of again like clean slate the whole thing Performance site speed a big component of that it is quite a bit quicker not significantly we spent a lot of time on site speed earlier But sort of a nice benefit and then these are some of the more like tertiary ones. I've got Site schema for SEO and LLMs as we're really digging into the site code We're like we're fleshing it out as much as we're possibly gonna do it. Let's make sure that we are setting up from a schematic perspective

Our website in the way that we need I've got accessibility and compliance Let's just clean all that up and like reduce all the ADA risk that we can as we're going through this process on site tracking is another one where it's like Rebuilding all the functionality. Let's tag all the buttons and carousels and you know navigation menu items so that we can just track on site behavior better We've got Shopify markets and international localization our previous theme wasn't super conducive with supporting that

And then similar to site team is cleaning a product information hierarchy. We spent a lot of time and I've geeked out about it on the podcast before Thinking about the taxonomy of our products and like what is all the information that we want to append to any given skew and how do we make merchandising decisions or on-site decisions or Ad naming convention decisions as support that and like let's just make sure all those dots are connected as much as possible

“So that's what I've got those are like my the big high level objectives as well as like some some tertiary ones that we've been thinking about as we've gone through the process”

I knew this was gonna come back to name and convention somehow with you. It always does did it it's all it's all interconnected

Yeah, it's cool. I love that. I think that's really smart. We actually had a Not a new design. We actually had a new theme go live today. We like refactor our theme. So like very similar I think we will probably do like a Re-design next year. We just kind of went at one step at a time. We have been on Single-page architecture as SPI. I don't know why we have this like really fancy Polish

Dev agency built our original site and like it's faster, but like really I

It's almost like trying to go headless and so we just finally decided to like rip the band

It off like it messes up like we can't run intelligence themes tests with it. It messes up tracking on a lot of things So actually like just went live today with it and did a lot of components. So I didn't touch anything front in anything visual That'll just be separate and we're just kind of like for now a be test our way through you know Design changes that we want to do but definitely Cleans up theme and even like it was interesting as we were doing it

This was actually Andrew's idea and I did it. I just read I just downloaded the theme files like when we finally got it to preview Toss it into clawed and and codex and it was just like hey we want to right run it over all of it But run it run a speed on it and it found a lot of things out like they probably should have found But like we found like a bunch of different apps still had scripts like super old apps

“We had like you know like super old stuff. So like I think it's uh, it's one of those like pinyass things”

But it's just like very necessary maintenance to do whether it's you know what we did or what you guys did and one of the other things I would I kind of did here a little bit but like I would do them future like I talked to Stuart from revoke Maybe like a year ago or fix months ago about like a lot of the work that they did to really enable them to be super AI you know forward and just like building their codep codebase very like component driven Like I would do that. Obviously we're using clawed a time for pages and stuff like that like

I'd also be thinking about that as like how we're how we're building it, you know for for for agent to code in as well Right so in the case of refactoring the theme are you as testing that or are you just did you roll it out?

Yeah, we just rolled it out. Yeah, we're like have like QA because it's actually our third time trying to go live and

Found some small things each time and rolled it back but now but it is pretty fast we at least did like the normal like page speed stuff It was cool like a good hack. I tweeted about it like I would recommend everyone just take their theme files

“Give it to claw or codex where I've used and just like hey how do I improve speed here?”

And it'll just like it'll just find stuff so I recommend that but now didn't didn't test it Motion just dropped there 2026 creative benchmarks report and it's been getting shared everywhere Slack channels LinkedIn Twitter Sharing it in our private group chats and it's great because everybody's been asking the same four questions forever What is normal? How many ads should we actually be shipping? What is a healthy hit rate in which formats really win? The report analyzes over 575,000 creatives from 6,000 advertisers in over a billion dollars in ads

Ben to answer these exact questions and the report has some really interesting findings like the fact that only four to eight Percentive ads actually become winners in over half of ads actually lose in promotion customers This report is especially helpful you can upload it into your motion dashboard with their run with AI chat and compare it Directly against your vertical benchmarks hit the link in the show notes. I promise you won't regret it

And as always go to motionapp.com until the marketing operator sent you

“One thing I noticed in like kind of the second way of of you explaining this test is the first way”

You're like we're testing like here are the objectives here are all the things we changed and Like an immediate knee-jerk reaction is like this is a huge showing you're changing a lot of stuff But then when you explained it further you actually made it you've actually been thoughtful about making sure that all the different variables you change weren't Like Couldn't be manipulating the same variables like the upside for example like you guys

You did a lot of optimization on call it product category discoverability with like your hamburger menu In theory that should improve conversion rate because more people can find the products they want And it should also improve aOV because your product and more discoverable if you also would attest to the bunch of your upside components Which in theory could also move the needle on aOV when now you don't really know which variable is is moving that aOV You know it was at the hamburger menu changes or was at the upside components so you were thoughtful and like making sure that you didn't have

Two of the same change effect in the same variable right like that's and then you'll you'll go into Those of those like iterative tests after that is that is that true and how you kind of set that up roughly I mean We're still it let's say you know whatever result we get from this test. We won't we don't have so few very variables It will be able to point to one thing and say like this is what drove this

We still have many things that are different between these two themes, but it is a it is It's just a more like palatable amount like I'm comfortable with like the amount of differences We have now whereas if we went in with a new homepage redesign and all new upsells and like a new way to navigate to kids We and we've got some cool components built out that I'm excited to test we've got um we've got like 5060 wallets

I would love to test being able to navigate to a more of those wallets from the PDP so we built out this like It's almost like a tab swatch set where you could go from like best sellers to NFL or to limited edition But without having like 60 swatches selected there so she's like way more navigation that's one of those tests We're like if that's worse it could be way worse and I'm not just gonna like throw that into the bucket so the things that we have tested that we have testing quote-unquote now

We'll get to this we'll be able to transition over and then we'll begin rolling out the new features sort of one by one

“Yeah, I love that. I I think there also is a way we just did this with our product page”

There also is a way to take these big swing tests and still port over Like all the findings or the optimized setups that you've tested into in the previous version like we just did this with our product page

Or we totally like we basically just gave the entire product page a facelift with that being said if you go and look at our like if I go and look at our PDP now

a lot of what's here. We tested into like our accordion sections and the information may include there like these You know, we tested into a what's included section on a landing page We took that learning and implemented in this new PDP build This like social proof video carousel like these are all things we Tested into in the previous version of the product page or it already been off the product page and then pulled it into

this updated PDP refresh which also drove a lift so it was like kind of the it was like kind of the it's finding that sweet spot and like this isn't that new big swing But we didn't just like abandon everything that we've Tested into in in terms like what's the optimized what are some of the optimized components on this page so

“I think that's that's an important note here and I'm sure you guys did the same thing right it's not like you just like”

Totally wiped out all of your your learning from the last decade and just like abandoned it's like you took those and put it into this newer

Overall better experience so you're still starting at like level five instead of level zero 100% oh yeah, yeah, like Like most of the features we built we've tested into over the last couple years this wasn't like We're throwing the baby out with the bath water and we're restarting from from some like brand new perspective It very much a lot of stuff that we did was informed by by previous test week from

I've got one other question for you guys on this so Connor so you tested incrementally And then you shipped PDP but when you shipped that PDP did you AP test that or just shipped that no, no, we so we yes We had we had a bunch of incremental PDP tests over the last I mean God two years and then we tested this net new swing and we just and we did test that as well So it was like stacking those incremental wins and Making sure those showed up in this new updated PDP design and then we split test that against the

globally too like every single PDP across every original PDP so it was like A ton of data that went through it and it drove like a 5% lift and in RPO so we felt we felt pretty good about that How cool and but there's probably like a bunch of variations on that bunch bunch of you know different variables on that new PDP Definitely like a lot of it. Yes, I mean I'd say if you go look at our PDP like If you go look at section by section, I'd say at least 50% had something different so now this is

And this is on curious to maybe bring this back to you Connor because now what we're doing is

basically our our like Gantt chart of our PDP has been

Like iterative testing and that's gotten us to a more optimized product page experience then we did this big facelift That we also split tested that included a lot of the things we we learned in our iterative testing and Now we're and that like kind of brought us up here and now we're going back into the iterative testing To see like which of these sections are more important like what happens if we remove the the Gordon Rams You approve section and the TV commercial like what kind of lift does that does that bring to the page or drop to the page like so now we're kind of going

Back into the alright, we we have a new experience. That's converting better and we feel like just an overall more aesthetic on brand experience And now we're going to back into some of these other sections again. So it's kind of like Building up to a peak and now we're like almost coming down the other side of that in a way. So are you guys doing That two corners that what the approach will be and it's funny what you described is very like sophisticated We're just pushing the bowl there up the hill and then you let roll back down

Yeah, we'll we'll take a similar approach. I mean we're probably not U.S. It sounds like you guys have a more rigorous system potentially like we've been rolling out new features We've rebuilt many of those features were retesting the PDP and then we've got a plan now for additional features at will test and then and then for us Really when it comes to more PDP testing it comes down to things like I'll give you guys a good example and then I've got Another example that I like here um as we launch new products or expand colorways and given products that typically

Influences what sort of PDP models we need to test and sometimes we can pull from different categories but Power banks for instance are Really the only product that we have that people are interested in buying multiple of the same silhouette Our other bundling tactics are like cross silhouette nobody's buying two or three wallets They're buying a wallet in a pen in a key case or something like that so so those two kidding experiences a multi silhouette kidding

“Experience versus a single silhouette kidding experience. I think are different and for power banks”

What we tested into just recently which I'm really excited about is a really flexible two and three pack Bundler where you can select different colorways at each of those So we built that out. We tested into that fantastic. We feel good about it. We're gonna like refresh it into the new van theme

And then when we launch drinkware in a couple months

We'll have this new feature to deploy for drinkware. We'll be able to test it again. So for us again. It's a little bit less like I don't know what you want to call it like scaling up and down the PDP of a single product and more Developing models that we can then test across category for different, you know purposes Kind of the reason I asked was because of where we were doing our PDP as well and like even like how to debate the other day about it internally like so we did exactly what you kind of said on your

For years on the homepage where like we tested Probably four things recently on our homepage. So we tested technically not homepage, but we tested nav So we got a winner on nav and then we tested something on our hero got a winner on that

So that's like our cool. That's our new you know version. We tested like a few different sections

So I think probably like the hero press like two to three other sections And so I don't really want to take the time going through every section on the homepage incrementally Like it was like focus on above the fold and so now what we're going to do is we're going to redesign our homepage based on that We'll probably test it, but it'll just be like a redesign. We'll have a bunch of variables

“So that's what we did I am I think that's probably the best way to do PDP. I just don't want to take”

That long like six months or whatever it is, you know three months like to to test because especially because there's Right if you think about like a like homepage like there's not as many features There's so many features on a PDP that like you could be there forever So I'm like, I just don't really want to take the time. So I'm like And I'm normally like I used to be in the minds of like let's test everything and now especially

I like we can develop a new PDP like you can develop a PDP in a day if you want to you know like a very different one I'm like I kind of just want to approach it a little bit more like an art project It's like let's just make the best PDP we can possibly make and like let's test that And I'm I'm pushing for that and I think some of my teams like no we want to test it more and come on like how will you know I'm like there are also ways to know like you might not be a strict AB test but like let's say we have a winning variant

Then we can go into a heat map data. We can go into you know Other data and like kind of build a hypothesis of like hey, you know even though we change multiple variables like this like Lee is why or

“Or not so just I don't know like I just don't want the speed so do you guys agree? Do you want to push back?”

Do you just agree on that at all? I like it a lot We're doing the same thing with our like one of our hero landing pages right now where we've put like literally tens of millions of dollars has been through this page in the last Three four years and now the new project on it is Let's go ship this to the brand team and have them make the most beautiful aesthetically pleasing on brand version of this and just start putting spend through it and see how it compares to the version before it

Our our serial lead also has has has this interesting idea or approach rather on this on this like big swing testing concept like because we did the same thing with our home page Like I don't know a year ago or six months ago and his his thought was all right Let's take this big swing where we're changing a bunch of different things and like let's just test it against the original version Make sure we we have in fact built an experience that's generating higher conversion AO VRPU

But then his um he is this interesting approach where His next step is like assuming the new version performs better. Let's do a multi-variate test where all you're doing is removing

“one section for each variant and the goal in that is seeing which section is the most important”

So you can see like which which removal which variant which section removed drives the biggest drop in our P U and that tells you which section is is driving the most performance on the page and you can almost like stack rank all the different sections Based on like whichever one has the biggest drop in our P U has the most importance in driving performance on that page and I can kind of like steer you a little bit in terms of where you go and optimize and act. It's like oh If this like Gordon Ramsay social proof section

When we removed it that drove the biggest drop Okay, well maybe we go in like focus there because assuming that the inverse is also true If we go and improve that section it'll also drive the biggest lift versus going in improving like a different section on that page So I thought that was like a cool a thoughtful idea on like how to how to how to go about this Like start with the biggest wing and then like back into the component set our that are driving the biggest lift

We talk about incrementality a lot, but how do you actually Amberationalize it to make your business better that is one thing that I've been really leaning in with my team recently and house Just played a tremendous role. We use it for all of our experiments all of our geolift testing But we now use it for our MMM as well. I've been a design partner. I've been one of the early design partners on house to see MMM See stands for causal so it's one of the only MMMs if not the only MMM that I've seen that's actually using your causal

Experiments to build the model and set allows me to just trust the data so much more

So it's not a black box but actually informs our roadmap and has been so crucial for allowing us to

They've also built a really amazing community with some of the best D to see growth operators out there

“If you want to check it out learn a little bit more about CMM go to house.io/operators to start making better data driven decisions today”

One I'm all for I think it's an interesting thought to say if cost of development Has been driven so close to zero Like do you need because I feel like part of incrementally testing the PDP is like also saving on development resources But if like if those if those become far more affordable and all of a sudden you could rebuild an entirely new PDP really quickly and test that Then it then it does make sense just take larger more

Exploreative tests and then kind of decompose it with conners process where you're like because at some point You're gonna find a PDP that you love and you'll want to know well, how do I improve it from here and then it's just a matter of like yeah Identifying at a more granular basis what is making The impact and the heat map example that you gave Cody as well as conners. I think are both good. I like it Let me ask you guys this. So we'll you know, I'll give I'll give I've already refreshed intelligence probably a hundred times

But I'll give it I'll give it quite a bit more time before we start doing any forms of analyses

“One thing we do that I think is really helpful again in terms of like decomposing the results of a of a given test”

We'll look at within the theme test like we'll be able to identify if it's driving better or worse performance by category And I think this is a really interesting way to like try to better understand what's working and why or the inverse of that One of the things we saw in the in the past I've mentioned it before we found this we were testing a new hamburger menu many months ago It had pretty negative performance and it was almost exclusively on our ring program

or actually though the like the like interesting data point here is that The category was unimpacted by this test and then rings had a significant decrease in revenue Persession And What that ended up being was like our font size was too small which which totally tracks for me, where it's like if you're buying

tech accessories if you're buying a You know a new cool trendy power bank like you're gonna be a little bit less

“Impacted by slightly smaller text whereas I think we have like an old I mean we know for a fact we have like an older”

Mail audience for our ring program and they were impacted by the like you know 14 point phone instead of 18 So us decomposing the test. I think we'll be like the next step for us But if you guys weren't my shoes. Is there anything else you'd be looking at to try to identify directionally You know what's working or not in this theme test? Yeah, I mean I'd look at heat maps I used to look at recordings more. I don't as much, but you know

Potentially looking at that. I mean, I'll talk about later about big like user interview fans so like You know even like subjectively like getting ideas for next test. You know like

What's working well? I always find that looking at like creative things when you have data is very different

Like sometimes you'll look at an ad and I got a cool ship. This looks pretty good And you get data back and it's like oh, of course like the thought like the hook the hook rate is low or something like that So I actually think it's interesting to like get some data and then go look at the site yourself and be like Oh, now I see that's maybe not the best UX or we could do it there So that would be like the process I would do. Yeah, we've been I mean we've been testing some

new offers lately and we've been looking at like a lot of heat map data to see how people are engaging with these sections and then making iterations based on that It's like oh, we we built this section to kind of dummy proof the offer and and make it very clear what it is But then you find that it's like in the bottom 25% of like engage components on your website

So it's like okay, clearly this isn't doing what we need it to do so we need to adjust this I've also we've been we've been launching a lot of like Products that are similar to other products. So as an example, we just launched a grill top or Like I don't know a month ago a month and a half ago

We already have a barbecue grill pan. It's a super cool product. It basically just allows you to like

Cook all the food you normally would in a pan on the stove top on your grill without like all of the veggies and stuff falling through into the grate So you get that like that grill taste and then we launched a grill top or which is just a bigger version You can cook a bigger meal on it. So we've had been having a lot of back and forth on Like what like where should we put these like graphic treated assets on the product page to differentiate between the two The two products and there's just you know you have like all these people in the room with all these different opinions, right?

And it's like No, we don't we don't need to like let our our subjective opinion drive Where this graphic asset lives on the product page we can go into clarity and in five minutes Look at the parts of the page that are getting engaged with the most and that's where we should put the the graphic treatment like put it in like

Like let's make the second image that like

comparison graphic where there's no reason to put it anywhere else like that is clearly the data driven decision here And like you just end especially with website like everyone has an opinion on website because everyone's channel touches the website

“And I think it's important like have some sort of data driven framework there”

So we've been yeah, I mean I guess a long window way of agreeing with Cody on like the heat map or microsoft clarity like click and engagement data I like it. Well, um, if you guys can keep your fingers crossed make any sort of sacrifices to the CRO gods It's a really big test. We got going on. I'd love any help that I can sacrifice my my old dust chair for you I'll burn it right in the office there. Thank you. Thank you Okay, so continuing on like the the vein of CRO testing and

We haven't done a test of the week in a while. We'll call this the the test of the of one-h I guess I I want to bring this up because what what I think is interesting is everyone talks about Especially in creative like the creative flywheel and volume and you know we're at a point now We're in like I don't know seven eight markets and we we certainly don't have the same Sourcing in all of our markets and this the size of the prize isn't isn't as big as it is in some markets

Is other so like Japan is one of our smaller markets growing but but still smaller and like we're just not We're not launching you know

“50 ads a week in the Japanese market the same way that we are in the US so I think we've had to kind of like”

Just be more thoughtful about where we point all of our resources over there and one the big changes we made you're on year um, it's just We were really chasing revenue targets and just doing a ton of pot of my funnel Marketing last year and it was can't really just miss managed and if you would go to look at our like net new

Visitor A in in meta last year was like 20% in the Japanese market and this is not a big market. This is like a low seven figure But if you go look at what we were doing it was just like offer ads offer after ads offer after offer ads So the big thing this year was Launching more ever-grin creative and just like training the algorithm to go and reach more new people which we've successfully done

UGC in Japan is like where UGC in the US was in like 2019 like you could launch a new UGC out and it will have like a 6% outbound click through rate and CPA will be cut in half So that was like phase one here this year is like we need more more upper funnel more creator let ads

And it became clear in the first quarter that like the egg pan was our big unlock over there

Like we launched a bunch of UGC and the egg pan UGC had like a third the CPA of any other products So like great we've just unlocked our hero product fantastic that's a huge learning struggle with some inventory we stocked out a couple times now on a better place and we're still producing You know egg pan creative, but again we just don't have all the resources over there that we have elsewhere So the big issue was well AOV is low like we improved conversion rate in a major way

But AOV also came down see if you are up more than AOV so like all right this doesn't that's fine, but We were over index on the egg pan And AOV did come down so the big win we've had lately is not necessarily peppering in another

“You know 15 egg pan UGC creators it's been how do we drive up?”

How do we drive up AOV on the egg pan and and also simultaneously get more sales on these other products?

So basically the the existing funnel was you know egg pan ad mainly UGC going to this egg pan landing page

Going to the egg pan product page and that was that was a winning funnel so the new task was the hypothesis was well If we instead of directing people from the land or right to the product page take them to a collection page and we create like Call it three new bundles that are just combining the egg pan with our other top selling skews which is like A fry pan a high-sided pan in the rice pot and then maybe one skew that's like all four of them together And then we can like position a discount we will maintain conversion rate because the egg pan is still there

As a solo skew but then also see a nice bump in AOV. So that's been the task is just 50 50 split between direct a PDP in collection page You know that the other downside hypothesis would be oh well you're adding an additional click into the experience and that could hurt conversion rate and people could drop off And that's been like a massive win for us in the last two weeks like we're seeing CVR's flat It's actually up slightly on the collection page variant But AOV is up 23% revenue produces up 27% we're getting more sales on other hero products because of the way that we're merchandising it

So we're by just like we're we didn't have to introduce a bunch of new ads around these products But we're less over-invested on the egg pan now, which is great for inventory levels and a number of other reasons and now Shopify is up like Again, this is coming from like very small numbers, but it's up like a hundred sixty percent So I just think it's a great example where people get so hyper-fixated on like more ads more ads more ads and often it's like Once you find it unlocked with your ads, you can often like look down funnel and continue to optimize that way

And we've been pretty jazzed on this and now we have like a very optimized destination

So that's been I was I just been very happy with that. It's a very like classic product merchandising play um whereas like you know, we were like sending traffic for the home page last year Like no product merchandising, no product funnels, so it's very much like You know two out of ten to a four out of ten, but a big win for our Japanese market this year. That's so cool So one actually I didn't realize it sounds like the way that you guys are running

Hexclad in Japan is

Pretty significantly different than in the US was that always the intention when you launch there that like hey

We don't we don't expect the like 12 piece to be the hero product. We think we're gonna have to go into this market with

“I think you said a rice cooker or a rice pan rice pan egg pan like all these different things”

We're gonna like approach this like we're gonna refigure out what our hero product is here Was that always a goal or is that something that was identified and then you guys have started testing into more recently? No, it was not the goal. We we developed this product and The obvious the obvious place to launch it is in Japan and then even when we did that it wasn't like it wasn't until we actually started building Funnels around this product that it really started taking off which is like again, it's the classic

If you build it they will if you like launch it they will come is not the truth like if you build funnels around it They will come and they will convert

So no it took us like I mean first off we didn't even have this product when we launched in the Japanese market

So it was like you know at least a year maybe even longer until we even launched this product That another six months before we like actually started building funnels around it and then we found these on locks in the

“Out of count and then the AOV bomb. So um, no, it's been like”

Figuring it out as we've gone Was very much not our hypothesis going in that like the egg pan funnels would be what it's not a super pricey product. I mean it's It's a single it's you know a single pan basically so it's been it's been interesting Rich panel CEO just predicted his own SaaS products will lose 90% of its revenue and he's the one building What will kill it they are betting that AI agents will replace traditional support tools entirely no dashboards no seeds no work flows

So what did they ship an autonomous AI agent that handles customer support from end to end reading order data issuing refunds updating shipments without a human ever touching it it means you can run a faster leaner support team at a fraction of the cost While resolving tickets instantly 24/7 rich panel is cannibalizing their own business to build the future of cx and support for brands Because if they don't someone else will go check out what they're building at rich panel.com and learn how you can automate your entire support operation today

I love the idea here right like you're having success with the egg pan you were driving people to the pdp your cell on it fantastic By adding this like Interstit show where someone can choose between a couple different bundles you've increased AOV without sacrificing conversion rate at all I think that's a great learning lesson Can you navigate to the egg pan bundles from the single egg pan pdp?

No, but that's a great idea like to have like a variant selector right on that product page is a is a really good idea We did make We did set up the core website in a bit like in a way that like if you go to a hex cloud.jp our Hero section will navigate to the collection page that we're running this split test way So we made the core website very like our hero sections now all about the egg pan

That then takes you to a collection page that has the egg pan plus the collections and then What is not what is the same is our hero business is our

Evergreen offer merchandising like we are always we always are able to run offers because we're such a big bundling

Brand so it's like all right if you bundle these things you get a discount and that's an evergreen offer

“So that's what's really working here is we're able to say hey if you buy one of these one the one of these product or skews”

Rather that has two products in it you're getting 10% off if you buy the the skew that has All four product you're getting like 20% off in the in the now the second best performing skew is the four the four product bundle Which is great. I was expecting it to be one of the the two product bundles, but um Yeah, so we'll probably sunset some of the bundles eventually that aren't doing as well and just Make it even more streamlined, but um, I like the idea of adding the the variants to the the egg pan product page. It's a good idea

Yeah, that's where I go all the time. It's like it's the same idea of what you brought up earlier Where the best way to test that idea is with the interstitial interstitial collections page Giving people the option to click between a couple and then it's like oh, yeah, and I have no idea could be worse But can you compress that further and just like give those people those same options if you make it visible from the single egg pan PDP

um the other thing is So this is a paid funnel you're describing you as must also merchandise the the new egg pan bundles from someone landing on the home page as well Yeah, so that's yeah, that's what I was saying where like if you hit the if you hit the home page

Like this collection page it's kind of going back to what you talked about earlier We actually added additional components other than just a pure collection page with additional skews like we have a hero section with a nice image that kind of explains the product. We have like a hero

Card as the first skew in the collection which also has like a hype reel over late on it

So it very much was not a Single variant test. It was like how do we just make a much much much better experience which includes multiple variables And then of course now we can go and kind of back into some of these more iterative optimizations Which is it's interesting. This is like a pattern across all of our our CRO programs

“Right, and that's what I was going to say is it's super fun”

This is a great example of like you build all these tools and strategies selling seven pcets in the US and then you get to apply a lot of them to the egg pan and Japan We're in a somewhat similar boat right now. This is a little bit different, but with TikTok shop like our hero product is just our tracker card Individual tracker cards for like $30 It's far in a way our most popular upsell it's extremely popular to be bought along with the ridge wallet But we're seeing success with it as it's own hero product in both TikTok shop

There's some Amazon success and We're trying it on D to see now and it's like it's a completely different funnel AOVs of like 40 bucks or whatever because we're rolling out bundles We're looking at cohort values because all of us, we're getting someone in who hasn't even bought the wallet yet So we're seeing higher sort of incremental LTV post purchase which is super cool and it's just fun. You get all these like

You've developed all these learning strategies over time and they kind of you get to reconfigure them in new ways Depending on product market channel, etc It's fun to yeah, it's fun because there's a lot of difference with there's also a lot of similarities Like right like how we're setting up the test like we've run similar AOV boosting test in the US just around different products And again, it's like it feels like buying media

I feel like I was at like homestead in 2020 buying like launch a new UGC ads and an account that's never launched UGC and you're like

Like literally almost overnight the CPA on those new ads is half of what it has been for Last 30 days and you're like it's just like so obvious, you know, whereas at like our brand scale now these these wins aren't as like Big right, you know, you rarely want to new ad that just like cuts the ad account CPA and half because you've been launching Literally tens of thousands of ads for the last four years. It's just harder to find those wins So it's fun to fun to kind of launch new ads and these less mature ad accounts in less mature markets as far as just like Facebook ads goes and

See those like kind of like 2019 2020 wins that you used to see it is um are you as gonna move Is the egg pan live in the US and are you gonna be applying any of the strategies from Japan and testing them in the US it is we actually just launched it in the US like Two months ago and it's it's not as big of a product over here obviously. It's a very niche like basically the There's this style of omelet that Japanese culture makes it's really it's really cool. It's like this very

Like delicate process. I guess and it's a really good so it and it's just less of a known thing over here So it hasn't performed as well and like we might roll out some of the strategies

“But honestly, I don't even know if the size of the prize is is worth going down that rabbit hole”

We still have it on like if you go to our our US site like we have a homepage section for right now because we basically Our go to market strategy for new products is every time we launch a new product for the most part We'll create a homepage module for it and that will sit there until we roll out the next product that is where they have a homepage module. So we've had that on our site for like I don't know two three months at this point and like again, it's it's not it's performing pretty well

But it's not like it. It's changing our business or anything like that Yeah, I need to egg pan. I'm looking at it now. Dude could be a great um take-tuck shop product Yeah, we should we do not I don't think we have a live there yet. So we actually should honestly the rice cooker is probably more of the has more upside in American culture because I think more people are probably likely to want to be making white or brown rice then this like Japanese style omelet

You true love it

Have you had one of those omelets? They look so good. I always love to watch people like making them on Instagram

Yeah, I've never made one, but I have had them last I was in Japan a couple years ago and yeah, do they're They know how to make some eggs over there. I'll tell you that they look so good All right, so we want to we want to kick it over to Cody because Cody's been doing a lot of Cody even going deep on on CRO process and like using AI to totally revamp your CRO program and an inefficient way and

“Just we were texting last night. I think you're doing a lot of things that”

Most people aren't doing so drop some drop some wisdom on the on the listeners right now on on the latest and greatest and CRO in 2026. I'll talk like no AI first because like the way I try to approach these like buildings AI workflows is like what's the best in class version of like fully human and like

How do we actually enable this but that's like my my thought process and and one thing

It like I've always been really

Actually think I used to be like pretty bad at CRO and like we used to be and like we just like you know come up with tests Big art that's that's a cool test and like not do it from first principles. So like I mean obviously That's the most important thing so is number one is like having a framework for Testing and have just like piece things together from what you know we've learned Seeing other things seeing like agencies do things well

But like really trying to be really specific so like we have like a you know a few different ones like always above the fold Highest traffic pages like probably like no brainer is to a lot of people

“But I think just like codifying it and having it you know there so highest traffic pages”

Right above the fold obviously if you can identify high traffic pages that are Lesser converting or lesser revenue process and like huge opportunities there just like trying to be you know high leverage with all of these things So like there's a few different things and then one of the ways that we test which I haven't seen many people do And I think it's controversial because everyone has different feelings about like

Statset and that you need is we work with this agency and they never ran like a B test where it's just like two variations

It's almost always they would run like It's technically not multi-variate, but it's like four variations if you have enough traffic And the way they would they're a thought process if you're taking it you know you only have a certain number of blocks and and this actually works really well in the CRO area Arrow where you know you have More ability to to conquer point of like cost to develop my going to zero you have more ability to develop

Test then you can actually run the test and you know your testing slots and so rather than just Taking a two-week test and doing a you know AB two variations. They're usually doing four variations So most tests that we run look like this. So you have obviously a control that's one of your slots Then we have four other slots and the variations stack and so you know if I'm testing a home page I'll I'll make it up right and you're testing like a hero module you might change the button

Copy on variant one and then I'm variant two you're gonna keep that change But you'll add on one other change in top of it. You might change a button copy plus you might change You might add a new button and then a variant three you'll have the button copy add a new variant and you're gonna change the image and So at least that way the one that wins you can isolate and be like all right. What's a step below it that it didn't have

“But you're able to just test a lot faster. So I think that's one thing that has helped but helped us to be able to”

Improve it right you're just getting more shots on goal Wait, so can you how do you can you talk me through how you would like interpret the results from a task? So it's basically like Plus let's like plus one. It's like original new variant as v1 v2 is like two new variables v3 is like three new variables

Yeah, so you can always isolate with the variables. I'll give a specific example

We just redid our collections pages and so on the first version of control We just changed like you know when at the top where you like the different categories like we like change the UX of those on v2 we kept the new version plus On each product card we indicated how many shades there were on v3 we kept those But then we actually added the shade swatches and then on v4 we kept all those but rather than driving just to pdp's we had a quick shot module

And so each test was slightly different each variation was different but it only had one component So like the quick shot module didn't do well Even though the one before it was a winner so we were clearly able to say like hey that's the variable that is losing here like people Don't want quick shot. It's almost like the Like what I said but in like the inverse

Right it was almost like big swing and then you like remove each component But you're saying like you add each component Which I think's better. I think that's about yeah, yeah, but like so how I'm gonna approach this pdp test is actually designing the perfect pdp like that's where I'm starting And then from there we're gonna kind of remove Components to kind of get back down and to like it won't be perfectly clean to get back to our control

But it'll be similar so I guess it's yeah, it's a different way of doing it

“But so that's how we run almost every test and again, I think you can only do it when you have enough traffic and I still think some people don't think it's”

That second off, but I'm willing to give up a little bit of that, you know perfection and and that's it to Fast and and in a in a test like this with the pdp Will the main KPI be Conversions or will you look at things like, you know, add the car begin check out will you look more upper funnels so that you don't need as much That's a good point. I think I said the setting goes maybe when you and I were recording like I'm really bad at looking at like the funnel steps

And you just I'm just looking at like revenue per session per visit, you know, I think we should do better at that

I had this thought earlier as as Connor was team you up. This is mostly a joke

“But you know, you're going so deep on AI you're going so deep on CRL. Do you think you're approaching any form of like CRRO psychosis?”

I Mean definitely like AI psychosis in general. Yeah, yeah, yeah, yeah, it's like but it's the CRRO version of that It's a very particular form of AI psychosis related to making optimizations on your website. It's oddly specific It's a look at the low-tam low-tam tweet potential here That I get you know very very much like so we have this the zero system the skills things like that

We can write things since I forgot if I said it, but so I have this like you know Agent like a you know a her me's agent and and Slack now and my director of growth and you come like put a test in and screenshot it

And I just tagged the agent and I was like hey rate this test and they're obviously did it based on our skill

Which is our framework and then I asked it to mock it up and it was like super cool It's just like it actually just like gave it created an HTML and it just put the screenshot and Slack without me asking Yeah, so I was like that was like a little bit of a psychosis moment. I'm like hey, they can do that

“Before Q4 hits the smartest e-commerce operators aren't just asking how do we spend more there?”

asking how do we make every visitor more profitable that is where after self comes in after cell helps e-commerce brands increase revenue per visitor Contribution margin and incremental profit by optimizing the highest intent moments card Check out post purchase and even the thank you page brands like hexclad rich and pillar Increasing their AOV across every cohort using these zero risk up cells. We've tested and they have no negative impact on conversion

It's just more free money in your pocket and with rocked. Thanks. You can offset your after cell subscription No redesigns no risky funnels just better economics. I want to see how much money you can make with after cell go to Aftercell.com/rocket revenue forecast put in your numbers and see how much more profit your funnel will make before BFCM starts Okay, so user user working on building the quote unquote perfect PDP right now. Are there any other like high leverage touch points that you guys are testing frequently

I mean we're we're trying to do it all right now like that's because we can just you know

“We just had I think she's on second week our new e-commerce zero person like now between her and I like handing things off”

I'm obviously gonna do less now, but like you could you know you can you can pair a little path Five different tests at the same time, you know, I just have them in different windows open

But yes, so the other thing I want to talk about is like I have always been really bad at like

Like I heat maps I struggle with like I have a hard time going to heat map. I'm looking Hey, I think I was the first person as a heat map API that helps because I can then just have clawed be like hey What's you know, what am I looking at but I don't know. I've just like struggled at like just looking at heat maps or even like user recordings like I used to watch them. I'm like what is this like so this person left like what does it tell me Or I just run on the patience the thing that has always made the most tangible sense to me is is asking customers

It's just like the qualitative and I just think that people are not doing enough of that I think software does it well, you know, like I just think and you see it's like much more like Gated driven and much less so we do a few different things, but I've found it to be incredibly helpful and obviously I but I think in general I just get so much more out of like a bunch of either customers or prospects Looking at a site and be like this is what I'm confused about so like typical

You know user research. I know you guys know the for people for listeners is like you know There's their sites like user research.com or there's a bunch of other things where you can you can give people a prompt and have them just pretend their Prospect go to your website often they're like, you know stay at home moms or whatever, but you can even like do it by demo depending on what tool you're using and they're just like hey

This I don't I'm confused. I'm looking for this product and I don't know where to find it or whatever and then obviously if you The old way to do it as manual now I can talk about how we do it with AI, but you can consolidate a lot of those together and you know try to automate that That's been I think one of the one of if not the biggest thing for me is so there's obviously you have all your Your quantitative your GA4 your heat map your intelligence that's all important

You have your reviews like you could obviously Use AI to analyze your reviews, but but just that user research and whether you're Physically getting on a call of people talking through it whether you're using one of these software is whether like whatever it is like I think people should do it more and it's been incredibly helpful for for us and any big project Look we kick it off like we'll do a study

before we're doing any big project and you'll use some of the tools you mentioned Yeah, yeah, so we have like we use two so we use type form which is just like a form builder We have a customer advisory group and so we'll we'll give it to them and we'll just be like hey Think about redoing our product page go to it open it take the survey answer, you know these questions right and some are open and did some are before after

We'll obviously use you know you don't refuse stuff like that and then we use we use listen labs is what it's called

I know simple monitoring is it too. It's it's very cool. It's like it's like Type form had a baby with like actually talking to customers and so you create your questions But people will For each question people will actually record themselves on video and so it's and so you know Obviously, then AI will there there AI models will we'll scan that and consolidate the insights

But so you can literally get people Talking and being like hey, I'm going through the website. This is what I'm confused about you know But you can send out a hundred of them hundreds of them and then either put it in clutter have their AI

“I think like analyze it to like find somewhere so it's almost like you get the”

You get like the depth in the insights of actually talking to customers But you get like the speed of like surveys. So yeah, those have just been so incredibly you know helpful for us Then are you are you just taking all this data? Actually one one a question for that. Are you doing more of the of the surveying or more of like actual It's not that you're not doing actual like you're not necessarily doing

Real interviews or someone on your team is not actually getting on of zoom call or Google meet with someone And ask him questions. It's either the surveys or listen labs and it's like a recording of the person Yeah, so it's it's combination of both of those and then you upload all the that data into Clawed and and Claude can then like look at all of it and say hey here's the number one issue that is popping up the most kind of the same way that People will like upload reviews to Claude and say like what's the number one thing people like about this or the number one thing people don't like and then create like

Adds and emails and stuff around that. Yeah, so we have like yeah, so we have like APIs from them that feed in, but yeah, so since we get data from Junip which is our reviews we get from type form which is like our customer surveys we do listen labs which is like the user research prospect stuff There's a few other sources like post purchase surveys we have and so that's like the qualitative side and so That'll be part of like when we kick off like a CRO process or something like that

It'll have that and then it'll also use a quantitative side that's like your heat map your GA4 or your intelligence test and so those are like Six seven like really rich data sources and it'll pull together and it'll build a brief that's like the first step is to like build a brief

And you know put it all together here's here's what people are saying here's how the data validates that or doesn't validate it

You know, and it'll like put it in their brief and actually like recommend how we should test so like that's our first, you know, step of it

“But I think I think a lot of people listen probably do a really good job with the data side and whatever”

I just like however you do it. I just think like talking to customers that qualitative like user research is For me and incredibly helpful like well we'll run one of these listen lab studies and I'll have like 20 ideas to test I look at heat map and I'm like, I don't know what I'm like. I don't know like I just I personally struggle with it So I just have an easier time with the qualitative stuff So are there any I'd love like

Are there any examples that stick out in your mind of things that you began testing or learnings that you developed from this more qualitative approach? Yeah, yeah, let me think I mean on you know

We are last six tests have been winners and I've never had that before and I truly think it's it's a big part because of this

just because of the system You know on on testing some of the homepage stuff. We tested this like shot by category where we used to just have We used to just have you because what we would do before is like, oh, we think people want best sellers Let's test best sellers and it's like well, let's ask people what they want, you know because I got a lot of our and you mentioned earlier a lot of our like best practice tests have not done well

I think that was like my biggest way. I was like all right. We have to do something different because like We're testing best sellers. We're testing social proof and it's not working. It's like what are people actually really want? Let's just ask them And so yeah, we used to just have like featured products here and it was like all right. What people want what we found out was existing customers as well as why I think you need a combination of existing and you know new new to brand people

Prospects existing customers want to see what's new. They they want to see your launches and what and what you have coming in All right, somebody new to site. They want to see you know What are the best sellers? What are highly rated so we have this one rather than having a bunch of different sections We have the shop I category that has different tabs that you can just filter on that's best sellers new kits face eyes lips and so that was that was one that we built that performed really well one's plofker one's plofker CRO launching

“The agency it's got to become it also you need to you need to in good data see fashion you need to come up with a name for your”

Like variable plus one approach to testing

Coining more terms. I had breakfast with Nate Logos over the weekend and he's all about that now as marketers

“It's our job. Yep more acronyms not there's not enough”

We have the name things and then and then short note an acronym. No, I'd like to move away from acronyms and I'd like more

Maybe I've mentioned this before but you know you've got a you've got like Bulgarian split squads and Romanian deadlifts and like those names are so

“much cooler than if they were just you know deadlifts and split squads”

So we need that like Cody's Cody's

Sierra's the testing. That's like blanched multi-variant Sierra testing. Yeah, I we got to fill in the blanched

“Cody for next episode. That's what we're doing the intro”

Well just brainstorm them for an hour. Yeah, yeah