←
BenLabay368 posts
BenLabay avatar

BenLabay

@BenJLabay

CEO @ speero.com Experimentation & User Research (now with AI)

◷ Joined January 2016▧ 9 Photos and videos
BenLabay avatar
BenLabay@BenJLabay·Nov 5

Not saying A/B is dead. The human web still holds most of the revenue. 100% A/B testing in 2026 is out of position, 0% is too. It's a portfolio, and when you can't run a clean test you climb confidence with cheaper proxies (simulation, research, narrow trials).

BenLabay avatar
BenLabay@BenJLabay·Nov 5

So what do you test? 4 buckets I'm working with: the eval layer (what the model sees and says about you), the data and content layer, the handoff moments where an agent pauses to ask or confirm, and decisions built from messier evidence than a clean A/B.

BenLabay avatar
BenLabay@BenJLabay·Nov 5

This isn't hypothetical for me. In a normal week I log ClickUp hours, update deals in Attio, and fix SEO issues in Webflow without opening any of them. I describe a test idea by voice, Claude interviews me on study design and writes the Airtable record.

BenLabay avatar
BenLabay@BenJLabay·Nov 5

A/B testing assumes the user opens something. A page. Pixels you can rearrange. That assumption is breaking. Agents mediate. No DOM. No session to randomize. A/B was always one method inside a bigger thing. What do you test when there's no UI?

BenLabay avatar
BenLabay@BenJLabay·Nov 4

So the bottleneck moved from "how do we ship more" to "how fast do our beliefs actually update from real signal." That's a program-level question. No single CRO or product team owns it in the old org model. IMO that's the growth function's job.

BenLabay avatar
BenLabay@BenJLabay·Nov 4

Leah's evidence that we're turning a humility tool into a confidence machine: once NIH trials had to pre-register the primary metric, the share reporting a significant win went from 57% to 8%. Now AI makes picking the metric and narrating the win 10x faster.

BenLabay avatar
BenLabay@BenJLabay·Nov 4

Why it matters: that shift rearranges who runs experimentation, where it reports, and what infra it needs. At Circus London, Leah Tharin asked a room of very experienced practitioners who works somewhere with a real dedicated growth function. Basically one hand went up. Mine.

BenLabay avatar
BenLabay@BenJLabay·Nov 4

For a decade we tested artifacts. Buttons. Headlines. Pages. AI collapsed the cost of producing those to roughly zero. The unit of test moved up: models that generate variants. Marcel Toben: "AI collapsed the cost of change. It did not collapse the cost of being wrong."

BenLabay avatar
BenLabay@BenJLabay·Nov 3

It takes 3 ingredients. Context (strategy, metrics, past tests), methodology (what good looks like in your shop), and connection (it writes to the same Airtable/Jira/ClickUp the program runs on). In Claude that's Projects, Skills, and MCP. Copilot and Glean, same recipe.

BenLabay avatar
BenLabay@BenJLabay·Nov 3

Same sentence typed into an agent instead. It doesn't file the idea, it asks: what's the construct, endowed progress? We have 3 candidate primary metrics. This audience overlaps the live pricing test, want to coordinate or re-segment? The PM walks away with a better brief.

BenLabay avatar
BenLabay@BenJLabay·Nov 3

Why this matters: the PM has the idea, the org has the strategy, the metrics library, the live tests, the past results. The form doesn't connect them. So the reviewer re-introduces that context by hand (if they're senior enough to know it), a translation tax that compounds.

BenLabay avatar
BenLabay@BenJLabay·Nov 3

A PM types "test a progress bar on onboarding" into a form. The system already knows the adjacent test was flat, the audience overlaps a live pricing test, and activation isn't this quarter's goal. The form tells them none of it. Intake shouldn't capture. It should interview.