How much traffic do I need to trust an A/B test result?

The exact number depends on baseline conversion rate, but aim for dozens of conversions per variant at minimum, not dozens of clicks. The closer the variants' conversion rates are to each other, the larger the sample needs to be to tell a real difference from noise.

Can I test more than two variants at once?

Yes, but every extra variant splits traffic further and pushes out the time to a significant result. In practice, 2-3 variants is a reasonable balance between test speed and how many ideas you can cover.

How long should a test run before drawing conclusions?

At minimum, until each variant accumulates enough conversions; in practice that often means several days, long enough to cover both weekdays and weekends. Cutting a test off after a few hours almost always means drawing conclusions from noise.

What if the variants come out roughly even?

That's a normal outcome — not every hypothesis produces a visible difference. Keep both as viable, favor whichever is simpler to maintain, and move on to testing the next hypothesis.

Do I need to test the landing page if I'm only testing the creative?

Better not to mix them in one test, but both layers need testing on their own. A mediocre creative paired with a strong lander often beats a strong creative paired with a mediocre lander on final ROI.

A/B Testing Creatives and Landers

A/B testing creatives and landers is the only reliable way to find out which setup actually drives conversions instead of just looking good at a glance. Without a systematic test, buyers end up scaling whichever variant they happened to see first, not the one that performs best. Here's how to structure a test properly, how much traffic you actually need before drawing conclusions, and why sticky sessions per visitor matter so much for clean results.

Why test if a setup already 'seems to work'

'Seems to work' and 'works as well as it could' are different things — a 20-30% gap in conversion rate between two lander variants is often invisible to the naked eye but very visible once you scale the budget. Testing creatives shows what visuals and copy actually resonate with your audience, not the audience of whoever you spotted that creative from. Testing the landing page — pre-lander, offer, block order — shows exactly where visitors drop out of the funnel before installing the PWA. Testing isn't a one-off check before launch — it's an ongoing process for as long as the setup runs.

What to test: creative, pre-lander, landing page, CTA

At the creative level, test format (video, static, carousel), the opening frame or headline, and the core message — those drive CTR and the quality of the audience that reaches the landing page. At the pre-lander level, test the engagement mechanic itself (wheel of fortune, quiz) and whether it actually leads into the offer or just entertains. At the landing-page level, test the headline, block order, CTA button color and copy, and how fast a visitor reaches the PWA install prompt. Never change more than one layer in a single test, or you won't know which change actually moved the result.

How to structure a proper test: one variable, an honest split

Change one variable per test — change the headline and the button color at the same time and you won't know which one worked. Split traffic evenly and randomly between variants, not by time of day or day of week, or traffic-quality differences end up driving the result instead of the variant itself. Don't draw conclusions from the first 10-20 conversions — statistical noise on a small sample easily masquerades as a winning variant. Aim for solid statistical significance, not a gut call after an hour of watching the dashboard.

Sticky sessions per visitor: why they matter

A sticky session locks a given visitor into the same test variant for the duration of their sessions on the landing page, instead of showing a random variant on every visit. Without sticky logic, the same visitor might see variant A first and variant B an hour later, which skews engagement and conversion numbers for both variants at once. That matters especially for PWAs, where a visitor may open the landing page several times before installing — the install needs to be credited to the variant they saw first. APEX's split testing has sticky-by-visitor logic built in, so there's no need to bolt on a separate service for correct traffic distribution.

How to read test results correctly

Creative CTR only shows ad interest, not audience quality — a high CTR paired with a low landing-page CR often means the creative is pulling in irrelevant clicks. Watch CR to PWA install and, where available, the share of visitors who go on to subscribe to push — that reflects real engagement better than clicks on a button. Track CPA and overall ROI per variant, not just conversion rate — a variant with a slightly lower CR but a noticeably cheaper cost per click can still come out ahead. Log test results in one place — without a history, you'll end up re-testing things you already checked a month ago.

Common split-testing mistakes

Running more than 3-4 variants at once spreads traffic so thin that no variant reaches a usable sample size in a reasonable time. Stopping a test the moment one variant pulls ahead, without waiting for the numbers to stabilize over several days. Ignoring seasonality and day-of-week effects — weekday and weekend behavior can differ enough to distort a comparison between variants that didn't run at the exact same time. Forgetting to switch off the losing variant after the test — it keeps eating traffic and budget for no benefit.

Scaling the winning variant

Before pushing the full budget to the winner, re-run the test on a fresh slice of traffic — sometimes a first-round win doesn't hold up on a larger sample. Scale gradually, watching whether CR holds as volume grows — audiences reached at larger scale are often colder than the ones that hit the initial test. Keep losing variants on file — a few weeks later, with a refreshed audience or a different geo, they sometimes become competitive again. Kick off a new round of tests on the same landing page at least every few weeks to stay ahead of creative fatigue.

FAQ

How much traffic do I need to trust an A/B test result?: The exact number depends on baseline conversion rate, but aim for dozens of conversions per variant at minimum, not dozens of clicks. The closer the variants' conversion rates are to each other, the larger the sample needs to be to tell a real difference from noise.
Can I test more than two variants at once?: Yes, but every extra variant splits traffic further and pushes out the time to a significant result. In practice, 2-3 variants is a reasonable balance between test speed and how many ideas you can cover.
How long should a test run before drawing conclusions?: At minimum, until each variant accumulates enough conversions; in practice that often means several days, long enough to cover both weekdays and weekends. Cutting a test off after a few hours almost always means drawing conclusions from noise.
What if the variants come out roughly even?: That's a normal outcome — not every hypothesis produces a visible difference. Keep both as viable, favor whichever is simpler to maintain, and move on to testing the next hypothesis.
Do I need to test the landing page if I'm only testing the creative?: Better not to mix them in one test, but both layers need testing on their own. A mediocre creative paired with a strong lander often beats a strong creative paired with a mediocre lander on final ROI.

Build your PWA in APEX in minutes

Cloaking, anti-bot, push, split tests and your own domains — in one service.

Get started free