A/B Testing Push Notifications
A/B testing push notifications is the fastest way to squeeze more out of a base you've already built, with no extra traffic spend. The catch is that most buyers eyeball the copy and call the test the moment CTR spikes, even though that spike doesn't always mean more money. Here's what's actually worth testing in push notifications and how to read the results correctly.
What to test first
Headline and body copy usually produce the widest CTR spread — test a genuinely different angle (urgency, a bonus, personalization), not just synonyms for the same phrase. Icon and image get tested separately from copy: the same text with a different icon can pull a different CTR purely from recognizability or contrast in the notification tray. Send timing is the third variable worth isolating, especially comparing an immediate post-subscribe send against a scheduled, delayed one.
How to set up a split test correctly
The test needs to be sticky by visitor — the same user should consistently land in the same variant across repeat sends, or the result gets diluted and can't be read honestly. Change one variable per test: swapping the headline, icon, and send time all at once won't show which one actually moved the result. Let the test run until the gap between variants is stable across a meaningful segment size, not stop it an hour after the first numbers come in.
Which metrics to watch beyond CTR
A high CTR isn't automatically a good sign — a clickbait or overly aggressive headline can boost open rate while also pushing up unsubscribes or lowering the share of clicks that actually convert to a deposit. The metrics worth stacking together are CTR as an interest signal, conversion to the target action (a deposit, say) as a click-quality signal, and unsubscribe rate as a base-fatigue signal. A variant with a lower CTR but a higher deposit conversion rate and fewer unsubscribes is almost always the better bet, both immediately and for the base's LTV down the line.
Timing and local time in tests
The same push sent at different local times across a geo can show a different CTR simply because the user is either active with a phone in hand, or asleep, or at work. When testing timing, anchor to the recipient's local time, not server time — otherwise a comparison of two send slots is really just a coincidence of audience activity, not a real timing test. Retention and triggered pushes (after an abandoned deposit, say) are worth timing-testing separately from a blanket send to the whole base — they run on a different send-window logic.
A/B testing pushes in APEX
In APEX, push split tests are sticky by visitor out of the box — no need to hand-build logic so the same user doesn't see both variants on different sends. The push engine combines segmentation, triggered and scheduled sends, and the tests themselves in one place, so a test's result shows up against conversion right away, not just open rate, and the winning variant can roll out to the full segment fast.
FAQ
- How many push variants should run in one test?
- Two is usually enough to gather statistics quickly and clearly attribute which variable worked; more variants split the segment's traffic thinner and need a much bigger base before the result is readable.
- How do you know when to stop a test?
- When the gap between variants holds steady across a large enough send volume per segment rather than swinging batch to batch — stopping after the first few hundred deliveries is usually premature.
- Can a high CTR actually be a bad sign?
- Yes — a clickbait headline often lifts CTR while cutting the share of clicks that reach the target action and raising unsubscribes; read CTR, conversion, and unsubscribes together, not CTR on its own.
- Should the icon be tested separately from the push copy?
- Yes — testing them together makes it impossible to tell which one moved the result; vary the icon in its own test against copy that's already proven stable.
Cloaking, anti-bot, push, split tests and your own domains — in one service.
Get started free