Sticky Split Testing: Why It Matters
Sticky split testing matters the moment a test runs longer than a single visit: without pinning a variant to a specific visitor, the same person can see variant A on one visit and variant B the next day. That kind of jumping around quietly breaks the whole point of comparison, because the conversion decision stops being tied to the variant that actually influenced it. Here's how sticky logic works and why a test without it can't be trusted.
What "sticky" means in a split test
Sticky means a returning visitor is remembered, through a cookie, local storage, or a visitor identifier, and served the exact same variant on every later visit as on the first one, instead of a fresh random draw each time. That covers not just a direct repeat visit to the same link, but any return at all, including a click on a push notification days after the first visit.
What breaks without sticky logic
Without a pinned variant, the same visitor can see variant A on the first click and variant B on a repeat visit or a push click-back. If the conversion happens on that second visit, it gets credited to whichever variant simply happened to be on screen at that moment, not the one that actually shaped the user's decision — the comparison between variants stops reflecting their real performance.
How this distorts test statistics
Under non-sticky randomization, the same person across several visits can smear both impression volume and conversion events unevenly between both variants, which makes the sample look bigger than it really is while the actual signal inside it gets weaker. Under those conditions, a test can report a "winner" that's really just statistical noise, not a genuine difference between variants.
How sticky logic works in a PWA setup
A visitor gets assigned a variant on the first hit, the tag is usually tied to the same identifier used for clickid or the push subscription, and that same tag decides which variant renders on every later hit, including one arriving from a push notification days after the first visit. It matters that the entry channel never overrides an already-assigned variant — otherwise the logic breaks exactly in the most common return scenario.
How to confirm sticky is actually working
The simplest check is revisiting the same link from the same device several times during the test and confirming the same variant renders every time. It's worth separately checking entry through a push notification click specifically, since that's the most common repeat entry point for an active base, and it should behave exactly like an ordinary repeat visit rather than a fresh random draw. In APEX, split tests are sticky per visitor by default, so this logic never needs building on top of your own infrastructure.
FAQ
- Isn't a fresh random draw on every visit actually the "fairer" test?
- No — for the same person, it destroys the entire point of comparison, since their decisions get smeared across both variants instead of being tied to just one of them.
- Does sticky logic work for entries from a push click, not just the first ad click?
- Yes, when implemented correctly — the visitor identifier gets pinned once and applies to every later entry, including push notification clicks days after the first visit.
- Do I need to build sticky logic myself when using APEX?
- No, APEX split tests are sticky per visitor by default — pinning a variant doesn't need any separate setup or extra work.
- What if a test already ran without sticky logic?
- It's more honest to restart the test with sticky pinning in place than to try cleaning up or reinterpreting already-distorted data after the fact.
Cloaking, anti-bot, push, split tests and your own domains — in one service.
Get started free