Sample ratio mismatch check

Question

How can I troubleshoot a Sample Ratio Mismatch in my feature flag?

 Answer

Split’s sample ratio check detects whether there is a sampling bias in your randomization. The check ensures the the percentage of the treatments selected by the targeting rules engine matches the requested percentage within a reasonable confidence interval.

The most common reasons for a sample ratio mismatch to occur:

• Customer Exclusion: Users may be being excluded from your metric impact analysis due to flipping targeting rules twice or more. If for some reason more users are excluded from one treatment than another a sample ratio mismatch will result. You can learn more about exclusion in Split in our product documentation.

• Dropped Impressions: If spotty network coverage or some other communication issue makes it impossible for instances of the Split SDK to transmit impressions to the Split cloud, you could see a sample ratio mismatch occur if for some reason more impressions for one treatment are dropped than those for another treatment. The frequency with which the SDK flushes impressions can be tuned with an SDK initialization parameter.

If you have concerns of any potential for a bias in your targeting that may result in a sample ratio mismatch, you can conduct an A/A test before you start your experiment.