How can I troubleshoot a Sample Ratio Mismatch in my split?
Split’s sample ratio check detects whether there is a sampling bias in your randomization. The check ensures the the samples selected by the targeting rules engine match the requested distribution within a reasonable confidence interval.
A sample ratio mismatch could show for a few reasons. The most common are a result of:
Customer ID Randomization: If the customer ID used to bucket users into treatments is not random, or seemingly random, the distribution will not be randomized. Put another way, if there is a bias in the customer ID used for randomization the sample distribution across treatments will be biased. In a simple example, assume 30% of IDs used for targeting are the string “false”, the other 70% are unique in nature. All 30% of those users will be bucketed into a single treatment. This is most likely to show in Split when using separate bucketing and matching keys for targeting.
Customer Exclusion: Customers may be being excluded from your metric impact analysis due to flipping treatments twice or more, and therefore causing a sample ratio mismatch. You can learn more about exclusion in Split in our product documentation.
If you have concerns of any bias you can conduct an A/A test before you start your experiment. In this scenario your metrics and sample for each treatment should stay stable and consistent. If this is not the case, an A/A test will highlight pre-existing differences in key metrics between your treatment groups.