When using a percentage based rollout, there will always be some randomness in how many visitors are assigned to each of your treatments. For example, when running an experiment with a 50% : 50% rollout, you are unlikely to see exactly 50% of visitors assigned to each treatment. However you should see close to that number, and if you see something very different, this may indicate what is called a Sample Ratio Mismatch (SRM).
We automatically check for a statistically significant deviation from the expected sample sizes for every split with a percentage based rollout. Learn more about sample ratio mismatches, potential causes and how we check for them here.
The size of a deviation which should be cause for concern depends on the total sample size. Smaller samples are inherently noisier, and more subject to deviations from the expected ratios of samples in each treatment, whereas larger samples tend to more closely match expected ratios. The calculator below can be used to help visualize the range of likely sample sizes when there are no underlying issues, and allow you to manually calculate a p-value for your sample size ratios.
Article is closed for comments.