Population Sample Size
When running an experiment it is important to ensure you have a large enough sample to be able to detect impacts of the size that are important to you. If your sample size is too low your experiment will be underpowered and you would be unlikely to detect a reasonably sized impact.
Each metric in your experiment has a Minimum Likely Detectable Effect (MLDE) - this is the smallest change which, if it exists, is likely to be detected and shown as statistically significant. Impacts smaller than the MLDE may be missed and not reach significance because the sample size was too low to confidently distinguish the impact from random noise.
The larger the sample you have the smaller the impacts your experiment will be able to detect. It is often a trade-off between speed (not having to run the experiment longer to get a larger sample size) and sensitivity (being able to detect smaller changes). By the using the calculator below you can see how large a sample is needed to have a good chance of detecting a given effect size, if it does exist
How is it calculated?
The equation consist of the following parameters:
Target Rollout (q1 and q0)
This is your distribution percentage as defined in your targeting rule, we usually compare two variants defining one as the baseline
Conversion Rate (P1 and P0)
This value sets the actual conversion and expected conversion based on the experiment hypothesis
Default Significance Threshold (α)
The significance threshold is 0.05 (5%), which means every time you experiment, there is a 5% chance of detecting a statistically significant impact even if there is no difference between the treatments.
Default Power Threshold (β)
This is the Probability of failing to reject the null hypothesis under the alternative hypothesis (Type II error rate), please note in Split Admin page, we use the inverse of this percent.
Zα = The standard normal deviate for α Zβ = The standard normal deviate for β Pooled proportion = P = (q1*P1) + (q0*P0)
A = Zα√P(1-P)(1/q1 + 1/q0) B = Zβ√P1(1-P1)(1/q1) + P0(1-P0)(1/q0) = 0.000 C = (P1-P0)2 = 0.000
Total group size = N = (A+B)2/C = 0 Continuity correction (added to N for Group 0) = CC = 1/(q1 * |P1-P0|)
The attached sheet implements the equations above, make sure to update the orange colored cells.