What happens with metric results when one treatment has no events?

Question

What happens on my metric cards when I don't have events coming in for one treatment?  Will I get statistical significance?  How should I view my metric card?

Answer

The system will test the difference between any mean values, even if one of them happens to be 0.  As a result, you can still get statistically significant results, a card that turns red or green.

In this scenario we use the absolute change instead of the relative change. If one treatment is 0 and the other treatment isn't 0, as long as there is a relevant sample size and the appropriate variance in the data, the stats calculations are not impacted.

In this case, the table on the Details and Trends page will show the appropriate data for the treatments.

In addition to seeing no events in the table for one of the treatments, the chart for Impact over time will be blank, reinforcing that we are using the absolute change.

But if you select Values over time, then you will see the means metric value for all of the treatments. A treatment that has no events will have the metric value zero.