Validate your metric value, by understanding its trend overtime, it’s dispersion and sample population.
You can click into any metric card on the metrics impact tab to understand its trend overtime and statistical information. The table and chart will be reflective of what you have preselected within the metrics impact page such as version, targeting rule and the treatment and baseline you are using for comparison.
The header section will display all the meta information associated with that metric such as owner, tags, and description.
With the line chart providing the trend over time, you can easily visualize the difference in the impact as well as how the error margin has changed over throughout the version of a split. The line chart represents the cumulative impact and is based on all the data we have received up until the last calculation update. You can view the data throughout the version by hovering over the point on the line chart, this will update the values in the table.
Note that the line chart will not be available if you have only selected one treatment in the metrics impact page and you will be prompted to go back to the metrics impact page to select a baseline treatment for comparison.
The % impact in this section in the table will always be shown if you have selected a treatment and a baseline treatment in the metrics impact page. The error margin and p-value will only be available if a statistical comparison can be made with the metric you are analyzing. A definition of these columns are listed below.
|Impact||The relative impact between the treatments you are comparing.|
|Error Margin||The chance (dependent on your organizations default significance threshold) that the interval between the mean +/- the error margin contains the true metric value.|
|P-value||Probability of seeing a result at least as extreme as the result we observed, if the null hypothesis were true.|
The information displayed within the metric dispersion section of the table is dependent on the type of metric you are analyzing. When available, you will be able to understand the minimum, maximum, median and the 95th percentile of your metric. The metrics dispersion will allow you to measure the spread of your data, or in other words the variability in your sample. This section will also include the absolute total contributing to the metric value. For example, if you are measuring the count of purchases per user you will be able to see the actual count of purchases in each treatment (as well as the uplift between the treatments). The table below highlights which columns will be available based on the type of metric you are analyzing and those which will be show as "N/A".
|Total / Average / Contributors||Mean||Stdev||Min||Median||95th Percentile||Max|
|Count of events per user||yes||yes||yes||yes||yes||yes||yes|
|Sum of event values per user||yes||yes||yes||yes||yes||yes||yes|
|Average of the event values per user||N/A||yes||yes||yes||yes||yes||yes|
|Ratio of two events per user||N/A||yes||yes||yes||yes||yes||yes|
|Percent of unique users||yes||yes||N/A||N/A||N/A||N/A||N/A|
|Count of events||yes||N/A||N/A||N/A||N/A||N/A||N/A|
|Sum of event values||yes||N/A||N/A||N/A||N/A||N/A||N/A|
|Average of event values||yes||N/A||N/A||N/A||N/A||N/A||N/A|
|Ratio of two events per user||yes||N/A||N/A||N/A||N/A||N/A||N/A|
|Count of unique users||yes||N/A||N/A||N/A||N/A||N/A||N/A|
|Mean||The mean is equal to the sum of all the data points in the data set divided by the number of contributors in the data set.|
|Stdev||This represents the variance of the data set as compared to the mean.|
|Min||This represents the smallest data point in the data set.|
|Median||This represents the midpoint of the data set.|
|Max||This represents the largest data point in the data set.|
|95th percentile||95% of the time, the metric value is at or below this value.|
This section of the table will provide the number users in your treatment, the number of users excluded from your sample because of flipping treatments and the number of users contributing to the metric result. The excluded column highlights only the number of people being excluded from flipping treatments rather than users being excluded because they did not meet the metric’s criteria. A description of these columns are listed below;
|In treatment||The number of unique users who meet the filter criteria and could contribute to the metric result.|
|Excluded||The number of users excluded from the analysis due to changing treatments within a rule or moving rules more than once.|
|Sample size||The number of unique users contributing to the metric result.|