Validate your metric, by understanding its value and impact over time, it’s dispersion and sample population.
You can click into any metric card on the metrics impact tab to understand its trend over time and statistical information. The table and chart will be reflective of what you have preselected within the metrics impact such as split version, targeting rule and the treatment and baseline you are using for comparison. In addition, you can select more treatments to compare the impact against the baseline in the impact over time tab or alternatively look at the metric value for all treatments in the values over time tab.
The header section will display all the meta information associated with that metric such as owner, tags, and description.
You can view the line chart data throughout the split version by hovering over the point on the line chart, this will update the values in the table.
Impact over time
The line chart in the impact over time tab allows you to easily visualize the difference in the impact as well as how the error margin has changed throughout the version of a split. The line chart represents the cumulative impact and is based on all the data we have received up until the last calculation update. You can view the data throughout the version by hovering over the point on the line chart, this will update the values in the table.
Note that the impact over time line chart will not be available if you have only selected one treatment in the metrics impact page and you will be prompted to go back to the metrics impact page to select a baseline treatment for comparison.
Value over time
The line chart in the value over time tab allows you to easily visualize the value of the metric in each treatment of your split. Similarly to the impact over time tab you will be able to see how metric value’s error margin has changed throughout the version of a split. This chart will be shown as default if there is no relative impact between your treatments.
The % impact in this section in the table will always be shown if you have selected a treatment and a baseline treatment in the metrics impact page. The error margin and p-value will only be available if a statistical comparison can be made with the metric you are analyzing. A definition of these columns are listed below.
|Impact||The relative impact between the treatments you are comparing.|
|Error Margin||The chance (dependent on your organizations default significance threshold) that the interval between the mean +/- the error margin contains the true metric value.|
|P-value||The probability of seeing a result as least as extreme as the result we observed, if the null hypothesis were true.|
The information displayed within the metric dispersion section of the table is dependent on the type of metric you are analyzing. When available, you will be able to understand the minimum, maximum, median and 95th percentile of your metric. The metrics dispersion will allow you to measure the spread of your data or in other words the variability in your sample. This section will also include the absolute total contributing to the metric value. For example, if you are measuring the count of purchases per user you will be able to see the actual count of purchases in each treatment (as well as the uplift between the treatments). The table below highlights which columns will be available based on the type of metric you are analyzing and those which will be shown as "N/A".
|Total / Average / Contributors||Mean||Stdev||Min||Median||95th Percentile||Max|
|Count of events per user||yes||yes||yes||yes||yes||yes||yes|
|Sum of event values per user||yes||yes||yes||yes||yes||yes||yes|
|Average of the event values per user||N/A||yes||yes||yes||yes||yes||yes|
|Ratio of two events per user||N/A||yes||yes||yes||yes||yes||yes|
|Percent of unique users||yes||yes||N/A||N/A||N/A||N/A||N/A|
|Count of events||yes||N/A||N/A||N/A||N/A||N/A||N/A|
|Sum of event values||yes||N/A||N/A||N/A||N/A||N/A||N/A|
|Average of event values||yes||N/A||N/A||N/A||N/A||N/A||N/A|
|Ratio of two events per user||yes||N/A||N/A||N/A||N/A||N/A||N/A|
|Count of unique users||yes||N/A||N/A||N/A||N/A||N/A||N/A|
|Mean||The mean is equal to the sum of all the data points in the data set divided by the number of contributors in the data set.|
|Stdev||This represents the variance of the data set as compared to the mean.|
|Min||This represents the smallest data point in the data set.|
|Median||This represents the midpoint of the data set.|
|Max||This represents the largest data point in the data set.|
|95th percentile||95% of the time, the metric value is at or below this value.|
This section of the table will provide the number of users in your treatment, the number of users excluded from your sample because of flipping treatments and the number of users contributing to the metric result. The excluded column highlights only the number of people being excluded from flipping treatments rather than users being excluded because they did not meet the metric’s criteria. A description of these columns are listed below;
|In treatment||The number of unique users who meet the filter criteria and could contribute to the metric result.|
|Excluded||The number of users excluded from the analysis due to changing treatments within a rule or moving rules more than once.|
|Sample size||The number of unique users contributing to the metric result.|