Metric details and trends

Overview

Validate your metric value, by understanding its trend overtime, it’s dispersion and sample population.

You can click into any metric card on the metrics impact tab to understand its trend overtime and statistical information. The table and chart will be reflective of what you have preselected within the metrics impact page such as version, targeting rule and the treatment and baseline you are using for comparison.

Header

The header section will display all the meta information associated with that metric such as owner, tags, and description.

Line Chart

With the line chart providing the trend over time, you can easily visualize the difference in the impact as well as how the error margin has changed over throughout the version of a split. The line chart represents the cumulative impact and is based on all the data we have received up until the last calculation update. You can view the data throughout the version by hovering over the point on the line chart, this will update the values in the table.

Note that the line chart will not be available if you have only selected one treatment in the metrics impact page and you will be prompted to go back to the metrics impact page to select a baseline treatment for comparison.

Table

Impact

The % impact in this section in the table will always be shown if you have selected a treatment and a baseline treatment in the metrics impact page. The error margin and p-value will only be available if a statistical comparison can be made with the metric you are analyzing. A definition of these columns are listed below.

Metric Dispersion

The information displayed within the metric dispersion section of the table is dependent on the type of metric you are analyzing. When available, you will be able to understand the minimum, maximum, median and the 95th percentile of your metric. The metrics dispersion will allow you to measure the spread of your data, or in other words the variability in your sample. This section will also include the absolute total contributing to the metric value. For example, if you are measuring the count of purchases per user you will be able to see the actual count of purchases in each treatment (as well as the uplift between the treatments). The table below highlights which columns will be available based on the type of metric you are analyzing and those which will be show as "N/A".

Sample Population

This section of the table will provide the number users in your treatment, the number of users excluded from your sample because of flipping treatments and the number of users contributing to the metric result. The excluded column highlights only the number of people being excluded from flipping treatments rather than users being excluded because they did not meet the metric’s criteria. A description of these columns are listed below;