Metric details and trends

Overview

Validate your metric, by understanding its value and impact over time, it’s dispersion and sample population.
You can click into any metric card on the metrics impact tab to understand its trend over time and statistical information. The table and chart will be reflective of what you have preselected within the metrics impact such as split version, targeting rule and the treatment and baseline you are using for comparison. In addition, you can select more treatments to compare the impact against the baseline in the impact over time tab or alternatively look at the metric value for all treatments in the values over time tab.

Header

The header section will display all the meta information associated with that metric such as owner, tags, and description.

Line Charts

You can view the line chart data throughout the split version by hovering over the point on the line chart, this will update the values in the table.

Impact over time

The line chart in the impact over time tab allows you to easily visualize the difference in the impact as well as how the error margin has changed throughout the version of a split. The line chart represents the cumulative impact and is based on all the data we have received up until the last calculation update. You can view the data throughout the version by hovering over the point on the line chart, this will update the values in the table.

Note that the impact over time line chart will not be available if you have only selected one treatment in the metrics impact page and you will be prompted to go back to the metrics impact page to select a baseline treatment for comparison.

Value over time

The line chart in the value over time tab allows you to easily visualize the value of the metric in each treatment of your split. Similarly to the impact over time tab you will be able to see how metric value’s error margin has changed throughout the version of a split. This chart will be shown as default if there is no relative impact between your treatments.

Table

Impact

The % impact in this section in the table will always be shown if you have selected a treatment and a baseline treatment in the metrics impact page. The error margin and p-value will only be available if a statistical comparison can be made with the metric you are analyzing. A definition of these columns are listed below.

Metric Dispersion

The information displayed within the metric dispersion section of the table is dependent on the type of metric you are analyzing. When available, you will be able to understand the minimum, maximum, median and 95th percentile of your metric. The metrics dispersion will allow you to measure the spread of your data or in other words the variability in your sample. This section will also include the absolute total contributing to the metric value. For example, if you are measuring the count of purchases per user you will be able to see the actual count of purchases in each treatment (as well as the uplift between the treatments). The table below highlights which columns will be available based on the type of metric you are analyzing and those which will be shown as "N/A".

Sample Population

This section of the table will provide the number of users in your treatment, the number of users excluded from your sample because of flipping treatments and the number of users contributing to the metric result. The excluded column highlights only the number of people being excluded from flipping treatments rather than users being excluded because they did not meet the metric’s criteria. A description of these columns are listed below;