Metric details and trends

Validate your metric by understanding its value and impact overtime, its dispersion, and sample population. You can click into any metric card on the metrics impact tab to understand its trend overtime and statistical information. The charts and tables are reflective of what you preselect within the metrics impact such as split version, targeting rule, and the treatment and baseline you are using for comparison. In addition, you can:

• Review the impact snapshot chart to provide you with an up-to-date, aggregated view of the expected impact over baseline for each treatment and an estimated range for that impact
• Select more treatments to compare the impact against the baseline in the impact over time chart
• Review the aggregated metric value for all treatments in the values over time chart

Accessing metric line charts and tables

To access metric line charts and tables, do the following:

1. From the left hand navigation, select the desired split.
2. From the Metrics impact tab in your split, in the Key metrics area, click the metric you want to analyze.
3. Select the desired chart to review.

Metric meta data

The metric meta section displays the meta information associated with that metric such as owner, tags, and description.

Line Charts

You can view the line chart data throughout the split version by hovering over the point on the line chart which updates the values in the table. Make sure you the filters you're interested in are properly applied (e.g., are you observing the right split version, targeting rule, or baseline?).

Viewing impact snapshot

The Impact snapshot chart provides you an up-to-date, aggregated view of the expected impact over baseline for each treatment and an estimated range for that impact. This chart provides you with the capability to analyze your data in a more detailed way.

Viewing impact over time

The Impact over time chart allows you to visualize the aggregated value of the metric in each treatment of a split. The chart represents the cumulative impact and is based on all the data we have received up until the last calculation update. You can view the data throughout the version by hovering over the point on the chart which updates the values in the table.

Note that the impact over time line chart is available only if you select one treatment in the metrics impact page and you are prompted to go back to the metrics impact page to select a baseline treatment for comparison.

Viewing value over time

The Value over time chart allows you to easily visualize the value of the metric in each treatment of your split. Similarly to the impact over time chart, you can see how metric value’s error margin has changed throughout the version of a split. This chart displays as default if there is no relative impact between your treatments.

Metric tables

The following section describes tables in the three time charts.

Impact

All criteria are necessary and sufficient to view your data when you select a treatment and a baseline treatment and a statistical comparison is made with the analyzed metric. A description of these columns are shown below.

Viewing metric dispersion

The information displayed within the metric dispersion section of the table is dependent on the type of metric you are analyzing. When available, you can understand the minimum, maximum, median, and the 95th percentile of your metric. The metrics dispersion allows you to measure the spread of your data, or the variability in your sample. This section also includes the absolute total contributing to the metric value. For example, if you are measuring the count of purchases per user, you can see the actual count of purchases in each treatment and the uplift between the treatments. The table below highlights which columns is available based on the type of metric you are analyzing and those which show as N/A.

Sample population

This section of the table provides the number users in your treatment, the number of users excluded from your sample because of flipping treatments, and the number of users contributing to the metric result. The excluded column highlights only the number of people being excluded from flipping treatments rather than users being excluded because they did not meet the metric’s criteria. A description of these columns are listed below: