Validate your metric by understanding its value and impact overtime, its dispersion, and sample population. You can click into any metric card on the metrics impact tab to understand its trend overtime and statistical information. The charts and tables are reflective of what you preselect within the metrics impact such as split version, targeting rule, and the treatment and baseline you are using for comparison. In addition, you can:
- Review the impact snapshot chart to provide you with an up-to-date, aggregated view of the expected impact over baseline for each treatment and an estimated range for that impact
- Select more treatments to compare the impact against the baseline in the impact over time chart
- Review the aggregated metric value for all treatments in the values over time chart
Accessing metric line charts and tables
To access metric line charts and tables, do the following:
- From the left hand navigation, select the desired split.
- From the Metrics impact tab in your split, in the Key metrics area, click the metric you want to analyze.
- Select the desired chart to review.
Metric meta data
The metric meta section displays the meta information associated with that metric such as owner, tags, and description.
You can view the line chart data throughout the split version by hovering over the point on the line chart which updates the values in the table. Make sure you the filters you're interested in are properly applied (e.g., are you observing the right split version, targeting rule, or baseline?).
Viewing impact snapshot
The Impact snapshot chart provides you an up-to-date, aggregated view of the expected impact over baseline for each treatment and an estimated range for that impact. This chart provides you with the capability to analyze your data in a more detailed way.
Viewing impact over time
The Impact over time chart allows you to visualize the aggregated value of the metric in each treatment of a split. The chart represents the cumulative impact and is based on all the data we have received up until the last calculation update. You can view the data throughout the version by hovering over the point on the chart which updates the values in the table.
Note that the impact over time line chart is available only if you select one treatment in the metrics impact page and you are prompted to go back to the metrics impact page to select a baseline treatment for comparison.
Viewing value over time
The Value over time chart allows you to easily visualize the value of the metric in each treatment of your split. Similarly to the impact over time chart, you can see how metric value’s error margin has changed throughout the version of a split. This chart displays as default if there is no relative impact between your treatments.
The following section describes tables in the three time charts.
All criteria are necessary and sufficient to view your data when you select a treatment and a baseline treatment and a statistical comparison is made with the analyzed metric. A description of these columns are shown below.
|Impact||The relative impact between the treatments you are comparing.|
|Error Margin||The chance (dependent on your organizations default significance threshold) that the interval between the mean +/- the error margin contains the true metric value.|
|P-value||Probability of seeing a result as least as extreme as the result we observed, if the null hypothesis were true.|
Viewing metric dispersion
The information displayed within the metric dispersion section of the table is dependent on the type of metric you are analyzing. When available, you can understand the minimum, maximum, median, and the 95th percentile of your metric. The metrics dispersion allows you to measure the spread of your data, or the variability in your sample. This section also includes the absolute total contributing to the metric value. For example, if you are measuring the count of purchases per user, you can see the actual count of purchases in each treatment and the uplift between the treatments. The table below highlights which columns is available based on the type of metric you are analyzing and those which show as N/A.
|Total / Average / Contributors||Mean||Stdev||Min||Median||95th Percentile||Max|
|Count of events per user||yes||yes||yes||yes||yes||yes||yes|
|Sum of event values per user||yes||yes||yes||yes||yes||yes||yes|
|Average of the event values per user||N/A||yes||yes||yes||yes||yes||yes|
|Ratio of two events per user||N/A||yes||yes||yes||yes||yes||yes|
|Percent of unique users||yes||yes||N/A||N/A||N/A||N/A||N/A|
|Count of events||yes||N/A||N/A||N/A||N/A||N/A||N/A|
|Sum of event values||yes||N/A||N/A||N/A||N/A||N/A||N/A|
|Average of event values||yes||N/A||N/A||N/A||N/A||N/A||N/A|
|Ratio of two events per user||yes||N/A||N/A||N/A||N/A||N/A||N/A|
|Count of unique users||yes||N/A||N/A||N/A||N/A||N/A||N/A|
|Mean||The mean is equal to the sum of all the data points in the data set divided by the number of contributors in the data set.|
|Stdev||This represents the variance of the data set as compared to the mean.|
|Min||This represents the smallest data point in the data set.|
|Median||This represents the midpoint of the data set.|
|Max||This represents the largest data point in the data set.|
|95th percentile||95% of the time, the metric value is at or below this value.|
This section of the table provides the number users in your treatment, the number of users excluded from your sample because of flipping treatments, and the number of users contributing to the metric result. The excluded column highlights only the number of people being excluded from flipping treatments rather than users being excluded because they did not meet the metric’s criteria. A description of these columns are listed below:
|In treatment||The number of unique users who meet the filter criteria and could contribute to the metric result.|
|Excluded||The number of users excluded from the analysis due to changing treatments within a rule or moving rules more than once. For more information, refer to the Attribution and exclusion guide on how we handle SRM and our approach to statistics.|
|Sample size||The number of unique users contributing to the metric result.|