Metric details and trends

Validate your metric by understanding its value and impact overtime, its dispersion, and sample population. You can click into any metric card on the metrics impact tab to understand its trend overtime and statistical information. The charts and tables are reflective of what you preselect within the metrics impact such as feature flag version, targeting rule, and the treatment and baseline you are using for comparison. In addition, you can:

• Review the impact snapshot chart to provide you with an up-to-date, aggregated view of the expected impact over baseline for each treatment and an estimated range for that impact
• Select more treatments to compare the impact against the baseline in the impact over time chart
• Review the aggregated metric value for all treatments in the values over time chart

Accessing metric line charts and tables

To access metric line charts and tables, do the following:

1. From the left navigation, select the desired feature flag.
2. From the Metrics impact tab in your feature flag, in the Key metrics area, click the metric you want to analyze.
3. Select the desired chart to review.

Metric meta data

The metric meta section displays the meta information associated with that metric such as owner, tags, and description.

Line Charts

You can view the line chart data throughout the feature flag version by hovering over the point on the line chart which updates the values in the table. Make sure you the filters you're interested in are properly applied (e.g., are you observing the right feature flag version, targeting rule, or baseline?).

Viewing impact snapshot

The Impact snapshot chart provides you with an up-to-date, aggregated view of the expected impact over the baseline for each treatment and an estimated range for that impact.

Analyzing dimensions using Impact snapshot

This chart provides you with the capability to analyze your data in a more detailed way using your dimensions and key metrics. Once you analyze your dimensions using the impact snapshot, you can:

• Unlock deeper insights. Understand unexpected spikes in your metric results that are driving top-line metrics at a dimensional level to better understand what action to take next.
• Run more data-driven experiments. Iterate on your next hypotheses or follow-up experiments using the insights on what worked or didn’t during your past experiments.

Important: Use this for exploratory analysis to help you investigate your unexpected results, Use this to inform your next experiment. Multiple comparison corrections are not applied. For more information, refer to the Multiple comparison correction guide.

Note: You can only calculate dimensional analysis for key metrics. Refer to the Metrics impact guide for more information on key metrics.

To analyze your dimensions, do the following:

Note: Dimensions are only calculated for relevant metrics. To properly calculate your dimensions, ensure that your event property and event property values are set within the event type.

1. Navigate to the feature flag and key metric that you want to analyze.
2. Under the Impact snapshot, go to the Select a dimension field and select the desired dimension. The Impact snapshot for the selected dimension displays.
3. If you're an admin, the Manage dimension link appears, which provides a direct link to the dimension management page.

Note: While we only calculate percentage impact for up to five dimension values configured, we bucket all additional property values that weren’t specified in the dimension definition as “Other”.

If you have questions or concerns, contact support@split.io.

Viewing impact over time

The Impact over time chart allows you to visualize the aggregated value of the metric in each treatment of a feature flag. The chart represents the cumulative impact and is based on all the data we have received up until the last calculation update. You can view the data throughout the version by hovering over the point on the chart which updates the values in the table.

Note that the impact over time line chart is available only if you select one treatment in the metrics impact page and you are prompted to go back to the metrics impact page to select a baseline treatment for comparison.

Viewing value over time

The Value over time chart allows you to easily visualize the value of the metric in each treatment of your feature flag. Similarly to the impact over time chart, you can see how metric value’s error margin has changed throughout the version of a feature flag. This chart displays as default if there is no relative impact between your treatments.

Metric tables

The following section describes tables in the three time charts.

Impact

All criteria are necessary and sufficient to view your data when you select a treatment and a baseline treatment and a statistical comparison is made with the analyzed metric. A description of these columns are shown below.

Title	Description
Impact	The relative impact between the treatments you are comparing.
Error Margin	The chance (dependent on your organizations default significance threshold) that the interval between the mean +/- the error margin contains the true metric value.
P-value	Probability of seeing a result as least as extreme as the result we observed, if the null hypothesis were true.

Viewing metric dispersion

The information displayed within the metric dispersion section of the table is dependent on the type of metric you are analyzing. When available, you can understand the minimum, maximum, median, and the 95th percentile of your metric. The metrics dispersion allows you to measure the spread of your data, or the variability in your sample. This section also includes the absolute total contributing to the metric value. For example, if you are measuring the count of purchases per user, you can see the actual count of purchases in each treatment and the uplift between the treatments. The table below highlights which columns is available based on the type of metric you are analyzing and those which show as N/A.

	Total / Average / Contributors	Mean	Stdev	Min	Median	95th Percentile	Max
PER USER
Count of events per user	yes	yes	yes	yes	yes	yes	yes
Sum of event values per user	yes	yes	yes	yes	yes	yes	yes
Average of the event values per user	N/A	yes	yes	yes	yes	yes	yes
Ratio of two events per user	N/A	yes	yes	yes	yes	yes	yes
Percent of unique users	yes	yes	N/A	N/A	N/A	N/A	N/A
ACROSS USERS
Count of events	yes	N/A	N/A	N/A	N/A	N/A	N/A
Sum of event values	yes	N/A	N/A	N/A	N/A	N/A	N/A
Average of event values	yes	N/A	N/A	N/A	N/A	N/A	N/A
Ratio of two events per user	yes	N/A	N/A	N/A	N/A	N/A	N/A
Count of unique users	yes	N/A	N/A	N/A	N/A	N/A	N/A

Title	Description
Mean	The mean is equal to the sum of all the data points in the data set divided by the number of contributors in the data set.
Stdev	This represents the variance of the data set as compared to the mean.
Min	This represents the smallest data point in the data set.
Median	This represents the midpoint of the data set.
Max	This represents the largest data point in the data set.
95th percentile	95% of the time, the metric value is at or below this value.

Sample population

This section of the table provides the number users in your treatment, the number of users excluded from your sample because of flipping treatments, and the number of users contributing to the metric result. The excluded column highlights only the number of people being excluded from flipping treatments rather than users being excluded because they did not meet the metric’s criteria. A description of these columns are listed below:

Title	Description
In treatment	The number of unique users who meet the filter criteria and could contribute to the metric result.
Excluded	The number of users excluded from the analysis due to changing treatments within a rule or moving rules more than once. For more information, refer to the Attribution and exclusion guide on how we handle SRM and our approach to statistics.
Sample size	The number of unique users contributing to the metric result.