Metrics impact tab

The Metrics impact tab shows the impact and traffic of your experiment or feature rollout on your customers. To make data-driven decisions on your features, review and interpret the data that you collect before deciding to roll out the functionality to more customers. The data on this tab allows you to ensure safe and reliable feature delivery while powering data-driven decisions. Before getting started, review the following:

• Visualize the traffic flow for each feature treatment over time to troubleshoot and analyze your flag behavior.
• Understand how your most important metrics (overall evaluation criteria) were both positively and negatively impacted to learn more about what your customers expect and how you should change your feature functionality.
• Compare the actual impact with your team's preliminary hypothesis.
• Ensure that you understand the impact and tradeoffs on your organization's guardrail and performance metrics.
• Share the impact with your team.

Viewing metrics

To view the impact of your experiments on your organization's metrics, from your selected feature flag, click the Metrics impact tab. The tab shows how your organization's metrics change when looking at customers in a particular treatment and when comparing treatments.

The Metrics impact tab is described as follows:

• View impact for. Select the version or a custom date, targeting rule, and treatments that you want to compare. When you select the treatment, you can see the number of unique keys in that treatment. For more information, refer to the Apply filters guide.
• Traffic over time. Visualize the number of unique keys (users, accounts, or your defined traffic type) that have been exposed to each treatment for the selected version or date range. This data is displayed as a line graph. A table also shows you the numerical values for the number of unique keys and the total number of impressions (some may be for the same key) for each treatment.
• Summary of key and organizational metrics. View how long your measurements have been running, and the last update time for the metrics displayed below. You can also force a recalculation of your metrics by clicking the Recalculate metrics button. This recalculation usually takes around 5 minutes but is dependent on the length of your experiment and the size of your data.
• Filter metrics. Filter down to metrics with a positive or negative impact by clicking the tile. You can deselect and view all by clicking the tile again.
• Key metrics. Select key metrics for this feature flag. If you are releasing a new feature behind this flag, display the key success metrics for this feature release. Learn about Configuring feature flag alerting for your key metrics.
• Guardrail metrics. Globally protected guardrail metrics adhere to an organization-wide alerting policy. See the Metric definition page for more information.
• Organizational metrics. The remainder of your organization's metrics are displayed. Any metrics that changed in a statistically positive and negative way are displayed first.

To learn more about analyzing and filtering data on the Metrics Impact tab, see Applying filters.

For detailed information about specific metric cards, refer to Understanding metric impact.

Automated calculation frequency

Automatic calculations are run for feature flag versions that include a percentage targeting rule. The duration between automatic calculations scales with the length of the version since the longer the experiment has run, the less likely that the data collected in the last few hours can move the metric. You can see the last calculation time on the Metrics impact tab.

The automated calculation schedule is:

• After 5 minutes, then
• After 30 minutes, then
• After 1 hour, then
• Every 1 hour until 12 hours, then
• Every 2 hours until 24 hours, then
• Every 1 day until 7 days, then
• On day 14, then
• On day 21, then
• On day 28

Manually recalculating metrics

You can manually run calculations on-demand by clicking the Recalculate metrics button. The following discusses when you should recalculate:

• If you create or modify a metric after the last updated metric impact calculation, recalculate the metric to get the latest results. Note: Most recalculations take up to five minutes, but can take longer, depending on the size of your data and the length of your experiment.
• If the version of this feature flag was created more than 28 days ago. If you manually recalculate after that, be aware that the recalculation only uses data within Split’s data retention period (90 days).The influence of data points prior to that are lost, even if the feature flag version is older than 90 days.

The following discusses when the recalculate button is disabled:

• When no impressions for this version are received within the current retention period (i.e., the last 90 days). To enable the recalculation, make sure your SDK is configured properly and that traffic is sent.
• A forced recalculation is already scheduled. A calculation is in progress. To recalculate, wait until it finishes.

If you have questions or need help troubleshooting, contact support@split.io.