Your metrics cards show different states depending on the label selected, the traffic distribution, the data available, and whether a baseline is selected. The following is a summary state of each card.
For more information about the Metric impact tab, refer to the Metric impact guide.
Note: Under the mean impact, you can see a range. These two values indicate the extreme values for your impact. The shorter the range, the more accurate the impact. The wider the range, more possibilities are available, which make the impact less accurate.
Desired card
When a metric card displays Desired and is green, it’s because of the following:
- The change matches the desired direction, for example, you wanted this metric to increase, and it did when comparing the treatment to the baseline.
- The p-value is less than the defined significance threshold of 0.05 or your organizational-wide significance setting. In this case, there is evidence that the treatment selected had a different impact on the metric than the baseline treatment selected.
- The metric had sufficient power. Data was collected from enough users to satisfy the desired minimum detectable effect and default power threshold.
Undesired card
When a metric card returns Undesired and is red, it’s because of the following:
- The change doesn't match the desired direction, for example, you wanted this metric to increase, but instead it decreased when comparing the treatment to the baseline.
- The p-value is less than the defined significance threshold of 0.05 or your organizational-wide significance setting. In this case, there is evidence that the treatment selected had a different impact on the metric than the baseline treatment selected.
- The metric had sufficient power. Data was collected from enough users to satisfy the desired minimum detectable effect and default power threshold.
Inconclusive card
A metric card returns Inconclusive if the metric impact is undecided. The card displays a yellow state if:
- There is insufficient evidence to believe that there was an impact. Because the p-value is greater than the defined significance threshold of 0.05 or your organization-wide significance setting, there is little evidence that the treatment selected had a different impact on the metric than the baseline treatment selected.
- The observed effect is less than the defined effect threshold. In this case, the observed effect doesn't meet the desired minimum detectable effect. Refer to the Sample size and sensitivity calculators guide for more information about detectable effect sizes using our calculators.
- The range goes from a negative to a positive range. It’s inconclusive because the impact could be positive or negative.
Baseline value = 0
When the baseline treatment has a metric value of 0, the relative impact can’t be calculated and displays N/A on the card. Instead, the confidence interval is provided based on the absolute impact between the two treatments. The same information is provided as the statistically significant and statistically inconclusive cards but in absolute rather than relative terms.
Resolving metric issues
The following explains why the metric cards returned the results that you got and what to do to resolve them.
Needs more data card
A metric card returns with Needs more data when the treatments being compared haven't reached the minimum sample size. We require a sample size of at least 355 in each treatment before we calculate significance for your metrics. For more information, refer to the Statistical significance.
Statistically not possible
A metric card returns with Statistically not possible for one of the following reasons:
- The metric is grouped across users. Metrics that are calculated across your customers and not normalized per customer (or the experimental unit) don't show a statistical comparison. These metrics are good for understanding overall trends, but if you want to see the statistical impact per customer, update the metric definition.
Note: The creation of new ‘across’ metrics is no longer supported, but existing ‘across’ metrics continues to return this state.
-
The Any targeting rule is selected. Select a different targeting rule from the With targeting rule menu list to calculate statistical impact where available.
No baseline
A metric card returns with No baseline when you are viewing metrics for a single treatment. Select a baseline treatment to see a statistical comparison where available.
Not available card
A metric card returns with Metric not available for the following reasons. If the troubleshooting tips don’t resolve your issue, and you continue to have problems with your metrics, contact support@split.io.
| Not available because | Troubleshooting tips |
|---|---|
| The calculation has not yet run for this flag. | The calculation runs within the first 15 minutes of a change in the feature flag's version. If it has been 15 minutes and you still see this issue, contact support@split.io. Automatic calculations are run for feature flag versions which include a percentage targeting rule. Click the Recalculate metrics button to run on-demand calculations at any time. |
| The metric either created or modified after the metrics impact was last updated. | The duration between updates scales with the length of the version. At the beginning of a version, calculations are run every 15 minutes for definitions updated in the past hour. The time between these calculations increases incrementally through the duration of a version. Feature flags update every hour for the first 12 hours and then alternate hours until it has been running for 24 hours. The calculation schedule then moves to daily until day 14 of a feature flag version. Final calculations run on days 21 and 28 of a feature flag version. The older the experiment, the less likely that the data collected in the last few hours can move the metric. Click the Recalculate metrics button to rerun your calculations. |
| No users have an impression for at least one of the treatments. | This message appears if you are comparing two treatments and one of the treatments has no samples. Ensure that the version and targeting rule you selected is serving traffic to both treatments. |
| No users have met the metric's filter by condition for at least one of the treatments. | This message appears if the metric has a filter criteria in its definition, for example, measure this metric for users who clicked this button. Ensure that the customers who are in the treatment and firing the track event used in the metrics calculation have also fired the filter event. |
| This metric is an average, but no events associated with this metric have been received for any user for at least one of the treatments. | This message appears when you are looking at the average value and Split hasn't received any events to take an average on. Ensure that you are sending the event that you want the average value for. |
| This metric is a ratio, but no events associated with the denominator have been received for any user for at least one of the treatments. | This message appears when you are calculating the ratio of two events and Split hasn't received any events for the denominator. Ensure that you are sending your events properly. |
| No users have an impression for the treatment. | This message appears if you are looking at a single treatment with no baseline and there are no samples. Ensure that the version and targeting rule you selected are serving traffic to the treatment you selected. |
| The calculation did not return. Our support engineering team has been notified. | The support team was notified and our alerts have been triggered. |
About individual metric card messages
The following describes the messages you get for each of the metric cards.
Desired, Undesired, and Inconclusive cards. When you hover over the question mark icon of the Desired, Undesired, and Inconclusive metrics card, the following appears:
- The metric values for both treatments
- The p-value
In addition, click the following for additional functionality:
Note: The following selections appear in all the metric cards described in this section.
- View details. Takes you to the Impact snapshot chart.
- More actions. Allows you to edit your metric definition. Also allows you to either add as a key metric or remove a key metric back to organizational metrics.
More data card. When you hover over the question mark icon of the More data card, the following appears:
Not available card. When you hover over the question mark icon of the Not available metric card, the message that appears provides specific reason why the metric value is not available.
Actions you can perform
The More actions selection appears when you hover over a specific metric card. This menu list allows you to do the following:
- Edit metric definition. Takes you to the metric definition page in order to edit the selected metric.
- Add to key metrics. Automatically adds a selected metric to the key metrics section.
- Remove from key metrics. Automatically removes a metric from the key metrics section.
- Manage alert policies. Takes you to the Alert policy page to manage your alert policies for the selected metric.
Comments
0 comments
Please sign in to leave a comment.