Manage alerts – Split Help Center

Get access to alert policies

Reach out to your customer success manager or support and we’ll enable in your account.

Overview

Understanding the impact of your feature flags wouldn’t be complete without the ability to know when critical changes are occurring. Split gives you the ability to create alerts that actively check for a degradation in your metrics. Alerts will be displayed both on the Targeting and Alerts tab on the split page.

Screen_Shot_2019-09-27_at_8.37.12_PM.png

Monitoring window

Actively monitoring each metric where an alert policy is defined, alerts can fire as soon as five minutes after a version change.

Split will continue to monitor and alert your team of a metric degradation for up to 24 hours after a version change. You can change the monitor window under the Monitor and experiment settings.

Alert details

If an alert is fired the following information will be provided:

Column Header	Description
Alert fired	Provides the timestamp when the alert was fired. The dropdown option is also where you can take action on this alert.
Activity	Provides information on the status of the alert, whether it has been dismissed or auto-resolved (learn more below). Hover over to access the timestamp of the action in this column.
Name of Policy	Provides the name of the metric alert policy. Click the name to go directly to the alert policy builder.
Rule	Provides the rule within the split's targeting rules that caused the alert to fire.
Relative Impact %	Provides the degradation percentage that was detected with the error margin of this percentage. Hover over to access the timestamp of when the degradation was detected.
Absolute Impact	Provides the degradation that was detcted in an absolute value with the error margin of this value. Hover over to access the timestamp of when the degradation was detected.
Metric value (baseline)	Provides the metric value in absolute terms as well as the baseline treatment used for alert monitoring.
Metric value (treatment)	Provides the metric value in absolute terms as well as the treatment used to compare against the baseline treatment for alert monitoring.
Relative Threshold %	Provides the degradation threshold configured in the alert policy definition in relative terms or a transalation of the absolute threshold, if this was used in the alert policy definition. Hover over this value to see which threshold type was used in the alert policy definition.
Absolute Threshold	Provides the degradation threshold configured in the alert policy definition in absolute terms or a transalation of the relative threshold, if this was used in the alert policy defintion. Hover over this value to see which threshold type was used in the alert policy definition.

Alerts actions

You can take the following actions on the alert tab, Kill split or Dismiss alert.

Action	Description
Kill split	If you decide to kill a split due to an alert, the default treatment overrides the existing targeting rules and is returned for all customers.
Dismiss alert	If you decide to dismiss an alert, this alert will be silenced for the remainder of the 30 minute monitoring period.

Understanding when an alert does not fire

An alert policy may not fire an alert due to an alert policy configuration or changes to an alert policy definition.

Cause of Alert Not Firing	Description
The alert policy has a different traffic type than the split you want to be alerted on.	If the metric that is associated with an alert policy has a different traffic type than a split, that split will not be eligible to be alerted on.
The alert policy does not have an alert condition for all environments.	Multiple alert conditions can be created within alert policies that correspond to each of your environments. If you are viewing a split in a environment, that does not have an alert condition associated with it, this split will not be eligible for alert policies in that environment.
The alert policy threshold has changed	An alert may have fired however if a user has edited the degradation threshold during the monitoring period to a level where this would no longer cause an alert, the alert will become inactive and show ‘auto resolved due to threshold change’ in the activity column.
The alert policy has been deleted	An alert may have been fired but if a user has deleted an alert policy during the monitoring period, the alert will become inactive and show ‘alert policy deleted’ in the activity column.
The metric definition has changed	An alert may have been fired but if a user has edited a metric that is tied to an alert policy during the monitoring period, the alert will become inactive and show ‘auto resolved by metric edited’ in the activity column.

Alerts may also not be firing due to how the split has been configured and may be ineligible for an alert policy.

Cause of Alert Not Firing	Description
No percentage allocation in the split’s targeting rules	Split requires user to have at least two treatments in a targeting rule to measure for and detect a metric degradation.
Alert baseline treatment not included in a specific targeting rule	The alert baseline must be included in one of the targeting rules and be allocated more than 0% of traffic.
The split has changed version	You will only be alerted if there are degradations in your metrics for the split's active version.
The sample size in the split is too low to detect a degradation	If either the baseline or the treatment’s sample sizes are less than the currently used minimum requirement for normality (355 samples in each treatment), we do not alert, regardless of the difference in the metrics.