A. Select the desired impact: Select whether you would like to see this metric increase or decrease. In traditional A/B testing tools, this would be deemed the winning direction.
B. Select traffic type: Select what traffic type you would like to measure this metric for.
C. Measured as : Define functions to perform or calculate specific analyses on events. These functions can be performed across your traffic type or per your traffic type. Metrics that are “Measured across users” represent a total value and will not be marked statistically significant. We recommend only using metrics that are “Measured across users” to get a summary of how certain events are occurring and to use metrics that are “Measured per user” to measure if there is a statistically significant change in a metric.
Statistical comparison possible. Measured per user.
When calculating metrics per user first each user's individual contribution is calculated. Each user adds a single data point to the distribution of the user metric results, hence they all have the same weight in the result. We use the resulting distribution for statistical testing of the metrics below.
(All the examples assume that users are used for the traffic type in the experiment and the users were exposed to a particular treatment of a particular Split version.)
| Function | Description | Example |
|---|---|---|
| Count of events per user | For each user, the number of times they perform an event is calculated. | Measure the average number of times a user visits your website. Measure the average number of support tickets a user files with your support team. |
| Sum of event values per user | For each user, the sum of their event values is calculated. | Measure the total value of items purchased by a user. Measure the total number of minutes a user listened to media on your website. |
| Average of the event values per user | For each user, the average value of the user's events is calculated. | Measure the average value of items a user purchases. Measure the average page load time a user experiences. |
| Ratio of two events per user | For each user, the ratio of the number of times two different events are performed is calculated. | Measure the number of hotel searches that occur for a user to make a hotel booking. Measure the number of invites sent for a user to accept the invite. |
| Percent of unique users | The number of distinct users who performed the event as a percent of those in the sample is calculated. | Measure what percent of your sample size clicked the checkout button. Measure the percent of users who filed a support ticket. |
Statistical comparison not possible. Measured across users.
When calculating metrics across users all the events for all the users are combined into one single value. These metrics describe the overall trend and behavior of the treatment and can be used for analytic and reporting purposes. Since the data is aggregated across the whole treatment, they don’t preserve enough information for statistical testing. Once all the events are pooled we compute the metrics below.
| Function | Description | Example |
|---|---|---|
| Count of events | The total number of times the event was performed is calculated. | Measure the total number of clicks on a new banner in your app. Measure the total number of downloads of a new white-paper on your website. |
| Sum of event values | The sum of the event values of the events performed is calculated. | Measure the total revenue generated by an increase in shipping costs. Measure the number of flight seats booked due to a new promotion. |
| Average of event values | The average of the event values of the events performed is calculated. | Measure the average item price sold on the website. Measure the average page load time of your website. |
| Count of unique users | The number of distinct users who performed the event is calculated. | Measure the total number of users who were exposed to a new homepage experience. Measure the number of unique users who filed a support ticket. |
Advanced section
D: Value field: Select which property you want to use as the value field. Learn more.
E: Property filter: Select the property you want to filter the event by. Learn more.
Advanced section
F: Filter by (optional): Apply a filter to these analyses.
has done: Customers who have triggered an event at least once.
G: Properties: Apply a property filter to the ‘has done’ event your filtering your metric by.
H: Cap at: Apply a cap to your metric value
With metric capping, any outlier values in your metrics will be capped and replaced with a fixed threshold value. This will reduce the variance and increase the sensitivity of your metric. When a metric cap is set to a per user per day, this will be 24 hours from a user’s first impression within a particular version of a split.
Alert Policies are currently in beta
Please contact support@split.io if you would like more information or if you would like early access.
If you have created a metric that is measured on a per traffic type basis you will be able to create an alert policy for this metric. Learn more.
Comments
0 comments
Please sign in to leave a comment.