Metric definition – Split Help Center

Metric definitions are designed to be human-readable, and are broken down as follows.

A. Select the desired impact: Select whether you would like to see this metric increase or decrease. In traditional A/B testing tools, this would be deemed the winning direction.

B. Select traffic type: Select what traffic type you would like to measure this metric for.

C. Measured as : Define functions to perform or calculate specific analyses on events. These functions can be performed across your traffic type or per your traffic type. Metrics that are “Measured across users” represent a total value and will not be marked statistically significant. We recommend only using metrics that are “Measured across users” to get a summary of how certain events are occurring and to use metrics that are “Measured per user” to measure if there is a statistically significant change in a metric.

Statistical comparison possible. Measured per user.
When calculating metrics per user first each user's individual contribution is calculated. Each user adds a single data point to the distribution of the user metric results, hence they all have the same weight in the result. We use the resulting distribution for statistical testing of the metrics below.

(All the examples assume that users are used for the traffic type in the experiment and the users were exposed to a particular treatment of a particular Split version.)

Function	Description	Example
Count of events per user	For each user, the number of times they perform an event is calculated.	Measure the average number of times a user visits your website. Measure the average number of support tickets a user files with your support team.
Sum of event values per user	For each user, the sum of their event values is calculated.	Measure the total value of items purchased by a user. Measure the total number of minutes a user listened to media on your website.
Average of the event values per user	For each user, the average value of the user's events is calculated.	Measure the average value of items a user purchases. Measure the average page load time a user experiences.
Ratio of two events per user	For each user, the ratio of the number of times two different events are performed is calculated.	Measure the number of hotel searches that occur for a user to make a hotel booking. Measure the number of invites sent for a user to accept the invite.
Percent of unique users	The number of distinct users who performed the event as a percent of those in the sample is calculated.	Measure what percent of your sample size clicked the checkout button. Measure the percent of users who filed a support ticket.

Statistical comparison not possible. Measured across users.
When calculating metrics across users all the events for all the users are combined into one single value. These metrics describe the overall trend and behavior of the treatment and can be used for analytic and reporting purposes. Since the data is aggregated across the whole treatment, they don’t preserve enough information for statistical testing. Once all the events are pooled we compute the metrics below.

Function	Description	Example
Count of events	The total number of times the event was performed is calculated.	Measure the total number of clicks on a new banner in your app. Measure the total number of downloads of a new white-paper on your website.
Sum of event values	The sum of the event values of the events performed is calculated.	Measure the total revenue generated by an increase in shipping costs. Measure the number of flight seats booked due to a new promotion.
Average of event values	The average of the event values of the events performed is calculated.	Measure the average item price sold on the website. Measure the average page load time of your website.
Count of unique users	The number of distinct users who performed the event is calculated.	Measure the total number of users who were exposed to a new homepage experience. Measure the number of unique users who filed a support ticket.

Advanced section

D: Filtered by (optional): Apply a filter to these analyses.
has done: Customers who have triggered an event at least once

E: Cap at: Apply a cap to your metric value

With metric capping, any outlier values in your metrics will be capped and replaced with a fixed threshold value. This will reduce the variance and increase the sensitivity of your metric. When a metric cap is set to a per user per day, this will be 24 hours from a user’s first impression within a particular version of a split.

Related articles