Attribution and exclusion

Overview

Unlike other experimentation platforms, Split can consume event data from any source as long as the event can be associated with a customer key and timestamp. Learn more about sending track events to Split or using Split's analytics integration with Segment.

With this architectural advantage, the Split platform allows you to send us data from any source, automatically identifies the sample population using Split impression events, and using event attribution logic intelligently attributes events to each sample based on targeting rule and treatment. Learn more about these concepts below.

Sample set

We combine the event data you sent to Split with the traffic assignment data collected by the treatment evaluations to determine whether the event was influenced by the experiment, based on whether the event occurred after the user was exposed to the experiment. Learn more about event attribution below.

When Split calculates the metric impact of a particular treatment compared to the baseline treatment, Split assigns the customers exposed to the split into samples. These samples are organized by treatment.

Split then examines the events of those users after they were exposed to the treatment to derive a metric value.

The distribution of that metric value by each sample is then compared for significance. Learn more about statistical significance in Split.

The process of mapping user-generated events to the experiments and treatments they were exposed to is known as attribution, and is a cornerstone of running accurate experiments.

Event attribution

As mentioned above, unlike other experimentation platforms, Split can consume event data from any source, as long as the events can be associated with a customer key and timestamp. Split combines this event data with the impression events (such as traffic assignment data) collected by the treatment evaluation through its attribution model.

Split's attribution model differs from other experimentation and analytics platforms in that everything is automatic and fully extensible, allowing third-party data collection. Split's attribution model is based on identifying when a user was assigned to a particular treatment, and attributing any events that occurred after that point in time.

(1) ATTRIBUTION WHEN NO RULE AND NO TREATMENT CHANGE

In most scenarios, when a user is shown a treatment, neither the treatment nor the rule change for the course of the experiment or until the version changes.

In this case, Split identifies when the user first saw the treatment, and attributes all events within 15 minutes prior to this first impression. If an event timestamp is more than 15 minutes before the impressions timestamp, it is not attributed to the experiment and treatment on the metrics impact page. This same logic applies to the end of the experiment. There is a 15-minute buffer for events to be received when the version changes before the final calculation is done and the metrics impact is frozen for that version.

Here is an illustration of this attribution following a single user's activity.

This example follows how Split attributes events when a user's rule and treatment do not change throughout the version.

e is an event, such as a click event. Events are represented below the timeline of the version.

r1 and on represent the user's impressions containing the rule (r1) and the treatment (on). At these points, Split is deciding whether the user is bucketed into a certain treatment based on the targeting rule you defined. In this example, all impression events in the timeline are for the same treatment and targeting rule.

The example in the timeline diagram shows the user's activity in your application. When calculating metrics, Split includes the shaded region, which is the events from 15 minutes before the user's first impression to 15 minutes after the end of the version. The events highlighted in pink are not included in the v1 analysis given that they fall outside the buffer windows.

(2) ATTRIBUTION WHEN RULE CHANGE AND NO TREATMENT CHANGE

In some scenarios, when a user is shown a treatment, the reason (or rule) that was used to determine the treatment may have changed. As an example, it is possible that the attribute being used in the evaluation changed and the user still received the same treatment, despite the attribute changing.

In this case, Split isolates the events and applies a 15-minute buffer to the first and last impression received for the unique rule and treatment combination.

Here is an illustration of this attribution following a single user's activity.

This example follows how Split attributes events when a user's rule changes and their treatment does not change throughout the version.

e is an event, such as a click event. Events are represented below the timeline of the version.

r1, r2, and on represent the user's impressions containing the rule (r1 or r2) and the treatment (on). At these points, Split is deciding whether the user is bucketed into a certain treatment based on the targeting rule you defined. In this example, there are two unique combinations where the rule is changing from r1 to r2 while the treatment is not changing.

The example in the timeline diagram shows the user's activity in your application. When calculating metrics for targeting rule r1, Split includes the events from 15 minutes before the user's first impression to 15 minutes after the first impression for the second rule, as represented by the blue coloured line at the top of the diagram. When isolating to r2, the same rules apply, Split includes all events from 15 minutes before the user's first impression for r2 to 15 minutes after the rule (in this example, version) change. This timeframe is represented by the orange bar at the top of the diagram. The events highlighted in pink are not included in any rule analysis given that they fall outside the buffer windows.

(3) ATTRIBUTION WHEN RULE CHANGE AND TREATMENT CHANGE

In some scenarios, the reason (or rule) that was used to determine the treatment may have changed as well as the treatment shown. For instance, when the attribute being used in the evaluation changed, and the user received a new treatment as a result of their attribute changing. As an example, imagine we target our free users (using the custom attribute free) to convert to paid and show some of the users (in treatment on) a prompt to take a certain action. After they convert, the customer attribute becomes paid, they no longer meet the targeting condition used to show them the prompt, and are shown the off treatment in a different rule. In this case, event attribution works the same as above where the rule but not the treatment changed. Split isolates the events and applies a 15-minute buffer between the unique rule treatment combination changes.

Here is an illustration of this attribution following a single user's activity.

This example follows how Split attributes events when a user's rule changes and their treatment changes throughout the version.

e is an event, such as a click event. Events are represented below the timeline of the version.

r1, r2 and on, off represent the user's impressions containing the rule (r1 or r2) and the treatment (on or off). At these points, Split is deciding whether the user is bucketed into a certain treatment based on the targeting rule defined. In this example, there are two unique combinations where the rule is changing from r1 to r2 and the treatment changing from on to off.

The example in the timeline diagram shows the user's activity in your application. Split includes the events that fall within the region represented by the blue coloured line at the top of the diagram when looking at targeting rule r1: the events from 15 minutes before the user's first impression in r1 to 15 minutes after the first impression for the second rule. When isolating to r2, the same rules apply, include the events 15 minutes before the user's first impression for r2 to 15 minutes after the rule (in this example, version) change. This is represented by the region under the orange bar at the top of the diagram. The events highlighted in pink are not included in any rule analysis since they fall outside the buffer windows.

User exclusion

When understanding attribution it is also important to understand what causes a user to be excluded from the analysis. In Split, there are two primary scenarios where the platform automatically removes the user from the analysis.

(1) USER SAW MULTIPLE TREATMENTS WITHIN THE SAME RULE

If a user is exposed to multiple treatments within a split targeting rule, you want to disqualify the user from the experiment and exclude their data when looking at your metrics. Otherwise, the user's data would be applied to both sides of the experiment, in both treatments, and cloud your results. For example, if a patient participating in a new drug trial was to switch to sugar pills halfway through the experiment, it would be inappropriate to ascribe their outcomes to either the group taking the drug or the group assigned the placebo.

In Split, this can happen for two reasons:

1. If the user is moved from a segment that is whitelisted to a segment that is whitelist off. In this scenario, the targeting rule whitelisted segment does not change, but the treatment does.
2. If you are using matching and bucketing keys, the bucketing key can change causing the same user (represented by the matching key) to receive a different treatment. If you are having issues, contact support@split.io.

(2) USER SWITCHED RULES MORE THAN ONCE

Split's engine allows users to move between rules once within the same version of a split. For example, if the targeting rule is for free users, the user may change to a paid user during the length of the experiment, therefore meeting the criteria of a different targeting rule. In the case of moving rules they could be served a different treatment and we do not exclude users who saw different treatments in different rules. However, if a user is frequently moving between rules, there may be an issue with your experiment and how you are targeting your users. To be safe and not provide bad data, we cautiously remove the user and their data from the analysis.

We exclude any user who moves between rules more than once. For example, if you have three targeting rules - 'free' 'paid' and 'VIP' - we'll exclude a user if;

• they had impressions in three or more rules, e.g. they had impressions in 'free', 'paid' and 'VIP'
• they switched between the same two rules more than once, e.g. they moved from 'free' to 'paid' and then back to 'free', so the timeframe of the rules overlapped and they got the rules mixed over time

The number of users excluded from your analysis for either of the above two reasons can be seen in the Metric Details and Trends view in the Excluded column of the Sample Population section.

Version change

When performing an online experiment, it may be the case that users are progressively added into the treatment throughout the course of the experiment as you edit your targeting rules. Changes to targeting rules trigger a new version in Split. By tracking version changes, Split can count only the events relevant to a particular treatment, and treat each targeting rule as a distinct sample population to compare. Note that you cannot look at your data across versions.

As the version is changed, Split runs a final calculate job 15-minutes after the change and freezes the metrics impact so that you can always revisit your results in the future.

Experiment window

Split allows customers to run an experiment across a 90-day time window. If your version is longer than 90 days, Split will show results for the first 90 days after a version change. 
 See above for attribution over these 90 days and what happens on version change.

If you need a longer experimental time window, contact support@split.io.

Metric Calculation flexibility

In addition to the rigor and accuracy built into our attribution model, we've also designed the system to be flexible enough to handle the fact that team's often realize during a feature release or experiment that they're interested in measuring metrics that were not originally within scope. Two of the main points of flexibility we've built in are:

• You can send in data to Split after an experiment is already running. Oftentimes, you might have already tracked some type of user action (e.g. clicks on a navigation bar) but might not have fed that data into Split ahead of running an experiment. Luckily, we don't need to set attribution off of the time an event is sent to us, you can define the timestamp of the events you send in to be when you logged the data. This allows us to receive data after events already occurred and attribute them to experiments using the rules described above by matching the timeframes using the time field for when your application logged that data!

• You can define a metric in Split after an experiment is already running. Similar to the scenario above, you might also have data that you have tracked during an experiment but haven't defined a metric in Split ahead of time. As long as Split has the events tied to a metric in our system, our system allows you to define a metric even after you've started running an experiment. On the next run of our calculation job, our system will then calculate the impact of your experiment on that new metric from when the experiment began regardless of when you defined the metric.