Why setting the order of treatments matters

With Split, rules are evaluated from the top down. 

1. Whitelists - users you explicitly assign to a treatment
2. Traffic Allocation - users you want to include or exclude from rules
3. Targeting Rules - specific subsets of users targeted by attributes
4. Default Rule - users included via traffic allocation, but not in a whitelist or targeting rule
5. Default Treatment - users excluded from rules via traffic allocation

As with rules, the order of treatments matters.  For example, if you have 3 (or more) treatments you’ll want to think through order based on how you want things to change as you change percentages, particularly if you want to maintain a consistent user experience, as described here.

Let's use changing the Default Rule as an example, with default, treatment1, treatment2 set to 33/33/34.  You want to move everyone out of default and set treatment2 to 67%.

In the first case below, setting default to 0% and treatment2 to 67% would cause the users from default to move to treatment1 and the users from treatment1 to move to treatment2.  Clearly, this may not be the right experience for your users.

Rather, in the example below, when you set default to 0% and treatment2 to 67% this does not impact the users in treatment1.  This would also be true if you set treatment1 to 66% and default to 0%.  The 34% assigned to treatment2 would remain consistent.  See how this can apply to rolling out an experiment below.

Let's say you have more than three treatments.  In this case, the default was the safe treatment and if any of the treatments were found to have a bad experience the plan is to move those users to default.  Below we are moving the users from treatment2 to default. Doing so will impact users in treatment1 as well since the first 40 buckets, which had been distributed between default and treatment1 will now all get default. Buckets 40-59 will then get treatment1. Users who were getting treatment3 and treatment4 will be unaffected.

One way to 'move' users between multivariate treatments while only removing users from treatment2 is to take advantage of Dynamic Configuration, assuming you can represent the differences between treatments using a set of configured values.  In this case, you'd set the configuration of treatment2 to the default settings, which would give them the default experience.

Allocating Traffic for Experimentation

Traffic Allocation allows you to exclude a percentage of users from an experiment. For example, you can set Traffic Allocation to 20% and then set the default rule to 50/50.  Effectively, each arm that's included in the Allocation will get 10% of traffic.  The advantage of this approach is that as you add traffic to the rules no users will switch from, in our example, treatment1 to treatment2.  They will move from unallocated to one of the two treatments.

But what if you want to see all of the impressions on the Metrics impact tab? Even if you're not going to include the unallocated or default users in the analysis. Again, order matters if you want to make sure adding traffic doesn't move users that are in the experiment between treatments.  In this case, you'd order the excluded users to be between the two treatments.

As you roll-out more users, by decreasing the percentage in default and increasing each treatment, existing users in those two treatments will remain in the same treatment they were in previously.

Whitelisting

Whitelist evaluations are done in the order of the treatments. So, in the case below, even if user 12345 is in the beta-users or Employee segments they will get current, because that's evaluated first.  Similarly, if user 67891 is in the users segment they will also get current, not treatment1, even though they are explicitly whitelisted in treatment1.

​