How do I ensure that once a user (or any traffic key) is given a treatment, they always receive the same treatment, irrespective of any changes to the Split rollout plan?
The goal is to 'maintain state' so that a user is always exposed to the same thing they saw the first time they visited, even if a change to the rollout plan would otherwise flip the user to another treatment. A sample use case is an offering, such as a loan application or promotional discount, that needs to remain consistent.
Split's Targeting Engine
Split is designed to be imperceptibly fast in computing treatments. In Java, for instance, treatments are computed on the order of a few hundred microseconds. This design principle means Split can be used in the most performance-intensive components of your application.
To achieve this speed, Split is designed to not retain state. Specifically, after Split serves a treatment to a user, it does not “remember” what was served. Given the scale of our customer use cases, retaining the combination of customer/key/split/treatment will negatively impact the requirement for speed in our SDKs.
Split ensures that we give end-users a “sticky” experience and don’t change treatments because the SDK uses a deterministic hashing algorithm. You can understand the details here. As long as a split’s targeting definition does not change, the user gets the exact same treatment every single time.
Of course, if you make changes to the split targeting definition, some users may be given a different treatment by moving from one treatment to another. This could happen if they fall under a different rule or if the percentages in a rule are changed. That is the intent for an overwhelming number of use cases but is not ideal in some scenarios.
There are two ways to 'maintain state': In both cases we use attributes. In the first, the attribute used is the creation date. In the second, the developer maintains the state and passes it as an attribute. The former relies on a date and does not require you to store state, while the latter requires logging the state of the treatment and passing it as the attribute.
Note: that in both cases using the “kill” switch will void both options.
For purposes of this article, we'll use a loan application as an example.
Solution 1: Use the creation date
Create targeting rules and pass a date as an attribute that represents the date the application was first rendered and passed at runtime to split your traffic.
When you change your targeting rules use date matchers with the date attribute so all applications rendered after that date receive the new allocation.
Using this methodology you can avoid storing the application state and instead store and pass the date the application was originated. In the below example, the application_creation_date is an attribute that you can pass when calling getTreatment with the application ID to tell Split the date the application is being generated.
Let's say we start the new program on November 1st. On day 1 we want v1_application=90% and v2_application=10%. You can set up the targeting rule as
On the 5th we want to split the traffic so that v1_application=80% and v2_application=20%. You can update your targeting rules as
In this scenario, the create date for an application will not change and it will continue to meet the first targeting rule. Only new applications will fall into the second targeting rule.
Finally, on day 14, we've learned that v2 is performing as expected so we set the ratios to v1_application=0% and v2_application=100%.
As noted above, the creation date does not change and existing applications will meet the first or second targeting rules. Only new applications will fall into the third targeting rule.
The downside to this approach is that it works if there are a limited number of targeting definition changes. It would be tedious to remember to change the date matchers with each rollout change and is error-prone given the manual nature.
This example will work with Split’s experimentation capabilities, showing the statistical significance for each targeting criterion above.
Solution 2: Storing which treatment was assigned to an application
With this solution, the developer must maintain state somewhere. While, by design, Split can’t keep state using the SDKs, you can retain the application_id/split/treatment combination by storing this data in, for example, a cookie or database. You could also store the data in a Split segment using the Admin API, assuming the number of unique users is not too large.
If you can store this state somewhere in a cookie or database you can configure the split once and not worry about changing date matchers with each change.
In this example, application_seen is the attribute to pass when calling getTreatment to tell Split if an applicant was already given a treatment. The value can be null or empty for first time applications without causing errors. Here’s how you’d make the call in Java:
Map<String, Object> attributes = new HashMap<String, Object>(); attributes.put("application_seen", "v1_application"); // fetch from db String treatment = client.getTreatment("USER_ID", "SPLIT_NAME", attributes);
Using this approach, any changes you make to the default rule will not affect users that had already been assigned a treatment.
If the number of applicants you see for the duration of a split is small enough (10k or less), you can create a segment within Split. See the segment API documentation. This will obviate the need to retain state in a cookie or database.
In this case, create a segment per split/treatment combination. Based on the treatment assigned you would store the ID in the appropriate segment. Then you could set up your Split using Whitelists
or you use the segment in a rule if more flexibility is needed
This, of course, involves a bit of work. Using the cookie or database solution is preferred over using segments.
Solution #2 will not fully leverage Split’s experimentation capabilities and is preferable for basic rollouts. You can measure changes across rules, but not see the statistical significance for each targeting rule since each targeting rule is serving a single treatment.