Why are impressions for the same User Id are showing different treatments? Even though the same user id should always receive the same treatment throughout the same version of the experiment.
There are few possible root causes for this issue
The Live impressions tab for the split has a "Targeting Rule" column, this usually shows the rule that was applied to the Impression. It's possible that a different rule was applied to the same user id, which caused a different treatment.
- If the Targeting Rule is specified (for example location in CA), the attribute is likely set in the SDK script, which will be a deciding factor for treatment. In the same example, if the location is set to CA, the user will get the "on" treatment, if it's set to a different value, using the same user id, the treatment is "off".
If a Bucketing Key was used in the impression along with the User Id, make sure to show the "Bucketing Key" column and verify if the same Bucket Id was used for all impressions for that specific User Id. When a treatment passes a bucket id and user id, the bucket id will be used for main id to process the impression.Note: the use of the bucketing key is being deprecated by Split.
When a user id gets the "on" treatment, they will continue to get the same treatment throughout the experiment. On the other hand, if a user id gets the "off" treatment, and the "on" percentage is increased it's possible that the same user id will get "on" treatment. So a change is feasible across multiple versions of the experiment.