Impressions for the same User Id are showing different treatments. Even though the same user id should always receive the same treatment throughout the same version of experiment.
There are few possible root causes for this issue
The Results page for the Split has "Label" column, this usually show the Rule that was applied to the Impression. Its possible that different rule was applied to the same user id, which caused different treatment.
- If the Targeting Rule is specified (for example location in CA), the attribute usually is set in the SDK script, that will be a deciding factor for treatment, in the same example, if the location is set to CA, user will get "On" treatment, if its set to different value, using the same user id, the treatment is "Off".
If an Anonymous Id (Bucket Id) was used in the impression along with the User Id, make sure to show the "Bucket Id" column and verify if the same Bucket Id was used for all that specific User Id impressions. When a treatment passes a bucket id and user id, the bucket id will be used for main id to process the impression.
When a user id gets "on" treatment, they will continue to get the same treatment throughout the experiment. On the other hand, if a user id gets "off" treatment, and the "on" percentage increased its possible that the same user id will get "on" treatment. So a change from "off" to "on" is feasible in multiple versions of the experiment.