Ruby SDK: Upgrading from 4.x to 5.x and above

Issue

Under the hood, Split SDK has a hashing algorithm that divides users across treatments. For example, given a 50/50 split between two treatments (e.g. “on” and “off”), the hashing algorithm decides which user is in the “on” treatment and which one is in the “off” treatment.

Split has used two hashing algorithms:

• Legacy Hash (or Algorithm “1”). This is simple implementation that is optimized for speed, but suffers from uneven distributions when you have < 100 users.

• Murmur Hash (or Algorithm “2). This is an industry standard implementation that is both fast and gives even distribution whether evaluating 10, 100, or a 1M users.

Ruby SDK versions 4.x and below, had an issue which resulted in any split that was meant to use the Murmur hash would instead use the Legacy hash. This leads to the following problem: If you are using SDKs from multiple languages, Ruby would not be consistent with other languages.

Version 5.0.0 and above of the Ruby SDK fixes the issue. However, as noted above, as you upgrade the Ruby SDK, you need to account for any active splits.

For any splits that were in the process of ramp (meaning some users are being shown one treatment and others are being shown the other treatment), once you upgrade the SDK, the hashing algorithm will change, which means the users may shift from one treatment to another.

Recommendation

In a perfect situation, the new SDK Ruby is applied when all experiments are in 100% distribution, if this is not possible, the recommendation is to create a new version for such current running Splits to reset the metrics calculation, potentially this will give existing users different treatments, however, they will not be excluded from metric calculation since no treatments changes are recorded within the same version.