Why are the counts different from other tools?

Question

Why are number of visitors/impressions and/or the number of events different when I compare Split to other tools?

Answer

Most companies have a variety of tools to count and track user visits and interactions on their web and mobile applications.  While it can be helpful to compare the numbers between, say, Split and Google Analytics, it is not unusual or unexpected that the numbers are different.  There can be a number of reasons for this:

Sampling and configuration settings

Sampling: Tools such as Google Analytics sample data in common configurations, sending a representative subset of the data instead of the entire data set.  This obviously results in a potentially dramatic difference in what you see when comparing numbers.  Google Analytics will warn you if you the data is being sampled.  You can configure the precision/speed ratio or reduce the timeframe over which data is sampled to avoid it all together.

Filtering: Many tools allow you to set filtering criteria to include or exclude specific traffic, perhaps blocking internal traffic, spam, bots, excluding time ranges or IP addresses, etc.  Make sure to use the same filtering logic across all tools, or at least account for the differences.

Time Zones and Time Windows: Some analytics tools use the the user's location while others may default to UTC or some other time zone.  This affects the day boundary for reports.  Also, the start time of an experiment may not coincide neatly with the output from another tool.  Make sure you are looking at the same window of time when comparing data.

Attribution and exclusion

Because different attribution logic is used by various tools it's not uncommon for values of a similar metric to vary by 10-15%.  It's important to understand how things like omni-channel conversions are handled.  For example, a user may get an impression/treatment on one device, perhaps an ad on a phone, and then convert (or perform some other tracked action) from a browser.

When experimenting, Split will exclude users from results under certain circumstances.  For example, a user that switches targeting rules more than once within the same version of an experiment.  This is not a usual circumstance and the number of users excluded for these reasons may be small, but there are cases where the design of a test could cause many users to be excluded because of changing targeting rules more than once.

Details describing how Split's attribution logic and how it's handled when users change treatments and/or rules within a version can be found in the Attribution and exclusion article in the Help Center.

In addition, Split's Metric's Details and Trends feature quickly highlights any excluded users.

Browsers and mobile devices

Implementation on the browser or mobile device can impact the collection of data.  This is exacerbated by the relative lack of control over user interaction. Abruptly closing a browser window or a mobile app can impede data from being captured.

Also, content blockers are becoming more common as users seek to avoid ads and more attention is placed on privacy concerns.  These blockers can impact a wide range of client-side trackers, not just ads.  Depending on what's blocked, they could cause the results computed by various analytic tools to differ.

Server side feature flags: One way to mitigate the issues posed by the lack of control on the client side, and create greater consistency in the user experience, is to move feature flags to a back end service for evaluation.

• Moving feature flags to the backend

Be aware that moving feature flags to the back-end may exacerbate the difference in counts if content blockers come into play since client-side content blocking doesn't impact server-side flags. 

When using the JavaScript or mobile SDKs, configuration options (such as these for JavaScript) can be tuned to ensure you'll capture the greatest possible number of impressions and/or events.  In particular, the RefreshRates can have a significant impact when lowered.

A number of articles in the Help Center describe why you may be missing or getting improper impression counts in Split, and how to avoid some of these issues:

• Block traffic until the SDK is ready
• Why are Impressions not showing in Split?
• Javascript SDK on slow networks
• Always getting Control treatments

Split has robust data pipelines and attribution logic. If you do find a mismatch in numbers that is greater than the expected variance between tools, we're happy to work with you to troubleshoot the discrepancy. Open a support ticket from our site or send an email to help@split.io