Statistical Comparison Not Possible

Comments

4 comments

  • Spencer Skeen

    For more clarification, we are serving our anonymous users a treatment based on a URL query parameter which we add as an attribute to the split "getTreatment" hook (via React SDK). The default anonymous users who do not have the specific URL query parameter are given the default treatment. When we attempt to compare the metrics, we receive the "Statistical Comparison Not Possible - Grouped By Anonymous" message. 

    0
    Comment actions Permalink
  • Dave Murphy

    Hi Spencer, in creating those metrics did you use an 'across' measure?  The message would be "Statistical Comparison Not Possible - Grouped across Anonymous".  When calculating metrics across users all the events for all the users are combined into one single value. These metrics describe the overall trend and behavior of the treatment and can be used for analytic and reporting purposes.

    Since the data is aggregated across the whole treatment, they don’t preserve enough information for statistical testing. It explained in more detail in this video.

    If you are not using an across measure then can you let us know your Org and we can take a look.  

    dave

    0
    Comment actions Permalink
  • Spencer Skeen

    I think that makes sense. The measure we are using is "percent of unique anonymous" for a given measured event. 

    0
    Comment actions Permalink
  • Spencer Skeen

    I also forgot to mention that we run these metrics on several other experiments which correctly show the statistical comparison. It seems to only happen on this one experiment in which the treatment is determined by an attribute sent to Split.

    0
    Comment actions Permalink

Please sign in to leave a comment.