How do I Add Users to an Experiment without Resetting Metrics?

Any change you make to a split will cause metrics to reset, for a very good reason: you don't want to bias your experiment or pollute your metrics by making a change that's inconsistent with the experiment to date.  If you are experimenting on users in the US and add users from South America after a couple of weeks the results may not be representative of a truly random population.  

That said, there are cases where you may want to add users to an experiment without resetting metrics.

Adding Users to a Treatment

In some cases, you may want to add specific users, such as QA team members or beta customers, to a treatment without impacting the experiment.  This is generally accomplished by directly assigning the keys/IDd using a rule separate from those splitting the population on which you are experimenting.  What happens if you want to give a new beta user access to a new feature during the test without resetting the metrics?  

The easiest way to do this is to create a rule separate from the rule(s) randomly splitting traffic by percentage.  Instead of adding individual user IDs to the rule, you can use a 'segment' and add users to the segment.  This does not directly change the split, so metrics will not be reset and, of course, the metrics will be calculated on a per rule basis. So, even if it's only a single user to start, you may want to use a segment so that you have the flexibility to add users in the future without resetting metrics.

Adding Users to a Test

Perhaps you want to only allow a portion of your users into an experiment, maybe to reduce the risk of a new feature by exposing it to only a smaller set of your overall population.  A simple way to do that is to use Traffic allocation.  You can set the treatments to 50/50 but only send a percentage of the visitors to the rules, with everyone else going to the Default treatment, as shown below.  By increasing the Traffic allocation you add more users to the test.

The Default treatment could be the same as 'off', or it could be a third treatment, and as you increase the Traffic allocation you increase the number of people participating in an experiment.

Using Dependencies

However, with this approach each time you increase Traffic allocation you are changing the split.  That means the metrics will reset.  

In order to add traffic without resetting the metrics, you can use Dependencies.  Dependent splits assign users based on rules in a parent.  For example, you might have a split called 'traffic_control' as the parent, which has a simple rule that sets 10% of users to on and 90% to off.

Then, in your test split, you could have a rule that splits the traffic using percentages, but only for the 10% of the users that qualify.  This has the same effect as Traffic allocation, but the percentage of users participating in the test is controlled by the parent split.  

In the above example, 10% of the users meet the criteria of being in the 'on' treatment in the parent split, and of those, 50% will get the new feature and 50% will not.  As you increase the percentage in the on treatment in the parent split, traffic_control, you will increase the number of users in your test without resetting the metrics.

Remember, there are very good reasons why you want to reset metrics if you change the profile of your test, so you should consider if doing the above will bias your results in any way.  But if the introduction of new traffic is purely random and from the same base population this may well be a technique worth investigating.