Experiments

Experiments measure the success of your website, application, backend performance, etc. Experiment metric results tell you if your new features are improving, degrading, or having no effect on your application users’ experience.

Experiments take in your experimental control variables, measure events, and display the results. Your experiment will show you if the data is in correct ratios (a passing health check), the running time has completed, and the results are conclusive, equipping you with data to drive your product decisions.

Create an experiment

Interactive guide

This interactive guide will walk you through setting up an experiment for the first time.

Step by step guide

Setting up an experiment follows these steps:

1. Navigate to the Experiments section on your navigation panel and click +Create experiment.

2. Give your experiment a name and designate an assignment source by selecting a feature flag and environment:

   • Choose a feature flag that has targeting active (not killed).
   • Choose an environment for which the feature flag definition is initiated (valid environments are enabled in the dropdown).

3. Define the scope of your experiment by setting a start and end time, a baseline treatment, comparison treatments, and a targeting rule.

   • Choose a start date on or after the date the feature flag was created.
   • The targeting rule can be any rule with percentage distribution (other rules are disabled in the dropdown). The default rule listed in the Targeting rule dropdown is the last rule in the Targeting rules section of a feature flag definition.

   Note: Based on your feature flag definition, the following fields are pre-populated by default: the start time is the timestamp of the flag’s current version, the end time is determined by your default review period, the baseline treatment is the flag’s default treatment, and the comparison treatments are all other treatments defined by the flag.

4. Write an optional hypothesis and add any additional owners. Then click Create.

5. Add key and supporting metrics to your experiment. Guardrail metrics will be measured automatically for every experiment.

Analyzing your metric results

After launching an experiment, you can view the results directly in the Experiments Dashboard. The metric results table summarizes key information for each treatment, providing a clear understanding of how the experiment impacts your key, guardrail, and supporting metrics.

Above the metric table, the Exposures bar visualizes the number of users exposed to each treatment group. This helps confirm that traffic was evenly distributed across variants, which is important for maintaining the validity of your experiment results. Monitoring exposures ensures that your experiment collected enough data and that assignment was properly balanced across variants.

Each row displays the metric name, treatment group, observed direction of impact (such as Desired, Undesired, or Inconclusive), relative impact percentage with confidence intervals, p-value, and raw metric value.

The Direction field provides a quick interpretation of whether the results for a treatment align with your defined goals. The Impact column shows the relative difference between the treatment and the baseline, along with its confidence interval. The P-value helps you understand the statistical significance of the observed impact; a low p-value (commonly < 0.05) suggests that the observed difference is unlikely due to random chance. The Value column displays the actual observed metric value for the treatment group, offering context for the size of the effect.

Experiment metric details

From any experiment page, you can click on a metric name to see metric details laid out in a dashboard format, showing charts with sample size information and metric impact calculations.

This section explains the information you see on the charts and guides you to read, interpret, and gain benefits from the information shown.

Sample size charts

Cumulative sample size over time

The Cumulative sample size over time chart makes it easy to compare treatment sample growth. You can quickly tell if both treatments are growing at a similar rate or diverging.

If one line consistently lags, you may have a traffic allocation issue, even if it’s subtle.

Any flat sections in a cumulative chart stand out immediately, indicating a pause in treatment assignment, traffic dip, or potential bug.

A smooth, steadily rising cumulative trend is a great visual cue that data collection is working correctly.

Sample population details

The Sample population details chart allows you to determine how close you are to reaching the required sample size for statistical significance.

Metric values charts

Values over time

The Values over time chart helps you see if a variant is consistently outperforming the others, or if there are fluctuations in performance that might need deeper investigation.

The shaded area around each line is the confidence interval (CI). Seeing the CI bands over time tells you how confident you can be in the metric at any point during the test. If the confidence intervals for two treatments don’t overlap, that’s a strong visual cue that the difference might be statistically significant.

If one variant is truly better, you’d expect to see a steady and widening gap between cumulative lines.

If the lines cross over or stay very close together, this suggests that the effect of the variant treatments may not be consistent or significant.

The cumulative values over time lines makes a clear visual case for which treatment performed better overall—perfect for post-test recaps and presentations.

Metric dispersion

The Metric dispersion chart provides full details of statistical results for data analysis. This chart summarizes the metric data for all treatments over the course of the experiment.

Metric impact charts

Impact over time

Seeing the impact over time visualization gives a clear sense of how fast the gains (or losses) are accumulating, and whether the treatment effect is growing, shrinking, or plateauing.

You can visualize the relative % impact of the treatments compared with the experiment's base treatment.

The shaded area is the confidence interval (CI) and gives you a range of values that you can be fairly confident contains the true value of your metric. Early in the experiment when sample size is small, the bands will be wide—reflecting uncertainty. As more data accumulates, the bands narrow, showing increased confidence.

Impact details

The Impact details chart provides metric impact results for data analysis. This chart summarizes the metric results for all treatments (compared with the baseline treatment chosen in the experiment settings) over the course of the experiment.

Current impact snapshot by treatment

The Current impact snapshot chart is a visual representation of the Impact details chart. This chart summarizes the metric results, showing a bar for each treatment (compared with the baseline treatment) using a green, gray, or red bar for a desired, inconclusive, and undesired impact respectively.

Alerting

When alerting is enabled for an experiment, and a statistically significant impact is detected on one of the experiment's key metrics, then an email is sent in the following format:

Subject: "Good News!/Alert: (Experiment name) had a positive/negative impact on (Metric name)"

Body: "Good News!/Alert: (Experiment name) had a positive/negative impact on (Metric name)

Harness Feature Management & Experimentation has detected a(n) DESIRED/UNDESIRED improvement/degradation of X% on your (Metric Name) metric. For more detail, visit the experiment's metric details.

Experiment: (Name and link to experiment)
Metric: (Name and link to metric)
Environment:
Baseline treatment:
Baseline metric value:
Comparison treatment:
Comparison metric value:
Relative impact: X% in the DESIRED/UNDESIRED direction

You are receiving this email because you are an owner of the experiment (Experiment name). Visit Harness Feature Management & Experimentation to modify the experiment’s alert settings."

The recipients of the alert email are listed in Experiment settings, accessed via the Settings button at the top left of the experiment page.