Introduction
When getting started with running experiments, the design phase serves as the foundation upon which data-driven decisions are built. This pivotal phase will enable your organization to rigorously evaluate the impact of changes made, ensuring that decisions are grounded in empirical evidence rather than assumptions.
This playbook will be an invaluable resource, as it guides your organization through each critical step of the experiment design process. It assists in formulating well-defined hypotheses, prioritizing them based on strategic objectives, selecting appropriate metrics that align with business goals, and meticulously crafting the experimental setup. Supporting each of these areas, you will find ‘Split resources’ throughout this playbook, which are downloadable frameworks or guides to help you as a starting point. By providing comprehensive guidance in these key areas, this playbook empowers you to conduct effective experiments that yield actionable insights, ultimately driving informed decision-making and optimizing outcomes.
Experiment Process & Lifecycle
What is it?
An experiment lifecycle identifies the stages and processes involved in planning and executing an experiment. The end to end experiment lifecycle has six key steps: Brainstorm, Plan & Design, Develop, Launch, Analyze and Iterate.
Why is it important?
For any experimentation program to be successful, teams should be able to remain agile and create repeatable steps. These steps give teams an outline for the who, what, when and where of experimentation. From conception to iteration, it is important that teams optimize the process itself as they go along, in order to move faster.
Program leads and individual team owners should consider the following:
- Who is involved with the experiment?
- What are the objectives of the experiment?
- What learnings will you gather from this experiment?
- Is your experiment aligned with the wider company goals?
Check out Split Arcade for our Experimentation Level 1 certification, which reviews these steps in detail.
Experiment Workflow
Split Resource: Experiment Workflow
What is it?
An experimentation workflow provides a structured and systematic approach to designing, conducting, and analyzing experiments. There are a variety of productivity and workflow tools out there, but our customers typically use Jira, Airtable, Trello, ProductBoard or Excel.
Why is it important?
It helps teams follow a clear sequence of steps, reducing the likelihood of errors and ensuring that experiments are carried out in a consistent manner.
The purpose of an experimentation workflow is to facilitate data-driven decision-making, reduce bias, and improve efficiency by guiding teams through the experiment lifecycle.
Having a workflow encourages organizations to define clear objectives and promotes a culture of continuous learning and risk mitigation by allowing teams to iterate on experiments, document details for reproducibility, and allocate resources effectively. Ultimately, it enhances collaboration among cross-functional teams and ensures that changes and innovations are rigorously tested before implementation, leading to more informed and efficient processes and outcomes.
Experiment Execution
Split Resource: Experiment Build Steps
What is it?
Once you have your experiment hypothesis or features to measure, there are a few steps to complete in order to configure your experiment in the Split platform whilst building the code itself. This is your development and rollout plan. This could be a document or a series of Jira tickets which details the who, what, when and why for the experiment.
Why is it important?
In order to get your experiments live as quickly and accurately as possible, it’s important to have a clear process detailing the feature build and QA. Follow the steps outlined in our help center below.
Split Help Center:
- Create a Feature Flag
- Create a Metric
- Edit Treatments
- Add Targeting Rules
- Adjust Experiment Settings
Experiment Ideation
Split Resource: Customer Journey Map
What is it?
Behind every digital experience is a creative ideation process. Ideation is the stage in the ‘design thinking’ process where you concentrate on idea generation through sessions such as brainstorming and creative workshops. This process brings together the diverse perspectives from each of the teams responsible for optimizing the customer journey. These teams include, but are not limited to: marketing, data science, analytics, product, engineering, UX, and research.
Why is it important?
The source of all your experiment ideas should always be from data. Leverage a mix of qualitative and quantitative data to locate and understand user pain points in the customer journey.
Qualitative data is a type of data that deals with descriptions, characteristics, and properties that can be observed but not measured with numbers. Typical examples in the realm of experimentation include user studies, interviews, focus groups, remote user labs, customer research, previous experiment data, competitor analysis etc.
Quantitative data is a type of data that is expressed in numerical terms and can be measured and quantified. Typical examples in experimentation include analytics data, heatmaps, session recordings, data warehouse, customer journey data etc.
A customer journey map lays out all the touchpoints that your customers may have with your brand. This should include how customers first heard of your brand through social media or advertising, to their direct interactions with your product, website or support team. It should include all actions your customers take to complete an objective.
Customer journey maps help teams to visualize what the customer is experiencing in real time and may unveil common pain points or customer challenges that need to be addressed.
Experiment Goal Tree
Split Resource: Goal Tree
What is it?
A goal tree is a visual representation of the hierarchical structure of goals and objectives within an organization, project, or any complex system. A goal tree consists of four buckets:
- Company Goal: measures a high-level metric, “big hairy audacious goal”, or challenge
- Business Unit: focuses on the lever they can control to reach that goal
- Optimization Goal: breaks down the business unit goal into strategic themes
- Experiment Goals: directly correlate with the tactical changes being made
Why are they important?
Goal trees facilitate alignment across different functions and roles in your organization. They empower teams to make small, directional changes that visually roll up to the larger company goal.
Choosing Experiment Metrics
Split Resource: Creating Metrics
What is it?
Experiment metrics are the quantitative measure of success for your experiment hypothesis. They are the specific measurements or data points used to assess the performance of an experiment. These metrics help you determine whether the changes you've made in one variation (often referred to as the "treatment" or "B" version) have had a significant impact compared to the original version (the "control" or "A" version).
Why is it important?
The choice of experiment metrics is critical because they define what success looks like for your test. Your metrics are important because they will inform how your experiment has performed, and
whether you have a winning, losing or inconclusive outcome. Typically you will have primary, secondary, and guardrail metrics for each experiment. Read our Split Help Center links below for more information on creating a metric in Split.
Split Help Center
Hypothesis Building
Split Resource: Hypothesis Framework
What is it?
An experimental hypothesis is an educated guess or a prediction about the outcome of an experiment. This is usually formulated from data, information, and insights gathered and includes statements around opportunities and outcomes.
Why is it important?
Hypotheses are important in controlled experiments because they help frame the design and expected outcome. A well-crafted hypothesis is crucial because it provides a clear direction for the experiment and helps you determine whether the results support or refute the hypothesis.
Split Help Center
Experiment Prioritization
Split Resource: Prioritization Framework
What is it?
A prioritization framework is a set of criteria to help teams prioritize a large backlog of experiment ideas and hypotheses. It is a structured approach used by organizations to determine the order in which they should conduct experiments.
Why is it important?
The prioritization framework helps teams decide which experiments to prioritize based on various factors, such as potential impact, resource requirements and effort, and strategic alignment.
Experiment Design
Split Resource: Experimentation Design Seed
Split Resource: Feature Decision Tree
What is it?
Experiment design centers around turning your user problems and solutions into a data-driven hypothesis, and building creative solutions with clear metrics to measure success.
Why is it important?
Having an experiment design process is important to ensure you will always have insightful experiment outcomes and factor in all considerations to help you plan.
These might include:
- How long to run your experiment for
- Which stakeholders need to be responsible, accountable, supporting, consulted & informed when it comes to experiment results and other stages
- Action plan for all outcomes (what to do if it wins, loses or remains inconclusive)
- All detail for engineers who build the experiment(s)
- Targeting, segments and metric details (knowing which events and properties need to be sent to Split)
Split Arcade
- Split Arcade Certifications Portal
- Level 1: Experiment Foundations | ~ 1.5 hours to complete
What is it?
The Split Arcade is a self-serve, interactive learning platform to help onboard and level up your team at scale. Our arcade provides persona-based, step-leveled learning certifications in key categories like Feature flagging, Experimentation, and Administration.
Why is it important?
Our Experimentation learning path ensures a holistic understanding of experimentation fundamentals from planning, design, and best practices, and covers Split specifics as well.
What is involved?
Roadmap to Experimentation
- Understand experimentation: what it is, and which experiments to run based on your needs
- Understand the value of experimenting in Split
- Understand the lifecycle of an experiment
Experiment Design & Planning
- Know how to connect organizational goals with metrics to create impactful experiments
- Understand how to work cross-functionally to brainstorm experiments rooted in data
- Know how to form a hypothesis and approach experiment design
- Understand how to prioritize problems for high-impact experiments
Creating Metrics
- Know how to select, add, and your organize metrics across experiments
- Understand Split specific features that help level up your metrics
- Know how to create metrics and alerts within Split using best practices
Engineering: Build & QA
- Understand what an A/A test is, and how to run it
- Understand how to navigate the results of an A/A test
- Understand Split experiment settings and when you would change them
- Know how to set up, QA, and launch an A/B test
Who should complete it?
The certification has been designed for product and business-focused learners, though it is beneficial for anyone going through experiment ideation and execution while trying to align with business goals.
We recommend completing this as a starting point to experimentation and using the Split platform.
Comments
0 comments
Please sign in to leave a comment.