There are many testing frameworks and cool tools which are readily available to help you make improvements to the way you work. It is not our intention to reinvent the wheel, and we encourage you to explore and choose a system which works for your organisation. The important point is not which framework you choose, but that you embrace an experimental mindset and encourage an investigative culture within your team to help optimise business processes and projects. The steps in this activity will guide you in planning, performing, measuring and acting on an experiment.
Before you start
One of the best times to identify problems and have ideas is when you are talking to team members about the challenges that they are facing. Collecting both qualitative and quantitative data that supports the problem you want to solve is useful when pitching an experiment to other stakeholders.
Details
Level of difficulty
Moderate
Stage
- Any stage
Suggested time
The activity of setting up an experiment depends on the complexity and number of people involved but often it will take a couple of hours to a couple of days, depending on the knowledge gathering required.
Participants
- Product owner
- Business analyst
- Squad lead
- Designer
- Web engineer
- Test engineer
- Account manager
- Stakeholders
- Users
Materials
- Whiteboard
- Markers
Steps
1. Understand the problem
We call this phase of the process ‘discovery’. It is a time to observe, ask lots of questions (e.g. the 5 Whys) and dig into the historical data that you already have, such as burn-down charts or platform analytics. Interview the people closest to the problem, quiz them on their current processes and carefully map out their user journey so that you’re ready for the next step.
2. Develop a hypothesis
Once you have analysed the problem, you must identify a solution by creating a hypothesis. A hypothesis is a prediction that you create prior to running an experiment. A good hypothesis makes it clear what it is you are validating or invalidating. A common format is:
If [cause], then [effect], because [rationale].
Remember that there are no bad ideas. Focus on nurturing a collaborative environment and looking at the problem from all angles, based on the knowledge that you collected in the first step.
3. Plan the experiment
Now that the hypothesis is ready to test, you need to identify the most appropriate solution for the problem and create a strategy to test it. We call this the Implementation Plan and it requires that you decide:
- who is in charge,
- who is involved,
- the duration,
- the metrics to track/measure the experiment, and
- how the experiment will be implemented.
4. Collect the data
The main difference between a well-formulated hypothesis and a guess is data. In the previous step you defined how to measure the experiment. It could be measured in money, time, community satisfaction, or another critical metric. Whatever you choose, this is the point where you collect the data that defines a clear criteria for success and failure. Remember, failure should not be feared; it’s all part of the learning. This is the point where you actually do the activities which you planned in step three, so get excited!
5. Make a decision
At the end of the experiment, you will need to analyse and interpret the data and determine if the experiment was successful. You could use the results to create new hypotheses or pivot the experiment to explore new solutions. Just make sure the whole process is documented and available in a way that everyone can view and understand.
Justification
Given that one of our core values is ‘scientific but not heartless’, it is no surprise that Codebots team are big fans of the scientific method, a problem-solving approach at the core of all sciences. We took this process and allowed it to be influenced by the Lean Startup, which is about running a lot of small experiments with a focus on metrics. Combined together, and with a few experiments of our own, this process was developed.
As stated, a core influence of the creation of this process was the Lean Startup and it’s build, measure, learn and feedback loop. A common misconception which we discovered with this process though, is that you should start with build. We always start with learn, as it encourages the team to think through and understand what they are trying to learn, before they start building. This way, the team is building to learn, not just building to build.
In the following image, we have included an example of a simple framework which we have found is a good starting point for experiments:
Written by Shannon England
Branding, Communications and Marketing
Shannon is our branding and marketing go-to. When she isn’t working on our communications and marketing campaigns, she is typically drinking coffee or at the local plant shop.
