It’s probably no surprise, given that one of our core company values is Scientific but not heartless, that we are big fans of experiments when it comes to building software. We have also been influenced by the Lean Startup by Eric Ries, which is about running a lot of small experiments with a focus on metrics.
There are many different experimental frameworks and cool tools that are readily available. It is not our intention to reinvent the wheel and we encourage you to explore and choose a system that works for your organisation.
The steps in this article will guide you in planning, performing, measuring and acting on an experiment. The important point is not which framework you choose, but that you embrace an experimental mindset and encourage an investigative culture within your team to help optimise business processes and projects.
Before you Start
One of the best times to identify problems and have ideas is when you are talking to team members about the challenges that they are facing. Collecting both qualitative and quantitative data that supports the problem you want to solve is useful when pitching an experiment to other stakeholders.
Step 1: Understand the problem
We call this phase of the process, discovery. It’s time to observe, ask lots of questions and dig into the historical data that you already have like burn-down charts or platform analytics. Interview the people closest to the problem, quiz them on their current processes and carefully map out their user journey so that you’re ready for the next step.
Step 2: Develop a hypothesis
Once you have analysed the problem, you must identify a solution. This is the really fun part as it involves brainstorming ideas for your potential hypothesis statement. A hypothesis is a prediction that you create prior to running an experiment. A good hypothesis makes it clear what it is you are validating or invalidating.
A common format is: If [cause], then [effect], because [rationale]. Remember that there are no bad ideas. Focus on nurturing a collaborative environment and looking at the problem from all angles based on the knowledge that you collected in the first step.
Step 3: Plan the experiment
Now that the hypothesis is ready to test, you need to identify the most appropriate solution for the problem and create a strategy to test it. We call this the Implementation Plan and it requires that you decide: who is in charge; who is involved; the duration; identify the metrics to track/measure the experiment; and how the experiment will be implemented.
Step 4: Collect the data
The main difference between a well-formulated hypothesis and a guess is, data. In the previous step you defined how to measure the experiment. It could be measured in money, time, community satisfaction, or another critical metric. Whatever you choose, now is the time to collect the data that defines a clear criteria for success and failure. And remember, failure should not be feared; it is all part of the learning.
Step 5: Make a decision
At the end of the experiment you will analyse and interpret the data to determine if the experiment was successful. You can use the results to create new hypotheses or pivot the experiment to explore new solutions. Just make sure the whole process is documented and available for others in the company to view.
I have included a simple framework as a good starting point for your experiments.
Ultimately, by adopting the science of experimentation, listening to your community of users and acting on what you learn, you’ll build truly loveable software.
Written by Christine Chien
Marketing Operations and Partnerships
Our very own Christine is a marketer by day, nerd by night. If she isn’t developing our marketing strategy, she is usually found by her 3D printer or at a local plant shop.
