I have spent a lot of time researching and thinking about this one over my career. And at times, I have sat on both sides of the fence. However, I have come to the belief that hours is the best way to estimate.
Of recent, some alarm bells are ringing about using story points. Ron Jeffries, co-founder of Extreme Programming (XP), who was involved with the team that invented story points has apologised for inventing them. It seems there are a lot of unintended consequences as a result of the introduction of story points. I spoke with Ron about this as I wanted to hear the genesis story from him. Why did they start using story points originally? We will dive into that later in this article.
One of the most common questions a software team gets asked is; how long will this take? The answer must have a unit of time. So, if you are using story points you will need to convert this somehow. Mike Cohn (Agile legend) writes about a technique on how do story points relate to hours. Or more broadly, I have seen some teams use their velocity (how many story points they can finish in a sprint) and since a sprint is a known length (usually 2 weeks), they can estimate the length of their prioritised backlog. So, it is possible to convert from story points into hours.
Whilst this helps answer the question about how long a story point will take, it does raise another question. Why swap to story points if inevitably you need to switch back to hours? In our interview with Jeffries, he said they first switched to ideal days which was a way to say how long something would take without interruptions. They then went one step further and started using points to further mask their estimations from management. I believe they were simply finding it hard to give reasonable estimates, which I empathise with. You can hear the story from Ron in the following video.
Now that we know about the genesis of story points, we can continue investigating further and look at the benefits that are uniquely story points. This is an important point and something to dive deeper into throughout this article. To put it as a question, can the advantages of story points also be replicated in a process that uses hours instead? Then we would not need to do the conversion thing between story points and hours and avoid a lot of confusion.
Where did story points come from?
As discussed in the opening paragraphs, story points originated at the Chrysler C3 project with a team whose members went on to found Extreme Programming (XP). You can watch this video hearing from Ron himself on this topic.
I spoke with Magne Jørgensen on this subject as well. He also has been on the path to find out the genesis story and he was kind enough to send me the following slide with his summary.
What does Scrum say about story points?
To continue the investigation, let’s go straight to another of the authorities and see what they are saying. The Scrum Guide tells us that the development team is responsible for the estimates but does not give much detail beyond that. It is left up to the development team to form their own technique. A quote from the guide is “… enough work is planned during Sprint Planning for the Development Team to forecast what it believes it can do in the upcoming Sprint.”
However, I was able to find an article written by Jeff Sutherland (Scrum co-founder) on Story Points: Why are they better than hours? Interestingly, Jeff makes a very powerful opening paragraph on the use of story points and links to a Microsoft paper on how Scrum is improving their processes Scrum + Engineering Practices: Experiences of Three Microsoft Teams. That is great to see Scrum having an impact. But if you open the Microsoft article and jump to the 2) Planning Poker section on page 3, it states “The teams used Planning Poker to estimate the person hours required to complete functionality within an iteration.” Person hours, not story points. Hmm, did Jeff read the article? Probably, but was too busy to spend more time addressing the issue.
Further in his article, Jeff then goes on to loosely claim the history of failed waterfall projects (that traditionally used hours for estimations) as more evidence that hours do not work and story points are better. I do believe that Scrum has had a major impact on improving projects, but I believe that it might be a long bow to draw that it is the use of story points that is a main contributor. From a scientific point of view, there are too many variables in the claim and we would need to do better experiments to draw any conclusions at this stage. Correlation does not imply causation.
Are people crossing the streams between Agile and story points?
Why do people like using story points?
To find out why people like using story points, we did some research to look at what the wider community is saying. By applying a bit of science, the intention is to gather some evidence to help make an informed decision.
To gather the evidence, the top 10 results for some search terms on Google, Bing, Yahoo, and DuckDuckGo were examined. The search terms included were:
- Why use story points
- Why use story points vs hours
- What are the advantages of using story points
- What are the disadvantages of using story points
The complete list of unique articles from these searches can be found at the end of this article. What was found, is a few common themes or messages appearing. In no particular order, here is the top advantages of using story points and top disadvantages of using story points:
Top 9 advantages of using story points (as reported in the linked blog articles)
- Accuracy
- Your hour is not the same as my hour
- Uses fibonacci-like formula
- Quicker estimations
- Velocity is a helpful metric for measuring team performance
- Flexibility by preventing need for frequent re-estimation
- It aligns with agile and/or scrum methodologies
- Team building
- Accounts for non-project related work
Top 7 disadvantages of using story points (as reported in the linked blog articles)
- Can’t be used to predict dates until velocity is known/previous data is required
- Converting between story points and time
- Confusing
- Not appropriate in all circumstances
- False inflation of story points by the team to improve velocity
- Gets incorrectly used to compare velocity between teams
- Teams change
It would make a few academics shudder at the thought of using blogs and search engine results as evidence, so I dug a little deeper and looked at some academic publications. Magne Jørgensen in the Journal of Systems and Software published A review of studies on expert estimation of software development effort. Evita Coelho and Anirban Basu in the International Journal of Applied Information Systems published Effort Estimation in Agile Software Development using Story Points. And a bunch of others. But I could not find any publications on comparing story points to hours from a reputable academic journal or the like. If you know of one, please email me.
Are story points better than hours?
Now that we have gathered some research, it is time to do some analysis.
Firstly, lets seperate out two concepts used in story points estimations that sometimes get tangled together:
- Story points use relative estimations
- Story points use a fibonacci-like sequences
It is possible to do relative estimations without using fibonacci-like sequences and vice versa. This is important to note for our analysis.
Relative Estimations
Relative estimations are used to compare something with a reference. It is believed by advocates of story points that using points, and not hours, can lead to better judgements and therefore better accuracy.
Being a scientist, I have looked for evidence on this and I have not yet been able to find anything conclusive. Actually, the best and most independent evidence I could find says the opposite. Jørgensen in his paper on Relative Estimation of Software Development Effort: It Matters with What and How You Compare has 6 suggestions for estimation practices. His first is to make comparisons to similar projects and use work hours. He found story points led to less accuracy.
Fibonacci-like sequences
Story points based approaches usually use a Fibonacci-like sequence such as 1, 2, 3, 5, 8, 13 … etc. It is perceived this can help with quicker estimations as there are discrete choices to be made.
It is possible to use a Fibonacci-like sequence using hours such as 1h, 2h, 4h, 1day, 2day, 4days, 8days … etc. So, if this is a format you like, you can still use it without story points.
A quick note, watch out for the central tendency of judgement where the result can get skewed towards optimistic.
Your hour is not the same as my hour
A common argument for using story points usually goes like this…
Suppose that an octopus and I each estimate the effort to wash my car. The octopus has four times as many arms as I do. So, presumably, the octopus can wash my car four times faster than I can. But the octopus will also be four times faster than me at washing any car. So, the octopus and I can agree that perhaps my two-seat car can be estimated at 5 story points and a second, larger car can be estimated as 10 story points. So, then, the octopus and I can agree on a number of story points, even though the octopus is presumably much faster than I am at washing cars.
You may need to read that paragraph again. It is confusing. It can get even more confusing if you introduce a giraffe that is awesome at large cars but not very good at bending down and working on small cars. The argument starts to lose its weight and when teaching people I believe in the keep it simple philosophy.
So, how do we take into account that some people work faster than others? The answer is to estimate based on what a qualified (or reasonable) person would be able to do. For our example, estimate on how long it would take a human to do the task, not an octopus. If you do end up with an octopus in the team, then great! That is your advantage and this should give you some extra time to do an awesome job.
I do understand that this argument is in the grey area. A counter argument I hear sometimes is that it is not as accurate as we are not taking into account different skill levels. I disagree with this as we are already doing an estimation (using a crystal ball to see the future) and when you try to be too accurate when estimating, you run out of wriggle room. Everything becomes too tight, especially around deadlines. So, by buying a little extra time in favour of the people trying to deliver the project, I am comfortable with that. I trust that the people doing the job will do the right thing. Sometimes they even finish early! How cool would that be.
Do the advantages and disadvantages of story points apply to hours?
Let’s return to the advantages and disadvantages of using story points and see if hours can have the same advantages, or avoid the disadvantages. Think of this like doing a pros and cons list for each approach. I have filled this out below but I encourage you to apply your knowledge and do the same.
Top 9 advantages of using story points compared to hours
| Advantage | Story Points | Hours |
|---|---|---|
| 1. Accuracy | People might be estimating better due to using Agile and not necessarily story points. Correlation does not imply causation. | Evidence suggests hours based relative estimations are better. |
| 2. Your hour is not the same as my hour | Can create more accuracy for individual team members. | Can estimate on a qualified member giving more wriggle room for delivery. |
| 3. Uses fibonacci-like formula | Uses points like 1, 2, 3, 5, 8, 13… | Can use time like 1h, 2h, 4h, 1day, 2day, 4days, 8days… |
| 4. Quicker estimations | Enabled by relative estimations + fibonacci-like formula | Can be enabled by relative estimations + fibonacci-like formula |
| 5. Velocity is a helpful metric for measuring team performance | Story points per iteration. | Other metrics are available if required. |
| 6. Flexibility by preventing need for frequent re-estimation | Doubtful as the team can change. Octopus out. Giraffe in. | If hours are used on a qualified person, the team can change. |
| 7. It aligns with agile and/or scrum methodologies | Agile does not mandate story points. | Hours can also be used well with Agile. |
| 8. Team building | Planning poker is fun! | Similar fun games can be done using hours. |
| 9. Accounts for non-project related work | This is the genesis of using story points. | A time allocation factor can do that same. |
Top 7 disadvantages of using story points compared to hours
| Disadvantage | Story Points | Hours |
|---|---|---|
| 1. Can’t be used to predict dates until velocity is known/previous data is required | Not a big deal as an iteration is short. | Avoided. |
| 2. Converting between story points and time | Admin overhead and is implicitly done anyway. | Avoided. |
| 3. Confusing | Training time and disruption. | Hours are familiar. |
| 4. Not appropriate in all circumstances | Would need other appropriate estimations. | Can be used in all circumstances. |
| 5. False inflation of story points by the team to improve velocity | Gaming the system. | Can compare estimation with implementation. |
| 6. Gets incorrectly used to compare velocity between teams | A side effect of management interference. | Avoided. |
| 7. Teams change | Needs re-estimating based on skills. | Same unless estimation is done with a qualified person in mind. |
Summary
If I met a software team that was using story points successfully and were comfortable with their processes, I would not recommend switching. Story points can work well once you have started and you can measure velocity. Be wary though that comparing velocity between teams is a dark path. If I came across a team that needed to improve their estimation processes, would I recommend story points? No, I would not.
The way I see it, is that there are no arguments that makes story points significantly better than hours. Simply, most of the advantages advocated for story points can also be constructed into an estimation process that uses hours. There is no scientific evidence I could find that definitively shows the use of story points is better. On the contrary, some evidence may indicate hours are actually better.
So, why do people use story points? We found out from Agile legend Ron Jeffries the genesis story behind story points. They first moved to ideal days to take into account actual work time. And then moved further again to points to mask this so they could create more wriggle room for their estimations and to manage the managers.
From there, a few people started doing it and after some time it caught on as the next thing. It does sound pretty cool to do some planning poker with relative estimations using story points. Then we get some velocity happening!
Maybe we could do some planning poker with relative estimations using hours instead!? And instead of having ideal days we can introduce other allocation factors that acknowledge the realities of building software.
Finally, Scrum (and Agile more broadly) has done some amazing improvements for software development, but using these improvements as evidence story points are better than hours needs more research. The phrase correlation does not imply causation refers to the inability to legitimately deduce a cause-and-effect relationship between two variables solely on the basis of an observed association or correlation between them. The best way for establishing cause and effect is a double-blind controlled trial (or the AB test equivalent).
If you are reading this article as part of the Codebots academy, carry onto the next lesson. If you have made it this far and read all this, well done! You can get qualified on an hours-based estimation process in the Codebots academy.
And thanks Mikaela for helping with the research :)
Research for the advantages and disadvantages of using story points across the community
The following table is the results of the research described in Why do people like using story points? section above.
| Author | Article | Advantages | Disadvantages |
|---|---|---|---|
| Derek Davidson (Scrum.org) | Why do we use Story Points for Estimating? | - Team estimating easier. - Your hour is not the same as my hour. - Accuracy. | - Can’t be used to predict dates until velocity is known. - Confusing. - Translation from story points to hours. |
| Dan Radigan (Atlassian) | Story points and estimation | - Fibonacci-like format. - Accounts for non-project related work. - Removes emotional attachment. - You can assign points quickly, without much debate. - Rewards team members for solving problems based on difficulty, not time spent. | - None. |
| Anastasiya V. & Viktoria K. (RubyGarage) | Story Points vs Hours: 3 Reasons to Estimate with Story Points | - Accuracy. - No correlation with skills and experience of the estimator. - Velocity is tracked. - Flexibility by preventing need for frequent re-estimation. | - None. |
| Avienaash Shiralige (AgileBuddha) | Agile Estimation: 9 Reasons Why You Should Use Story Points | - Accuracy. - Tracks velocity. - Flexibility by preventing need for frequent re-estimation. - Stresses teams less. - Focuses client conversations on the right questions. - Team building. - Credible evidence. - Faster estimations. | - Need to convert back. |
| Mike Cohn (Mountain Goat Software) | The Main Benefit of Story Points | - Your hour is not the same as my hour. | - None. |
| Jeff Sutherland (Scrum Inc) | Story Points: Why are they better than hours? | - Accuracy. - Reduce planning time. - More accurately predict release dates. - Help teams improve performance. - Agile projects have more success. - Velocity. - Hours completed cannot convey completed features. - Faster. - Better. - Cheaper. | - None. |
| Willem-Jan Ageling (Medium - Serious Scrum) | Scrum — Management tells us to use story points | - Standardisation. - Measuring team performance. | - Teams change, which impacts velocity calculations. - You have to assume the team always does similar work. - Teams may be encouraged to finish an experiment rather than recognising something is a dead end. - Not the best solution for all teams. |
| Maarten Dalmijn (Medium - Serious Scrum) | 12 common mistakes made when using Story Points | - Quick estimations. - Team estimating easier. - Fibonacci-like. - Easier estimations. | - Translating back to hours. |
| Sidu Ponnappa (Gojek Engineering) | The Point of Story Points | - Accuracy. - Fibonacci-like sequence. - Velocity. | - Requires previous data for comparison. - Can’t be given a time equivalent. |
| Nicholas Muldoon (Velocity Counts) | Why do high performing Scrum teams use story point estimation? | - Fibonacci-like sequence. - Quicker estiamtions. | - None.. |
| Marcin Geb (Software Plant) | Story Points in Jira Explained | - Accuracy. - Fibonacci-like sequence. - Your hour is not the same as my hour. - Velocity. | - Less efficient in lengthy, multi-year projects. - Converting between story points and time. |
| Mohit Tyagi (To The New) | How to Estimate Story Points in Agile? | - Helpful for pre-planning sprints. | - Not appropriate in all circumstances. |
| Joshua Kerievsky (Industrial Logic) | Stop Using Story Points | - Velocity. | - Decreases agility. - False inflation of story points to improve velocity. - Doesn’t actually increase accuracy because humans are always bad at estimating. - Confusing. |
| Dusan Kocurek (ScrumDesk) | Agile estimation explained: Effort vs. Time, Storypoints vs. Hours | - Agile methodology. | - None. |
| Jeff Sutherland (Openview) | Scrum Points: Why Story Points Are Better Than Hours | - Velocity. - Agile. - Accuracy. | - None. |
| Mike Cohn (Mountain Goat Software) | Don’t Equate Story Points to Hours | - Your hour is not the same as my hour. | - None. |
| Avienaash Shiralige (AgileBuddha) | Story Points and Man Hours – When To Use Them and Why? | - Quicker estimations. | - Requires previous data for comparison. |
| David Hawks (Agile Velocity) | 5 Reasons Why Estimating in Story Points is a Superior Method | - Quicker estimations. - Do not need to re-estimate. - Velocity. | - None. |
| Harry Ulrich (Agile Velocity) | Why Your Agile Team Should Use Relative Story Point Estimation | - Accuracy. - Quicker estimations. - Estimations get more accurate over time. | - None. |
| Anshika Misra (Apple Inc.) | Story points - Advantages and Disadvantages | - Follows Scrum methodologies. - Accounts for non-project related work. - Estimations get more accurate over time. - Open discussion builds team understanding of a story. - Easy to compare features to others. - Your hour is not the same as my hour. - Accuracy. | - Confusing. - False inflation of story points to improve velocity. - Gets incorrectly used to compare velocity between teams. - Pressure to maintain velocity reduces quality. |
| Jack Yang (Perficient) | Using story points or hours for estimation? | - Follows Scrum methodologies. - Accuracy. - Your hour is not the same as my hour. | - Confusing. - Converting between story points and time. |
| Willemo (The Weekly Byte) | Story Points vs. Hours – Pros and Cons | - Re-Estimations easier. - Quicker estimations. - Fibonacci-like sequence. - Team estimating easier. | - Imprecise by nature. - Teams change. |
| Yvette Francino (Tech Target) | Scrum project management: Estimating with story points | - Velocity. - Fibonacci-like sequence. - Accuracy. | - Confusing. - Less precise. - Gets incorrectly used to compare velocity between teams. |
| Unknown (Eylean) | Estimating in story points compared to hours | - Accuracy. - Fibonacci-like sequence. - Improved developer confidence. | - Requires previous data for comparison. - Gets incorrectly used to compare velocity between teams. - Converting between story points and time. |
| Unknown (Base 36) | Picking A Side in the Story Points vs. Ideal Days Debate | - Accuracy. - Team bonding. - Your hour is not the same as my hour. | - Confusing. - False inflation of story points to improve velocity |
Written by Eban Escott
Founder of Codebots
Dr Eban Escott received his Doctorate from UQ (2013) in Model-Driven Engineering and his Masters from QUT (2004) in Artificial Intelligence. He is an advocate of using models as first class artefacts in software engineering and creating not just technologies, but methodologies that enhance the quality of life for software engineers.
