Choosing the right metrics for continuous improvement

Categories Agile
Agile metrics

I have come to a realisation lately that I’ve left a useful tool lying at the bottom of my agile toolbox without using it as much as I should. This tool is metrics.

Sure, I’ve been measuring velocity and have been using it to forecast delivery. From time to time, I’ve also dug deep into Excel exports from Jira to support some of my observations (“On average, we need two sprints to complete every story”).

What I haven’t been doing though is picking a good, basic set of metrics and tracking them over time as part of our continuous improvement.

Therefore, writing this blog post serves as much as a way to gather my own thoughts about metrics as sharing the result on the blog.

What’s the point of metrics?

“Scrum is founded on empirical process control theory” says the Scrum Guide. All this grand statement really means is that rather than just purely guessing, we base our decisions on observations. We experiment and we learn.

To help with these observations and to see how we’re improving over time (or not!), we measure things. Metrics are useful for several different purposes:

  • Forecasting – Based on progress so far, what are we likely to finish by when? Frequently updating our forecasts and keeping them realistic allows us to adjust our plans by removing, adding or reordering backlog items as needed.
  • Product performance – How well is our product doing? Are the features we’re rolling out making the impact we intended them to? If not, let’s experiment and see how we can improve!
  • Continuous improvement – Where do we see opportunities to improve our way of working? And once we’ve made changes, are we seeing the improvement we were hoping for?

In this article, I will be primarily looking at the third point.

How do we make good use of metrics as part of our continuous improvement?

1. Pick metrics that matter and are within the team’s control

Let’s get a couple of obvious but important points out of the way first.

Firstly, for a metric to be useful as a tool for continuous improvement, it needs to be within the team’s control to affect the metric. There is no point tracking something where nothing we do will make any difference. The exception, of cause, is if having this data can help us convince someone outside of the team why something needs to change.

Secondly, there needs to be a direction we want to move the metric in, either up or down (or maybe we want it to stay the same, without moving). Making the metric move the right way must have a positive impact. Otherwise, what’s the use of the metric?

For example, “Number of lines of code added in sprint” is rarely a useful metric. There is no good or bad direction for this metric to move. A high value could either indicate that we have done a lot of work or that we’ve written bad code with a lot of duplication. As long as the application does what it needs to do, the number of lines of code going down would be a good thing!

2. Don’t imposed the metrics on the team

Good metrics trigger discussions within the team. Together, the team members identify problems or opportunities and decide how to address them. For this to happen, the metrics need to be understood and accepted by the team.

Therefore, when identifying metrics, ask the team. Explain what the purpose of metrics is and let them decide what to measure. This will make sure the metrics are genuinely useful, rather than some box ticking exercise that keeps the Scrum Master busy.

3. Make the metrics visible

For the metrics to have the desired effect, they need to be visible to the team and easy to interpret. A simple line or bar chart on a whiteboard is likely to have a much bigger effect than a spreadsheet stored on a wiki or network drive somewhere.

Also, make sure to not just show absolute numbers. A line chart going up or down, comparing this sprint’s measurement to the previous sprints will give a much better understanding of whether we’re going in the right direction.

4. Don’t measure too many things

It is easy to get carried away and start tracking a lot of things. Particularly if we’ve got a shiny tool to do it for us or a Scrum Master who loves Microsoft Excel. However, if we’ve got 14 metrics moving in the right direction, we might pay less attention to the 15th. That’s a real shame if that’s the one that affects our bottom line. Therefore, settling for a small number of metrics makes clear what’s important.

Rather than trying to identify everything that we can measure, we should agree some basic metrics to make sure we’re doing alright in the areas that are important to us. Also, we need to make sure that these metrics are cheap and easy to collect. It needs to be possible to collect them at least once per sprint. Otherwise, the feedback loops will be too long and we won’t be able to do anything before it is too late.

5. … but have more than one metric

So, just a few metrics are better than a lot of them; does that mean using just one single metric is even better? Probably not.

Having additional metrics, such as adding one to measure quality (e.g. the number of bugs) can help us make sure we don’t make the wrong trade-offs without realising.

If looking at one metric in isolation, chances are we will be able to improve it quite easily but at what cost?

6. Add further metrics when you need them

Rather than trying to identify up front everything that may ever become useful, start with a small set of metrics. Then add more when needed.

For example, let’s say that we’ve concluded that our infrequent releases are preventing us from delivering value to the users as quickly as we could. Let’s track the number of days between releases (or even releases per day!).

On the other hand, if we’re already releasing frequently – or have bigger problems to address – let’s not bother with this metric.

7. Don’t encourage people go game the metrics

If our metrics start being used in ways they weren’t intended, the they will lose their reliability.

One such unintended way is if they start being used to measure performance, e.g. through comparing teams to each other. Another is when management request the team to improveme the metrics, for example by linking a bonus to reaching high velocity. Or, indeed, explaining in no uncertain terms what might happen if they don’t!

The quickest and easiest way increase velocity is to increase the estimates. If people get a bonus for making bigger estimates, how can we possibly trust our velocity from then on?

8. Measure to improve the system, not the performance of individuals

Even worse than team performance metrics are individual metrics and goals. For instance, let’s say we create a leader board for who on the team completes the most backlog items. Maybe this information is even fed into their performance reviews.

The problem with this approach is that it moves the focus away from the team doing their work together as effectively as possible. Each person will feel the need to optimise their work. We get sub-optimisation at the cost of the team’s total performance. Why spend time solving the complicated but important problems that will deliver the most value? It’s much quicker and easier to just do the simpler items in the sprint.

Some possible metrics to choose from

With all the above in mind, the following list may be a starting point to pick a few metrics from.

Area Why measure? Some possible metrics
Productivity Are the changes we’re making to our ways of working having a positive effect?

(and, obviously, forecasting)

Release burn up / burn down

Velocity (or the number of items completed per sprint if using #noestimates)

Started vs finished items per day

Cycle time in days

Cumulative flow diagram

Sustainability Can we keep working as we are for a long period of time? Team happiness, such as a Spotify style team health check or something more lightweight
Quality Are we introducing a lot of bugs? Might we be taking shortcuts? Total number of open bugs

Number of bugs created / resolved during sprint

Value

 

Velocity is one thing but do we spend our efforts on the right things, making a difference? Value delivered (measure impact or ask PO / customer)

Focus factor (how big proportion of the points we complete are part of our big goal and how much is maintenance etc?

Responsiveness Being agile means we can act quickly. Lead time (how long does it take us from when we have an idea (create a user story) until it is completed?
Predictability If our throughput varies a lot, forecasting will be hard, as will identifying trends to judge the effectiveness of our continuous improvement. Unplanned work (items brought in during the sprint)

Velocity variation

 

A useful source while writing this article was the presentation Data Driven Coaching by Troy Magennis – well worth a look!


What metrics do you use and what impact have they had on how you do things? Share your experience in the comments below!

 

Magnus Dahlgren – Scrum Master (CSP) and Agile Coach


1 thought on “Choosing the right metrics for continuous improvement

Leave a Reply

Your email address will not be published. Required fields are marked *