Oh the numbers………

Unfolding the confounded numbers

We live in an analytical world, a world where Metrics are unavoidable from the stock market to sports to medicine, an integral component to help make decisions that change the world and most importantly lives. So why is it that when we talk about using Metrics on Agile teams, where we spend all of our workday and where the company spends their budget money, there is an unsaid negative connotation to it.

In my honest opinion, it’s the lack of understanding and level setting between the management and teams on the usage of these Metrics that devalues them in the organizations. The teams that understand the science behind these Metrics end up owning these and leveraging these to Continuously Improve themselves.

Let’s break it into what these Metrics are really used for and what they shouldn’t mean before we dive into which Metrics are/should be important to an Agile team.

What Metrics shouldn’t mean-

1. Apple to Oranges — Each team is different and goes through different stages as per the Tuckman model, so obviously their performance will vary. For example, each team will have different attributes like team members, management support, business maturity, etc. Needless to say, it wouldn’t be wise to use these numbers to compare them with each other.
2. Metric Shaming — Metrics aren’t a way to assess the teams. It is to be used as a gauge to identify the weak spots to get the help that the teams need instead of using these numbers to reprimand them. In the end, these are just numbers, and a smart team knows how to make these numbers look better to avoid issues. If our goal is to help teams, we don’t want them to mask the real issues.
3. Set the Standards High- We cannot and should not be forcing the teams to meet a standard when it comes to Metrics. Primarily, there shouldn’t be any fixed standard for Metrics, and as we established in #1 above, each team will be different. There are some Metrics that will be more predictable over time (e.g. the Velocity of the team) — but to dictate these standards to any team would be an anti-pattern and would be setting them on the path to failure.

What Metrics really should be used for-

1. Educate and Champion — Before you use the Metrics, make sure the teams understand the honest reasons behind the use of these numbers and how these numbers can help them go to the next level.
2. GPS- Use the Metrics as a way to navigate the teams, the areas where they might be succeeding, and their weak attributes. Once identified, use this information to devise the coaching plans for the teams.
3. It’s a journey- If you are an Agile team, you are continuously looking to improve as a team. These numbers can be helpful in that journey. The team could focus on one number at a time and with the coaches’ help, they could figure out ways to find underlying processes and issues and take steps to improve on them.
4. Crystal Ball- The Agile Metrics can be used by the product and planning teams as a way to predict and plan the deliverables. If the teams have reached the point where they are consistent, the product can use these numbers to organize the next big market release or product launches.

Let’s look at some ground rules to be used with the Metrics to make them more meaningful-

1. Each team is allowed to have different Metrics. Understand how Metrics work before rolling it to the teams.
2. Each team should be allowed to have unique values for a metric. There is no “one size fits all”. Sometimes the only competition a team is going to have is with their older selves.
3. Each metric should be used with the story and not used in isolation. e.g. if velocity decreases for a sprint it could be that it is a shorter sprint because of 1 corporate holiday or the majority of the team is taking a day off. Make sure you understand the underlying story when analyzing the Metrics.
4. Just because you are running Metrics, it does not mean you can replace the assessment surveys or retrospectives for the team. It also doesn’t override the team’s way of finding methods to improve processes.
5. No Metrics should require anyone on the team to have majored in stats/mathematics. They should be simple to understand and follow.
6. You are not married to the Metrics; if they are not helping the team in any way, part your ways sooner than later.
7. Not all Metrics apply to all frameworks, Educate yourself on the difference between Scrum Metrics, Kanban Metrics, and Lean Metrics.

Now lets review what Metrics we should be using for the teams. I will list a wide variety and let you pick and choose which ones are more beneficial to your team’s needs. You are always allowed to try before you buy-

1. Time to Value or Lead Time- This is the time-lapse from the time a request is made to the time it is delivered to the market. It is a very critical Metric to identify the steps where we could improve the leanness and make the processes better to have a positive impact on time to market. It also brings attention to queue sizes and WIP Limits which is a more important Metric for teams to understand and use for their benefit.
2. Deployment Frequency — Although this metric is not one of the shiniest coins on the block, it does give a good story on the DevOps side and the health of CD pipelines. If we combine it with defect frequency and the escaped defects Metric, we can use it to assess where the needle is tilted more on quantity vs quality.
3. Sprint Burndown- This is one of my favorites as it gives you a progress measure on the sprint. This type of burndown descent also helps you focus on some improvement areas. Most of the tools we use today for Scrum and backlog maintenance will have this as an inbuilt Metric and easy to access.
4. Velocity- It is the measurement of story points completed in each sprint. I prefer average velocity more than velocity. Most Agile teams are humans and when humans are involved, not every sprint will generate the same number. There will be good sprints and bad sprints, so an average will help you keep leveled. That’s the number you should be using for planning sprints.
5. Scope Change- This Metric helps you see how much traffic is being generated in and out of your active sprint. This includes stories added after the sprint planning and incomplete stories removed from the sprint. This helps you evaluate the reasons and see patterns if this happens a lot on an ongoing basis. Analyzing the reasons and using this as means for continuous improvement can be game-changing.
6. Committed vs Completed- It is important to understand the gap between the 2 numbers per sprint. The wider the gap is, the closer we need to look at the team and behind the scenes. Usually a 5–10% gap is acceptable and actually encouraged. Anything more than that should be investigated.
7. Epic and Release Burndown- Epic burndown and release burndown help you in many ways to understand the rate at which Epic/release is being completed, the same way as Sprint burndown. These metrics help you track the market releases and product launches.
8. Blocked Time — The time for which a feature/Story is blocked for and contributes to the holding cost. This is the time that’s getting added to Time to Value but also the time we are keeping customers away from that functionality and the company away from that revenue.
9. Escaped Defects- Is the number of defects/problems found in production once the functionality is delivered. It can directly affect customer satisfaction as well as expected ROI. This number also speaks volumes about the quality process.
10. NPS — Net promoter score (NPS) is a customer satisfaction indicator. This score signifies how likely it is for a customer to recommend you to friends, family, and others.
11. Happiness Metrics — This is one score that should matter to organizations/management the most. The higher this number is, the happier your teams are — and all the other Metrics will fall in line eventually. If you have burned out teams and unsatisfied employees, it’s not possible to get the best out of them.

To summarize, the Agile Metrics can be your best friends or your worst enemies. Be smart about which way you want to tip the needle!!