How (Not) to Measure Quality

25-minute Talk

Measuring quality is hard, defining what quality is is difficult, being aware of why you measure is fundamentally important. Learn how to choose and combine metrics to create something valuable.

Virtual Pass session


2:45 p.m. – 3:30 p.m. Tuesday 22nd


Room F1 - Track 1: Talks


People in IT


  • Identify different approaches on how (not) to measure quality.
  • Assess commonly used quality metrics against different purposes.
  • Be aware of possible side effects of measurements.
  • Understand how metrics can be combined to even out each other's weaknesses.
Measuring quality requires many questions to be answered. The most obvious ones may be: “What is quality?”, but also “How can we measure it?”, “Which metrics are most accurate?”, “Which are most practical?”. In my experience, one question is often not answered or postponed untill it is too late: “Why do we want to measure quality?” Is it because we want to control how well our developers are performing? Is it to detect problems early? Is it to measure the impact of changes? Is it the product or the process we care about? Is it to improve locally in a single team or globally across the company? Is there a specific problem that we are trying to solve, and if so, which one? Instead of trying to define what software quality is–which is hard and depends on a lot of factors–we should first focus on the impact of our measuring. Some metrics may work great for one team, but for the company as a whole. Some will help to reach your team or organizational goal, some will not help at all, and some will even have terrible side effects by setting unintended incentives. Some can be gamed, others might be harmful to motivation. Consider an overemphasis on lead time, which can lead to cutting corners. Or measuring the number of bugs found, which can cause a testers versus developers situation. In this talk, I share some general motivations for measuring quality. I review various commonly used metrics that claim to measure quality. Based on my experience, I rate them with regards to how they may be helpful or harmful to achieve actual goals and which side effects are to be expected. I give some examples how the weaknesses of one metric might be countered by another one to create a beneficial system.

Related Sessions

Virtual Pass session
4:00 p.m. – 4:45 p.m.
Room F3 - Track 3: Talks

25-minute Talk

Virtual Pass session
1:20 p.m. – 1:30 p.m.
Room F1+F2+F3 - Plenary

Active Session

Virtual Pass session
5:00 p.m. – 6:00 p.m.
Room F1+F2+F3 - Plenary

45-minute Keynote

9:00 a.m. – 5:00 p.m.

Full-Day Tutorial (6 hours)