How (Not) to Measure Quality

Measuring quality requires many questions to be answered. The most obvious ones may be: “What is quality?”, but also “How can we measure it?”, “Which metrics are most accurate?”, “Which are most practical?”. In my experience, one question is often not answered or postponed untill it is too late: “Why do we want to measure quality?” Is it because we want to control how well our developers are performing? Is it to detect problems early? Is it to measure the impact of changes? Is it the product or the process we care about? Is it to improve locally in a single team or globally across the company? Is there a specific problem that we are trying to solve, and if so, which one? Instead of trying to define what software quality is–which is hard and depends on a lot of factors–we should first focus on the impact of our measuring. Some metrics may work great for one team, but for the company as a whole. Some will help to reach your team or organizational goal, some will not help at all, and some will even have terrible side effects by setting unintended incentives. Some can be gamed, others might be harmful to motivation. Consider an overemphasis on lead time, which can lead to cutting corners. Or measuring the number of bugs found, which can cause a testers versus developers situation. In this talk, I share some general motivations for measuring quality. I review various commonly used metrics that claim to measure quality. Based on my experience, I rate them with regards to how they may be helpful or harmful to achieve actual goals and which side effects are to be expected. I give some examples how the weaknesses of one metric might be countered by another one to create a beneficial system.