The Numbers Trap

do you track numbers?

  • We have to get those defect counts down!
  • We have to reduce those cycle times!
  • We have to increase lines of defect-free code written for each release!

We forget that numbers are just a gauge and they don't tell the whole story.

The obsession with increasing/decreasing counts becomes your core purpose and product quality suffers as a result.

Instead of focusing on numbers, we should focus on our core purpose: creating the highest quality product possible.

Here are some things to think about when tracking QA stats:

  • numbers are just a gauge
  • the numbers may not be accurate for a variety of reasons (changes in tracking/reporting methods, team members with varying skill levels doing the reporting causing duplicate bug reports)
  • because the defect descriptions & counts may be inaccurate, using them as the sole measurement of product quality may be misleading.

The most important questions to ask are those that deal with customer satisfaction:

  • Are the user numbers increasing or decreasing?
  • In a poll of customer satisfaction, are there more satisfied or dissatisfied users?
  • Looking ahead, is product development addressing the features most in demand?
  • Do we have the resources to create quality code to create those features?
  • Do our development and QA teams work in harmony for the core purpose of creating a better product?

Spend time tracking numbers, but spend more time tracking product quality as viewed through your customer's eyes.