Tolerability of Software Defects Across Industries

When analyzing software testing/bug tracking through a Priority/Severity matrix the same defect can be categorized differently depending on the industry and product of the project (Nadig, 2017). While different organizations might use different terms or levels of classification, the terms here will be defined in four categories. In this post, I will take a look at the definitions of labels used for defects and some examples across industry.

Priority determines how quickly the defect turnaround time must be (Nadig, 2017). Four categories that can be used to determine defect priority levels are:

  • P1 – Showstopper
  • P2 – Critical
  • P3 – Major
  • P4 – Minor

A defect’s Severity reflects how critical the defect is and what the impact of the defect is on the entire system’s functionality (Nadig, 2017). Four categories for labeling the severity of a defect are:

  • S1 – Showstopper
  • S2 – Severe
  • S3 – Moderate
  • S4 – Minor

A content-based website, such as one for MIT’s Math Department, may label accessibility and responsive design issues as P3/S4 which means it is should be fixed soon, but not if there are higher order defects to fix. A news organization like CNN may categorize a defect that causes only the first 1,000 articles to be displayed in their paginated list of articles as P3/S4 assuming the articles can still be found via searching their site. The older articles are less likely to be browsed to and they likely do not bring in as much ad revenue as more recent and timely articles so they do not represent a threat/risk to management.

On the other hand, MIT’s Math Department may consider a typo in a professor’s name, title, or credentials to be a P1/S2 since not only the internal faculty would not be happy about the issue, but their site is one way that they drive funding to the organization as a whole and in the organization’s eyes typos/not correctly advertising their faculty’s experience can hamper their efforts. CNN may classify a misconfiguration that causes their SSL certificate to appear as invalid to be a P1/S1 defect because the issue can lower a site’s PageRank score and lower the results in search engine listings causing revenue loss.

A tolerable (P3/P4 and S3/S4) defect for a financial website may be a website styling issue or responsive design. A credit union account I have has a non-responsive website and iOS app that displays all of my account information but none of the text is aligned and it is just a poor experience.  These are obviously things that should be fixed but the main function of the site and app are to convey my account information on demand which is why I still use them. A tolerable defect for tax preparation software may be a functionality issue where the user needs to click the ‘Next’ button twice to progress to the next step in the process. The user would likely expect to need to click a button once, but the software can still generate correct returns so it is not a showstopper.

If a financial website for a credit union could only accept a maximum 10 concurrent connections that could be considered a P1/S1 defect. An issue like this should not only be under automated testing, but it may stem from the development environment not being tested in a similar environment to production. To mitigate this, all configuration and deployment scripts should be kept in the source code management and optimally systems would be configured with tools such as Puppet to ensure testing environments were similar to production.

A medical device maker may consider an issue where the unit does not turn itself off after use a P4/S3 defect because it does not halt the use or usefulness of the product. A showstopper defect could be one that causes the device to function normally but not provide any output to the end user, definitely a P1/S1 defect. A defect of this caliber would render the device essentially useless, such as the recent string of smart lock and smart TV devices where silent firmware updates caused the main feature of the devices to stop functioning which enraged the end users.

A car navigation system may have a tolerable defect of not being able to show a user’s saved home address as hopefully the user will be able to remember their address to enter it manually if needed. This may be a more critical issue if a company finds that many of their end users are actually using this feature in an unexpected way. This would also be a good opportunity to document how users are currently using the feature and prioritize its implementation with their software engineering team. An example of a showstopper for a car navigation system would be one that prevents spoken audio from playing turn-by-turn navigation because this is a safety feature that prevents users from needing to take their eyes off the road.

Works Cited

Nadig, S. (2017, April 17). How to Set Defect Severity and Priority: Defect Triage Process. Retrieved August 28, 2017, from Software Testing Help: