Software Testing Economics

I have been thinking about this subject for almost a year, and I don’t know the answers to many questions yet.

A year ago, I was surprised to hear that the statement ‘Exhaustive testing is impossible’ was not obvious to everyone. I happened to speak with a person who asked ‘Why?’ and I referred him to the book by Glenford Myers, Software Reliability: Principles and Practices, where an example was given. The answer was: “Well, but that is not mathematically proven there!”. I recalled my mathematical background and said that there is Cantor’s theorem which states that a set of natural numbers is countable and if it is countable, then it is not finite. And if we get a calculator which adds natural numbers, then it will be IMPOSSIBLE to test it exhaustively, and so on and so forth.

Another source for arguments is the ISTQB syllabus which unambiguously states that exhaustive testing is impossible. Yet another source is a consequence from the definition of a Turing machine – that its holding problem is unsolvable, as stated in a standard course of automata theory. And finally, common sense: we are all software testers (those who are not yet should want to become testers, as there is no other way for human evolution). And we have some testing experience and understanding that it's impossible to write ALL test cases and select ALL test data. That we cannot travel ALL the paths in code, and perform ALL actions that users can do, and son and so forth.

It seems to be an absolute truth, but questions on this theme arise again and again at various levels, and especially at the top management level:

• “When are you going to complete the software tests?” (“To complete” means “to have everything done”).
• “When will ALL the defects be found?” (Of course, nobody will be satisfied with a half or even 99% of found bugs)
• “And when will we deliver bug-free software to the customer?”
• If there is a defect in operation - “Why was this defect not found?”
• “And why were not ALL the defects in operation found before commissioning?”

How should you answer these questions? They all imply exhaustive testing, which we cannot conduct.

• If the damage from an unclosed defect does not exceed the costs of its correction, you don’t need to correct this defect
• If the damage from a potential defect does not exceed the costs of its searching and correction, you don’t need to search this defect
• If the overall damage from all known unclosed defects does not exceed the benefits of implementing the existing system version, it should be implemented (even with such known unclosed defects)