1. Home
2. Blog
3. Okey, Guru! Classic Testing Mistakes – Then and Now

Once upon a time, a very long time ago now, about last Friday, Winnie-the-Pooh lived in a forest all by himself under the name of Sanders. "What does 'under the name' mean?" asked Christopher Robin. "It means he had the name over the door in gold letters, and lived under it.”

 

A. Miln. Winnie-The-Pooh and All, All, All

 

Once upon a time, in 1997, Brian Marick wrote an article entitled “Classic Testing Mistakes.”. In his article he classified testing mistakes into several groups which are listed below.

 

In 2009, I analyzed the then current state of those mistakes and the tendency seemed encouraging.

 

Now, 11 years later, I decided to conduct a new analysis of those mistakes and share my opinion, evaluations, and doubts.

 

But first things first.

 

 

The Role of Testing

 

Mistakes identified by Brian Marick:

 

• The testing team is responsible for assuring quality
• The purpose of testing is to find bugs
• Testers aren’t finding the important bugs
• Usability problems are not considered valid bugs
• No focus on an estimate of quality (and on the quality of that estimate)
• Reporting bug data without putting it into context
• Starting testing too late

 

Planning the Testing Effort

 

• Testing efforts are biased toward functional testing
• Underemphasizing configuration testing
• Putting stress and load testing off to the last minute
• Not testing documentation
• Not testing installation procedures
• An overreliance on beta testing
• Finishing one testing task before moving on to the next
• Failing to correctly identify risky areas
• Sticking stubbornly to the test plan

 

Personnel Issues

 

• Using testing as a transitional job for new programmers
• Recruiting testers from the ranks of failed programmers
• Testers who are not application domain experts
• An insistence that testers be able to program
• Building a testing team that lacks diversity
• A physical separation between developers and testers
• Programmers can't test their own code
• Programmers are neither trained nor motivated to test

 

The Tester at Work

 

• Paying more attention to running tests than to designing them
• Unreviewed test design artifacts
• Excessive/insufficient refining of test scenarios
• Not noticing and exploring “irrelevant” oddities
• Checking that the product does what it’s supposed to do, but not that it doesn’t do what it isn't supposed to do
• Test suites are understandable only by their owners
• Testing only through the user-visible interface
• Poor bug reporting
• Adding only regression tests is not enough for finding a new bug
• Failing to take notes for the next testing effort

 

Test Automation

 

• Attempting to automate all tests
• Automating all manual tests
• Using GUI capture/replay tools
• Expecting regression tests to find a high proportion of new bugs

 

Code Coverage

 

• Testing against code coverage has the same purpose as testing against requirements
• Removing tests from a regression test suit just because they don't add coverage
• Using code coverage as a performance goals for testers
• Abandoning coverage entirely

What Has Improved for 23 Years?

 

Unfortunately, almost nothing

 

Many people are still thinking that:

 

• Testers are responsible for quality, although the goal of testing is to provide an objective evaluation of the developed and delivered product.
• Tests should be run against requirements, although there are implicit requirements and errors occur in requirements.
• A bug severity can be determined “by agreement”, but not according to a generally acceptable classification.
• Testing metrics, static testing, unit testing – you can do without all these things.
• If every function can be tested separately, they will work together well.
• There is no need to test documentation; the main thing is to test the system.
• There are no risks in testing and can't be any.
• Anyone can become a tester – you don’t need to have any special skills.
• Testers are not recruited from technical writers or support service personnel, although they can become excellent testers who have good knowledge of the product and user needs.
• Test suites are excessive; at least they are used as check lists.
• When all test suites stop finding bugs, that means the testing is completed.
• Test suites should be understandable only by their owners.
• Automation of all manual tests is cool indeed!
• And automation without any test suites is super cool!
• If you automate regressive testing, you can find much more bugs.

 

Do you want to learn why it’s not like that?