Software Testing Economics. Testing Strategy

What does all this mean in terms of building a test strategy? Testing cannot be completed in the general sense. It can only be stopped, and therefore we need some test completion criteria, which are both technical and economical. I would dare to say they are only economical, but this might not be acceptable for everyone. There is a basic difference between what we, testers, do, and what developers do:

• Develop everything - Possible
• Test everything - Impossible

Software developers can do everything. They are able to answer the question “How much time will it take to code everything?” – two and a half months, for example. But why? Because there are 18 features, one will take 3 days on average.

And how much time will testers need to find all defects? And how much time will testers need to write all test cases? And how much time will testers need to check all the features? There will never be enough time. It means we should say when we stop software testing and why we stop it. And such criteria should be both technical and economical.

We should estimate labor efforts for testing, yet not in general but such as we need to achieve a certain effect. We must be able to plan a certain point when testing should be stopped and we can definitely expect to get a certain state of the system.

And here we could stop. We know (in purely technical terms) what testing labor efforts are spent for, and it’s usually for the following activities:

• Reviewing requirements
• Developing test cases
• Running test cases manually
• Automating test cases
• Analyzing the results and generating reports

Investments in labor must return (ROI) as a high quality product being tested. We know what we have to spend time and money for – we must engage people with certain expertise, pay for automation tools, and so on. We must invest in testing, including labor efforts. But any investor would ask what they will get in return. And what can we provide in return? High quality.

Let’s look at the anatomy of the planning process.

It should be planned so that the costs are realistic (within the project budget) and effective (can bring in the expected result). If we promise that we need 5,000 men-hours and we’ll test something, then we’ll be punished after that without apologies. If we promise that we are genius software testers and find all defects in three days, then we’ll be punished too but later when undiscovered defects emerge at the production stage. Why should we promise anything then? We must stand by our words.

That means we should use certain methods and models for estimating the labor efforts for testing (I told about that in 2011 at the SQA Days in Moscow — my presentation was entitled “Estimation of Labor Efforts for Testing in Maintenance Projects”). Based on your estimations, you can plan testing activities. You should also track the testing process in order to timely discover delays against the plan and correct it.

So, suppose we have absolutely correctly estimated the labor effort, then the required amount of men-hours was assigned to us, and we did everything right. But since it never happens like than in real life, we need to see what may happen if something goes wrong.

We may underestimate testing labor efforts. We may underestimate them from the very start. For instance, we asked for 5,000 men-hours, but they cannot allow such an amount for the project, and suggest getting 3,500 men-hours and getting away with it somehow. Well, let's try. Another example: some things are always changing in projects, from fashion requirements to platforms. All these changes may deteriorate the quality of the product under test, because we had an initial plan but once it becomes irrelevant we don’t have it. There was an underestimate or circumstances changed – everything can happen, and all this can lead to product quality deterioration, when we don’t have enough capabilities to test everything we planned, and with planned quality. What and how should we do in that situation and what hidden pitfalls are there?

Why does underestimation occur? For instance, the whole project budget was underestimated. Sometime the cost of resources turns out to be unexpectedly high. For example, we planned that junior testers would write test cases. I (and not only me, I hope) would like very much to see a junior tester who can properly write test cases. I have never seen such people in my whole life. Evidently, you will have to write and rewrite what they have done.