Software Testing Economics. Typical questions and answers

Question

Is it worth for testers to stick to their 500 man-hours they estimated, or agree to the management’s estimation, thereby shifting responsibility for the outcomes to them?

Answer

Don’t indulge in illusions that you are shifting responsibility. If you agreed to 300 man-hours instead 500, then you take the responsibility for testing everything within the 300 man-hours limit. They will come and ask: So, what about your promise? You said you would test EVERYTHING in 300 man-hours, and there would be no defects. But defects exist. What did we pay you for?

If you know how to conduct software testing within 300 man-hours, and 200 man-hours was enough, then go for it. Otherwise, here’s a gun-powder barrel on which you’re sitting, and be happy that you are unaware how long the fuse is and don’t know when it will explode.

How much time will it take to make an estimation for the project and who will do it?

Answer

It doesn't take much time, but you should understand that our company has always been a process-based company. We have a process for testing efforts estimation, and this process is flexible enough.

Speaking about the role, it’s usually a test lead or a test manager who does such estimation. But sometimes there is only one tester in the team — a manager, lead, and automation engineer all in one — who will make the estimation.

Speaking about resources, software testing is done by those who are qualified enough to do it.

Speaking about time, it may take from four hours to two or three days, if the project is manageable. If the project is huge, such estimations should be done regularly because they can get outdated along the way. It’s not such a long time that it could impede the work, provided however that you have an estimation process in place. If you do it every time from scratch, it’s a deadlock.

Should we include an estimation of labor efforts for labor efforts estimation in testing?

Answer

It depends on the granularity of your planning. If you are planning on an hourly basis, then you certainly should. If you are planning on a daily basis, I would say no.

Question

As already mentioned above, you should compare labor efforts and costs of debugging with possible damages of defects being discovered during the operation. How should we assess a damage from a defect, especially if it is not found yet?

Answer

As follows: we are managing risks. Any defect is actually a downside of a risk. The risk is that we may get a defect at the production stage and then see: we should know the customer’s business well enough, and we should understand what financial losses they might have if some functionality is unavailable for a day.

This assessment can be very useful for justifying software testing costs. When the customer asks, “Why should you have 500 man-hours for testing?” you can answer, “OK. Let’s cut it to 300. This functionality will fail every month, and the bank's core system will remain down for six hours. Can you calculate how much will this downtime cost to you?” This will be the amount of losses. You don’t want to pay for 500 men-hours – then you will have losses, which will cost as much (usually more than 500 hours for testing).

If you don’t know some details about the business, you can ask those who know. We cannot test everything. But we can test a large part of everything, and get a certain result. We can test less, and then there may be other consequences. Someone should take responsibility for decisions related to financial risks. Our task is to identify those risks and provide choice.

If risks are properly identified and assessed (with such financial figures as benefits, losses, costs), then we can speak the same language with the customer and top management. The higher the management level, the more they are concerned about money and the less about technologies.