Software Testing Economics. Typical questions and answers continued

Question

If everything cannot be tested, does it mean that quality will always be unpredictable? And how can we predict it?

Answer

It can be predictable. Yes, there are quantitative control methods — “six sigma”, statistic control, etc. You can assess the number of defects at output with a probability of up to 99.97%. If there are more than 30 measures, then statistics are quite reliable, and you can predict the number of defects. This requires a certain process culture and certain engineering culture in software testing, but it’s possible to make assessments to a specific level and with a specific accuracy. Then you will say that if tests are conducted with certain parameters, you will have two serious defects per 10,000 code lines in production. We can assess costs and benefits and defend this assessment.

Let's consider the following situation: we have microeconomics, and a hired team which can conduct software tests of a product of required quality for the customer for a certain amount of money. Along with the team microeconomics, there is big market microeconomics. When the customer asks us to provide the highest quality, we will, in return, ask him to provide us with resources.

The cost of resources in our team’s microeconomics and the cost of resources in market microeconomics (for the testing period) are different. The same work is more expensive in the market. Do you have such situations and how do you keep the balance between the cost of services provided by the internal team and the cost of service provided by an external team?

Answer

It's difficult to give an answer to this question based on my own experience. As an engineer, I am excluded from the project economics. As a manager, I sometimes know something about the project economics, I know of labor rates for our employees, but nothing else. If your employees have the same expertise as in the market but their cost is lower, this is a great advantage which can help you retain the customer.

But it’s not always so good. For instance, there are cases when a big project enters the market, and sucks out all the experts by using higher rates. Another example is a company for which I served as QA Director for two years. They employed unique experts (who, among other things, corrected the Linux core) who twice a year came to the owner and asked for salary increase. The owner could not reject the salary increase because the loss of experts means the loss of business. Yet to increase salaries means that the company loses its margin, profit, and business. If the cost of your employees is higher, there’s a danger that competitors would offer to do the same things but 1.5 times cheaper. They might fail doing the work, yet such price dumping is typical at tenders. Even if they fail the project, it doesn’t matter much for us since we failed to get the project.

A solution is to provide advanced and continuous training for employees. And you can assign more complex tasks to “expensive” employees, to motivate them with interesting work. And assign routine tasks to new employees who are “cheaper”, and then train them, etc. But it’s absolutely obvious that this should be based on a reliable foundation of processes. If everything is not written down and depends on a couple of experts, communicating information from one employee to another will cost a lot and take time.

How can you assess costs for training and for testing? Are there any methods for finding a balance between hiring experts from outside or less expensive training of your employees?

Answer

I don't know any special studies on this subject. If you provide training in general, then it’s a strange approach. If you provide training in specific things necessary for a given project, then such an approach will greatly depend on the project, and it would be strange to give any general figures.

Employees are introduced into a project when the project already exists. If you get them beforehand, there’s a risk that the project won’t happen and costs won't be returned.