Weapons of Math Destruction
-
Share:
-
-
-
The author is a mathematician and expert in financial market analysis. Her own experience, observations and research have led her to the idea that sometimes algorithms based on Big Data and machine learning can increase inequality and even threaten democracy.
Why and how can it happen? Computers, with ever increasing frequency, make important decisions that impact human life. Whether a high-school graduate will be enrolled to a college, or a candidate will be invited to a job interview, this or that employee will be laid off, granted a loan or not, how much health insurance will cost - such decisions are often made on the basis of mathematical models.
However, any model (by definition) is a kind of simplification. What should be included in a model, and what should be thrown away as insignificant, depends on those who create the model. As Cathy O’Neil puts it, “Models are opinions embedded in mathematics.” Not all models are equally good.
One of the critical problems mentioned by O’Neil is the use of proxy variables. Often you don’t have a possibility to get data on some desirable property, and in this case you can use variables with values that correlate with such an unknown variable.
For example, when employers look for an employee, they want to determine how conscientious, responsible and decent a candidate is. It is really hard to measure such important personal traits... Many American companies are increasingly using credit scores as a proxy variable – it’s a numerical expression that characterizes the borrower’s ability to pay back the debt. Having a good credit record is now becoming necessary for getting a decent job.
There seems to be some common sense: a conscientious person will most probably pay their bills and fulfill other financial obligations in time. It’s not so evident, however. A quite decent person may slide deeper into the debt pit or even go bankrupt. For example, after losing your job (it may happen for reasons beyond the person’s control, especially during a recession), an average American loses his/her health insurance as well. An injury or sudden illness can make a person bankrupt, and most personal bankruptcies in the U.S. are caused due to enormous medical expenses.
In a way, such a model works as a “self-fulfilling prophecy” – the credit record being spoiled, it becomes harder to find a job, debts are accumulating, and employment is getting increasingly difficult. And a valuable citizen is becoming cash starved.
Furthermore, proxy variables are used even in cases where more reliable data is available. For example, in Florida, the amount of car insurance depends on the driver’s credit score even more than on their driving record: drivers with clean driving records and poor credit scores pay an average of $1552 more than the same drivers with excellent credit and a drunk driving conviction. A more reliable indicator is just ignored. Is there any fairness or common sense? It’s a kind of rhetorical question...
Another example given by O’Neil in her book is the scoring system for teachers, which is widely used in many schools. Teachers were assessed depending on how their students improved on their test scores in a year. Teachers who underperformed were dismissed. The educational progress of the students was measured on the basis of their tests scores at the end of the term.
Such an approach may seem reasonable at first glance. But if you think further, it becomes evident that students’ scores do not depend only on the teacher's professional skills and personal traits, and it’s a sophisticated problem to distinguish the teacher’s input from other factors. In terms of statistics, the sample is laughably small: teachers were evaluated on the basis of 25–30 students’ scores. It’s a tiny amount. To analyze and compare teachers’ performance in a statistically reliable way, it should be assessed based on thousands or even millions of randomly selected students, as Cathy O’Neil points out. Of course, it is just impossible. The chosen model itself is very doubtful. Nevertheless, based on these scores, real people, probably quite good teachers, were dismissed.
Another side effect is that input data can be falsified, and people being assessed have serious motivation to do that. Actually, it happened. Fraud in term test scores had been revealed in many American schools. Thereby, teachers tried to secure themselves from dismissal or even get bonuses.
O’Neil also touches upon the theme of personalized advertising. It may seem a harmless thing at first sight. And what’s so terrible in that you are prompted to buy something you have googled recently? Along with quite legitimate and acceptable uses, however, there are certain malpractices verging on fraudulent activities. For example, the organizers of ad campaigns specifically track poor and vulnerable groups to offer them high-interest payday loans (and this when they could actually get much more beneficial loans).
Even more dangerous, according to O’Neil, is the use of personalized ads for political campaigns. Gone are the days when presidential nominees had to present their programs to all Americans. Modern technologies enable them to deliver personalized messages to every citizen and promise him/her what they would like best. One message for those who are preoccupied with environment protection, another for those who worry about the flow of migrants; one thing for college graduates, another for elderly farmers... No one will have a complete picture. Even if you visit a candidate’s official web site, it does not necessarily mean that you will see the same information as your neighbor. In fact, there is an unlimited possibility of manipulating the voters.
In the last chapter of her book O’Neil focuses on enormous possibilities of influencing the public opinion by such Internet giants as Facebook and Google. They can control what the users will see in their feed or search results. What if they want to influence the outcome of elections for their own benefit? O’Neil does not blame them of anything, stating that there is no evidence, no reason to believe that they have already done something like that. But they do have such a possibility, she thinks. And it’s hard to disagree with her opinion.
My review makes no pretense to completeness; the book contains much more real life cases. You should read the whole book :)
Victoria Slinyavchuk
Consultant on Software Testing