Its not malice but incompetence thatll kill us
I’ve been reading You look like a thing and I love you by Janelle Shane. And, honestly, it’s some of the best skewering of Artificial Intelligence I’ve come across. But amid the funny stories of AI incompetence – only recognising sheep when they’re in fields, thinking a goat in a tree is a giraffe etc. – there’s a serious point about the impact of these limitations.
As more of our daily lives are governed by algorithms, the quirks of AI are beginning to have consequences far beyond the merely inconvenient. Recommendation algorithms embedded in YouTube point people toward ever more polarizing content, traveling in a few short clicks from mainstream news to videos by hate groups and conspiracy theorists…
…The algorithms that make decisions about parole, loans, and resume screening are not impartial but can be just as prejudiced as the humans they’re supposed to replace—sometimes even more so. AI-powered surveillance can’t be bribed, but it also can’t raise moral objections to anything it’s asked to do. It can also make mistakes when it’s misused—or even when it’s hacked. Researchers have discovered that something as seemingly insignificant as a small sticker can make an image recognition AI think a gun is a toaster, and a low-security fingerprint reader can be fooled more than 77 percent of the time with a single master fingerprint.
People are generally quick to chalk up to malice what is better explained by incompetence. The righteous outrage in the papers is full of pinstriped fat cats rather than honest mistakes and fallible processes (sometimes rightly so, but probably not as often as portrayed).
It seems our fears for the future fall into this same trap. What truly scares me about AI is generally the outcome of a perfectly running system. Skynet murderbots conquering the world. Or batallions of worker drones tilting the balance further in favour of capital and the technologically competent.
But my recent studies of machine learning, and this book specifically, make we wonder whether the true worry shouldn’t be bias and incompetence. Not a murderous bot. Rather one that just isn’t ready for prime time. In that sense the problem isn’t the technology itself, but the underlying systems that deployed it (I didn’t explicitly write “capitalism” here, but I wouldn’t fault you for reading it in 😇).
If there’s malice in the system, that’s where to find it.
When people think of AI disaster, they think of AIs refusing orders, deciding that their best interests lie in killing all humans, or creating terminator bots. But all those disaster scenarios assume a level of critical thinking and a humanlike understanding of the world that AIs won’t be capable of for the foreseeable future. As leading machine learning researcher Andrew Ng put it, worrying about an AI takeover is like worrying about overcrowding on Mars.
One of Shane’s principles of “AI weirdness” is that it does not really understand the problem you want it to solve. Another is that it will take the path of least resistance to achieve what you tell it to.
When you’re using AI to play a game this can lead to some obnoxious cheating. When you’re deploying it in the real world this can result in further entrenching bias, hierarchy and revealed preference. Often in ways we don’t understand and expect. Particularly if the implementors aren’t aware of the underlying bias in the system they are a part of.
The problem with designing an AI to screen candidates for us: we aren’t really asking the AI to identify the best candidates. We’re asking it to identify the candidates that most resemble the ones our human hiring managers liked in the past. That might be okay if the human hiring managers made great decisions. But most US companies have a diversity problem, particularly among managers and particularly in the way that hiring managers evaluate resumes and interview candidates. All else being equal, resumes with white-male-sounding names are more likely to get interviews than those with female-and/ or minority-sounding names. 5 Even hiring managers who are female and/ or members of a minority themselves tend to unconsciously favor white male candidates. Plenty of bad and/ or outright harmful AI programs are designed by people who thought they were designing an AI to solve a problem but were unknowingly training it to do something entirely different.
As always my emphasis