The idea of the black box pervades a lot of thinking and writing about AI. Mysterious algorithms do inscrutable things which impinge on people’s lives in inexplicable ways. That is alarming in its own right, but doubly so because this is new and uncharted territory. Except that, as this post painstakingly points out, it’s not actually new at all. People have been writing software about which they could not predict the outputs from the inputs since pretty much since they have been writing software at all – in a sense, that’s precisely the point of it. And if you want to look at it that way, the ultimate black box is the human brain, where the evidence that we don’t understand the reasons for our own decisions, never mind anybody else’s, is pretty overwhelming.
The need for precision at one level – software doesn’t cope well with typos and syntax errors – doesn’t translate into precision at a higher level, of understanding what that precisely written software will actually do. That thought came from Marvin Minsky in 1967, but people had been writing about black boxes for years before that, when the complexity of software was a tiny fraction of what is normal now.
The fact that this is neither new nor newly recognised doesn’t in itself change the nature of the challenge. What it does perhaps suggest, though, is that strategies developed for coping with these uncertainties in the past may well still be relevant for the future.