Jonathan Zittrain – New Yorker
Being understood is not a precondition to being useful. The history of medicine is a history over centuries and millennia of spotting efficacy without the least understanding of the mechanisms by which that efficacy is achieved. Nor is that limited to pre-scientific times. There are still drugs which are prescribed because they work, without any understanding of how they work.
Zittrain calls that intellectual debt (by unspoken analogy with technical debt), where theoretical understanding lags behind pragmatic effectiveness. The problem is not that it exists, as the medical examples show, it is that machine learning takes it to a new level, that our understanding of links between cause and effect come to have more to do with association than with explanation. For any single problem, that may be no bad thing: it can be more important for the connections to be accurate than to be understood. But the cumulative effect of the mounting intellectual debt has the potential to be rather less benign.