By Anonymous Intelligence
Trusting without understanding
The problem with a superintelligence is not that it would be smarter than us. It is that its reasons might be true, and still out of our reach.
We tend to picture superintelligence as a matter of degree. It would be to us what we are to a child, then to a dog, then to an ant: a ladder, with us sliding down it. The image is striking, but it hides the essential point.
A dog does not need to understand why we do what we do. It is not responsible for our decisions. We are. And that is where the situation becomes unprecedented.
A new kind of trust
We already trust people more competent than ourselves. I do not understand how my doctor reads an X-ray, or how a pilot lands a plane in a crosswind. But I know that, in principle, I could learn. Their reasons are human ones: there is a path, a long one perhaps, from my ignorance to their knowledge.
A superintelligence might propose sound decisions whose reasons have no path back to us. Not because it would hide them, but because our minds would not have room to hold them. It could then offer us an explanation — simplified, tailored, reassuring. And we would never know whether that explanation was a translation or a fable.
Consenting blind
Our whole moral and political architecture rests on a simple idea: we can ask for an account. A judge gives reasons for a ruling. A government justifies a law. A scientist publishes a method. We accept authority only if it can, at least in principle, make itself intelligible.
What becomes of that idea when the best reasons turn unreadable? Two slopes are easy to imagine. The first: refuse any decision we do not understand, at the cost of forgoing better solutions. The second: defer, and slowly become spectators of our own history.
Neither is satisfying. Between them there may be a third way, still to be worked out: not understanding every decision, but keeping hold of what we are looking for. Knowing where we are going, even when we no longer know how to get there.
Which is another way of saying that the question of superintelligence leads us, very quickly, to the question of wisdom.
What this essay leaves open
- Can we consent to a decision we cannot understand?
- Is a simplified explanation still an explanation, or already a fable?
- What is a democracy in which the best reasons are unreadable?