Philosophy
When the Machine Says “I Don’t Know”
An automated adviser can say it’s unsure. Critias’ account of self-knowledge suggests asking whether those warnings track the adviser’s actual mistakes.
The Platonic Dialogues: CharmidesA fictional diagnostic adviser gives the Charmides a modern test case. A system’s warnings mean more when they track how often it gets things right.
Orientation
At a Glance
- The standard Critias setsThe temperate person would know what he knows and where his knowledge stops. He could make the same distinction in others.
- A phrase and a recordAn automated adviser can say it is unsure whenever it is programmed to. Its record shows whether those warnings correspond to a higher chance of error.
- How calibration worksIf rain is forecast at seventy percent on many comparable days, it should rain on about seven in ten of those days.
- Blind spots can go unflaggedAn adviser that performs well on familiar examples may sound confident on a case it handles poorly. That possibility gives the comparison with Socrates its bite.
Charmides 166e–167a, 170a–b, 170c–e, 170e–171c, 175b–c · Guo and colleagues On Calibration of Modern Neural Networks
Suppose a hospital installs a piece of software that advises doctors on difficult diagnoses. Most of the time it gives a confident answer, but sometimes it says that it isn’t sure, that the case falls outside what it can judge reliably, and that a specialist should be consulted. The designers are proud of this feature, calling it a sign of the system’s humility. The question a careful doctor should ask is what that humility actually consists of, and what would count as evidence that the machine really knows when it doesn’t know. The feature is reassuring, and that is part of the danger, since reassurance is exactly what a worried doctor wants from an adviser.
Critias’ definition of temperance gives a demanding standard to measure against. The temperate man, on his account, would “examine what he in fact knows and what he does not,” and would also be able to “inspect others, to see what a person knows and thinks he knows.” The ideal is a knower who can tell, reliably and across every subject, where his knowledge stops. That’s exactly the ability the hospital is hoping its machine has, at least in its own field, and the Charmides is a useful guide to how hard it is to have. Critias never shows how such a knower would work, and the hospital’s designers face the same question in a more concrete form.
The first thing to separate is the words from the ability. Saying “I’m not sure” is easy, and a system could be built to say it often, or in a reassuring tone, without its uncertainty bearing any relation to how likely it is to be wrong. An adviser that said “I don’t know” to every question would be perfectly humble and perfectly useless, and one that said it at random would be worse than useless, since it would teach the doctors to trust it exactly when they shouldn’t. A convincing verbal admission is not evidence of anything until it is checked against the record.
The evidence that matters is what forecasters call calibration. A weather forecaster is well calibrated if, on all the days she says there is a seventy percent chance of rain, it rains on about seven in ten of them. A diagnostic adviser is well calibrated if its expressions of uncertainty track its actual error rate, so that it is usually right when confident and noticeably less reliable when it says it isn’t sure. That can be tested only by comparing its statements with outcomes over many cases, and no single admission of doubt, however graceful, can stand in for that record. It also has to be checked separately for different kinds of case, since an adviser can be well calibrated on common illnesses and badly calibrated on rare ones, and an average across all cases can hide the difference.
The Charmides adds a further lesson, because the doctor argument showed that a knowledge of knowledge alone could tell only that one knows, without telling what one knows. Judging whether medical claims are true requires knowing medicine. The machine’s uncertainty flag could be built separately, but whether it works is a question about its performance on actual cases. Even a well-calibrated flag in one setting gives us no reason by itself to trust the flag on a different kind of case.
The dialogue’s closing review adds another warning, since Socrates says there that it is impossible to know, “in some way or other, what one does not know at all.” A case outside what a system was built to handle may produce a confident wrong answer with no flag at all. That possibility matters because it is just on an unfamiliar case that the doctors would most like a dependable warning.
It’s worth being clear about what this comparison doesn’t claim. Plato knew nothing of software, and the Charmides isn’t a prophecy about machines. The point is only that the dialogue’s distinctions, between saying and knowing, between knowing that and knowing what, and between recognizable gaps and total blind spots, apply to any adviser that claims to know its limits, human or mechanical, and give a sharp set of questions for testing the claim. The same questions would apply to a human consultant famous for admitting doubt, since the habit of saying so proves nothing until it is matched against how often he turned out to be wrong.
My verdict is that a machine’s “I don’t know” is evidence that it can recognize some of its limits when it is shown, over many cases, to track its actual reliability. Even then, the evidence is local, tied to the sorts of cases on which it has been assessed. Critias’ demanding proposal sets a much wider standard. The Charmides helps us ask for a more modest and testable thing, which is calibration in a field, earned by a record.
Test your understanding
Question 1 of 3
What would most strongly support the claim that a diagnostic system's expressions of uncertainty are meaningful?
Guided Inquiry
Questions for Socrates
Explore these questions with Socrates, with this Feature as your reading context.
- If calibration shows only that a system recognizes some limits within tested cases, what should doctors reasonably infer—and not infer—from an “I don’t know” response?→
- How does the distinction between knowing that one knows and knowing what one knows affect the value of a machine's uncertainty flag?→
- What kinds of evidence or practices could help doctors respond responsibly to the possibility of total blind spots that produce no warning?→