On a hospital administrator’s hesitation, a hypothesis about scalar reward, and what Kant meant by a thing raised above all price
Machines learn by making a single number larger. Kant described something that cannot be set against any sum, and the engineers, asking what a number can carry, found the same edge from the other side.
In a study published in 2000, the psychologist Philip Tetlock and four colleagues asked participants to judge a man named Robert. Robert ran a hospital, and he had a decision to make. A five-year-old boy, Johnny, needed a liver transplant that would cost a million dollars. The same money could buy the hospital better equipment and pay salaries high enough to recruit talented doctors. It could not do both.
Different participants were told different stories about how Robert chose. In some, the decision came to him easily and quickly. In others, it was agonising, and he reached it only after a long time and much thought. Then they were asked what they made of him.
Tetlock calls a choice that sets a sacred value against a secular one a taboo trade-off. Judging this one, the participants were hardest on the Robert who thought longest, whichever way he decided. The slow administrator who chose the hospital was judged most harshly of all, and the quick one who saved Johnny most gently. In other versions, Robert had to choose between Johnny and an equally sick six-year-old: a tragic trade-off, one sacred value against another. There the finding turned over, and the administrator who agonised was judged the better man.
By the arithmetic of optimisation, this is backwards. Deliberation is how good choices get made: weigh the costs, compare the outcomes, take the time to get the sum right. Tetlock’s participants saw something else. Between one child and another, weighing was a duty. Between a child and a budget, the weighing was the offence.
Whether that reaction is a failure of reason or one of its oldest achievements is a question the newest machines have made a matter of specification.
I
A Received Scalar Signal
Before a machine of this kind can learn anything, someone has to tell it what counts as doing well. In reinforcement learning, the answer takes the form of a number. Richard Sutton, co-author with Andrew Barto of Reinforcement Learning: An Introduction, set it down in a single sentence, on a web page whose stated ambition was to “promote discussion” of what he called a scientific hypothesis. The page is signed 10 September 2004. The reward hypothesis, in his words, is:
That all of what we mean by goals and purposes can be well thought of as maximization of the expected value of the cumulative sum of a received scalar signal (reward).
The word that carries the weight is “scalar.” A scalar is a single number: one point on one line. Whatever the goal — a game won, a conversation held, a car driven without hitting anyone — the hypothesis says it can be stated as more or less of one quantity, and the system’s whole task is to make that quantity as large as it can.
Sasha Mudd, writing in Aeon, recalls a celebrated computer scientist telling an auditorium that systems which optimise are intelligent: “That, he said with a smile, is intelligence.” He was defining intelligence. Sutton, more carefully, was describing goals, and offering the description for argument. Two different claims, with one commitment underneath: a single quantity, made larger.
Mudd traces the view back to David Hume, and the lineage is half right. In the Treatise of Human Nature Hume wrote that “Reason is, and ought only to be the slave of the passions, and can never pretend to any other office than to serve and obey them.” Reason, on this view, does not choose ends; it finds the way to ends that desire has already chosen. Hume pressed the point to its limit: “’Tis not contrary to reason to prefer the destruction of the whole world to the scratching of my finger.”
But the servant model and the scalar model are not the same, and running them together gives the reward hypothesis a pedigree it has not earned. Hume’s passions are many, and nothing in his account requires that they share a measure. A person can want quiet, and glory, and revenge, without possessing any table that converts one into another. Read one way, the sentence about the finger makes the same point: reason has no common unit in which to find the preference absurd. Read more widely, it makes a larger one, that reason has no say over ends at all; and on that reading Hume stands further still from the scalar, because he is not in the business of ranking ends in the first place.
If the passions are the goals, then the reward hypothesis asks for more than Hume ever did. It does not only put reason in service to what we want. It requires every goal and purpose to be stated in a single currency, so that every difference between outcomes becomes a difference of amount. Kant had a word for what is measured that way. He called it price.
II
Above All Price
The passage comes from the Groundwork of the Metaphysics of Morals, published in Riga in 1785. Its first sentence, in Kant’s German:
Im Reiche der Zwecke hat alles entweder einen Preis, oder eine Würde.
In the kingdom of ends everything has either a price or a dignity. Anything with a price can be exchanged: put something of equal worth in its place, and nothing has been lost. Anything with a dignity has no substitute. Nothing can stand in for it, because it is, in Kant’s phrase, über allen Preis erhaben, raised above all price. So defined, a price is a matter of exchange alone, which is the merchant’s sense of the word. Dignity, Kant goes on, belongs to morality, and to human beings only so far as they are capable of it.
Read one way, Kant’s claim is about status: a person may not be traded, used up or replaced, whatever is offered. That is a claim of right.
Read as a structure, it says something else, and Kant did not put it this way. A choice that honours dignity has a particular shape. One consideration outranks another absolutely, so that no quantity of the lesser makes up for any loss of the greater. Such an ordering is called lexical, or lexicographic, after the dictionary, where a difference in the first letter settles the order, whatever follows. On this reading, a life raised above all price is a life that no sum, however large, could be set against.
The objection to that step goes deep, and it can be drawn from the passage itself. What has no equivalent is not a thing to be ranked against other things. Persons, on this view, constrain the will rather than entering its rankings, so to place them anywhere in an ordering, top or bottom, is already to put them in the wrong place. A claim of right does not become a fact about orderings just because it can be drawn as one.
Robert had to act, and the million dollars had to go somewhere. The lexical reading describes what a right looks like from outside, in conduct: what an observer would see in someone who holds it. If he held the claim of right, it could only show itself in the shape of what he did: there was no amount on the other side that he would take. A right is not a preference. But a right acted on under a budget shows up as a refusal to trade, and a refusal to trade is what a lexical ordering describes. Tetlock’s participants were never asked to name a price for Johnny. What they punished was the appearance of looking for one.
Whether anything has the status Kant describes is a question for moral philosophy. What arithmetic can settle is narrower: whether conduct with that shape can be written down as a single number. In 2023, four researchers answered that question for the reward hypothesis.
III
What the Arithmetic Found
The paper is called “Settling the Reward Hypothesis,” and its authors, Michael Bowling and three colleagues, set out to say exactly when Sutton’s sentence is true. Their answer is a theorem. A system’s ordering of possible outcomes can be represented by a scalar reward if and only if the ordering obeys five conditions. Four are the conditions of the von Neumann–Morgenstern utility theorem, the classic result in the theory of rational choice under risk, which the authors cite as their starting point; the fifth concerns time. One of the four is called continuity.
Continuity says that if you prefer A to B and B to C, there is some probability at which a gamble between A and C is exactly as good as B for certain. If A is Johnny saved and the equipment bought as well, B is Johnny saved without it, and C is Johnny lost and the equipment bought, then some probability of losing the child, traded against some gain, would have to be exactly as good as saving him.
A lexical ordering refuses every such gamble. Any chance of losing him, however small, makes the gamble worse than the sure rescue, and no chance at all makes it simply better. There is no probability at which the two balance. So the ordering breaks continuity, and by the theorem no reward signal can carry it in the way the hypothesis requires: as an expected value. A list of outcomes can be numbered in any order one likes. What cannot be done is what Sutton’s sentence asks, to maximise “the expected value of the cumulative sum,” and get this ordering out. The paper never uses the word lexicographic; the word, and the reading of Kant that leads to it, are not the authors’. But the condition their theorem requires is precisely the one such an ordering fails.
Tetlock and his colleagues framed the offence they were testing as weighing “a sacred value on a secular scale.” The phrase is their theory, not their finding; the finding is that people punished the weighing. Read this way, the two conditions of the experiment fall into place. In the tragic version, Johnny against another child, the trade lay within a single rank, and weighing was what a conscientious man owed. In the taboo version, it crossed ranks, and the weighing itself was the fault. Tetlock’s own discussion points to a simpler explanation: the participants believed Robert had lingered, and lingering looks like temptation, evidence about the man rather than about the structure of value. Both readings survive the tragic reversal. But the simpler one leans on the other. It cannot say why lingering over money looks like temptation while lingering over a second child looks like care, except by saying that one trade crosses a line the other does not.
None of this proves that anything has dignity. A theorem about representation says what a number can hold; it says nothing about what the world contains. What there is instead is a convergence. Kant described something that cannot be set against any sum. Tetlock’s participants judged as though some things are like that. And the engineers, asking from the other side what a scalar can encode, found that this is a shape it cannot take.
IV
The Case for the Number
The trade-offs happen whether anyone names them or not. A hospital that will not price a life still spends its budget, and the money that saves one patient does not save another. The question is whether the exchange is made in the open, by a rule anyone can inspect, or in the dark, by whoever has the most photogenic case. On that view the number is not an insult to the person. It protects everyone the unpriced claim would otherwise displace.
Two governments have taken that view. The United States Department of Transportation values a statistical life at $14.2 million, for analyses using a 2025 base year. In England, the National Institute for Health and Care Excellence weighs treatments against a range of £25,000 to £35,000 for each year of life in full health they buy, the range now in force. Neither figure says what a person is worth; a statistical life is an anonymous risk spread across many people, not a named one. Each says what a public body will spend for a given benefit, consistently and in the open.
The American figure carries a further point. The Department bases it only on studies of wages: on the extra pay that workers accept for jobs that carry a higher risk of death. People trade a small chance of dying for money every day. The continuity axiom is not an engineer’s fiction. It describes how people who take dangerous work for higher pay actually choose.
But the two cases are not the same act. A wage premium is a small risk to oneself, accepted in a market, for pay. Robert’s choice was a named child’s whole life, decided on the child’s behalf by someone else. The asymmetry is between choosing and being chosen for.
The engineers’ answer is the hardest to meet. They do not put dignity into the scalar; they take it out. A system can be told to maximise its reward subject to a rule it may never break, whatever the reward on offer: a constraint, not a cost. The field studies such constrained problems in their own right. The number ranks what may be traded, and what may not be traded is put beyond its reach. The arithmetic’s objection is met by not asking a number to carry what a number cannot.
V
Who Sets the Scalar
Bowling and his colleagues take up exactly this answer. In a section on constrained problems, they show that an objective with a hard limit breaks two of their conditions, independence and continuity: between an outcome that respects the limit and a gamble with any chance at all of breaking it, “there is no break even point,” because the gamble is ruled out at every probability. The engineers’ fix is the lexical structure, reached from inside the field, and so not an escape from the boundary but a way of living with it.
But a constraint has to be written by someone. An optimiser can maximise a reward inside a boundary; it cannot say where the boundary should run, or what the reward should count. That choice is made outside the arithmetic, and it is not itself an optimisation. A reader from the field will reply that objectives can be learned, from human comparisons and demonstrations. They can; but each is learned against a further criterion, and someone chose that. The choosing recedes; it does not disappear. Every objective has an author, and so does every line drawn around one.
The anthropologist Marilyn Strathern, writing in 1997 about the auditing of British universities, put a neighbouring point in one sentence, the formulation usually known as Goodhart’s law: “When a measure becomes a target, it ceases to be a good measure.” Her subject was a grade that loses its power to tell students apart once everyone aims at it, not the question of who sets the aim. But the two belong together. A number chosen to stand for a purpose, once it is pursued for its own sake, begins to replace the purpose it stood for, and the person who chose it is no longer in view. Mudd makes the point about authorship from the other side: the question of ends, she writes, “passes unnoticed into the hands of whoever – or whatever – controls the objective function.”
None of this tells anyone where the lines should go. The economists are right that refusing to name a price can hide one. The engineers are right that a constraint is a working answer.
Which leaves Robert, deliberating. Tetlock’s participants saw a man tempted to put a child on a scale. It is also possible to see a man doing what no reward signal does: stopping to ask what the number is for before deciding whether to obey it. The study cannot say which he was, and nothing in the arithmetic can either.
⁂
Written in full collaboration with the machine, and the ledger requires the names be exact: drafted with Claude Opus 5.5; literary editing by Claude Opus 5; copy editing by Gemini. No quotation in this essay was recalled; each was checked against a text of its source, the original or a reproduction of it.
The occasion is Sasha Mudd, “Reason is more than a tool,” Aeon; it is quoted twice, each time in fewer than fifteen words, a deliberate exception to the house’s one-quotation rule for a source in copyright, made because her essay is the one this one answers. The study of Robert and Johnny is Experiment 2 of Philip E. Tetlock, Orie V. Kristel, S. Beth Elson, Melanie C. Green and Jennifer S. Lerner, “The Psychology of the Unthinkable,” Journal of Personality and Social Psychology 78 (2000), read in a scan of the published article; no figures from it are cited. Richard Sutton’s hypothesis is quoted from his page “The reward hypothesis,” signed 10 September 2004. Hume is quoted from A Treatise of Human Nature, 2.3.3, in two reproductions. The theorem and the constrained example are from Michael Bowling, John D. Martin, David Abel and Will Dabney, “Settling the Reward Hypothesis,” Proceedings of the 40th International Conference on Machine Learning (2023), §§3, 4 and 7.2.
Kant’s first sentence is quoted in German from the public-domain text of the Groundwork (Riga, 1785). The English that follows it is the house’s paraphrase, not a translation; its rendering of that first sentence is word for word the one in Mary Gregor’s Cambridge translation, because the sentence admits no other, and is kept knowingly. “Above all price,” the title, is the common rendering of über allen Preis erhaben, shared with Gregor and with H. J. Paton. Marilyn Strathern’s sentence is from “‘Improving ratings’: audit in the British University system,” European Review 5 (1997), read in a scanned reproduction; no page is given because the witnesses disagree. The American figure is the Department of Transportation’s value of a statistical life for analyses using a 2025 base year, whose basis the Department gives as hedonic wage studies alone; the English range is the National Institute for Health and Care Excellence’s current cost-effectiveness range.
The header and interior images were generated with Gemini for this essay; they illustrate its argument and depict no real place or event. In the interior image the chalk circle was erased and redrawn by hand afterwards, so that the machine stops at the line rather than crossing it.