i’ve had better luck illustrating the point with a less abstract case: the 2000’s called and it’s your turn for jury dury. the case for today is that of a single mother who downloaded some Disney movies off Limewire for her kids to watch so she could get some time to herself to take care of chores.
should the jury find her guilty, you suspect that the judge will fine her $250,000 and cancel her home internet connection. you think such a punishment would do more net harm than good. but you don’t get to decide the punishment (that’s for the judge to announce after the jury deliberates), you just decide the guilty/not-guilty verdict.
you look at the evidence: the mother definitely downloaded those files. what verdict do you deliver the judge?
in my head, there’s a direct causal chain:
if i believe (3) and (4) will function as stated, then it’s equally accurate to say that in step 2 i am deciding whether or not to confiscate $250,000 from this mother and cancel her home internet connection.
but a huge number of people i present this to refuse to admit that equivalence. there is some question about whether weakening the norm might cause more damage than mistreating the mother, but does that even weaken the point? the common answer from those who bring it up is “there’s too much uncertainty to say”: build a complex enough machine, and people are eager to deny the downstream effects of their actions.
(you can overcome most of the degradation-of-norms issue by making this a secret hearing, and still a lot of people will hesitate to admit the equivalence between their verdict in step 2 and the effects of step 3/4)