This post was co-written by Brian Knab and Miriam Schoenfield.
1. A Simple Case
Consider two people contemplating the origin of the universe.
The simple deist is confident that a being exists that designed the universe. She is aware that cosmologists have developed non-design theories about the origins of the universe. However, she's confident that the non-design thesis is false.
So, according to the simple deist: deism is true, and the non-design thesis (which we’ll call “adeism”) is false. Deism and adeism form a partition of her possibility space.
The simple adeist, on the other hand, is confident that deism is false. She's confident that the universe came about without any help from a designer at all, and that the non-design thesis is true.
So, according to the simple adeist : deism is false, and the non-design hypothesis is true. Deism, and adeism form a partition of her possibility space.
It turns out: There is a deisgner! (So deism is true, adeism is false). Who is more accurate? The deist, obviously.
The Brier Score straightforwardly confirms this -- the simple deist is more accurate, according to the Brier Score, than the simple adeist.
2. A Problem Case
Now, consider again two people contemplating the origin of the universe. Both of them are admittedly somewhat uncertain about the existence of a designer. Both of them are aware of a large number of non-design theories of the origin of the universe.
The sophisticated deist is more confident in deism than adeism. She has, moreover, also carefully considered all of the available non-design hypotheses, and has concluded that only one of them could possibly be true. The rest, she thinks, are non-starters.
The sophisticated adeist is, on the other hand, more confident in adeism than deism. She has also carefully considered all of the available non-design hypotheses, and although she thinks it’s likely that one of them is true, she has no opinions concerning which is the true one. In her estimation, the non-design hypotheses are all equally likely.
Now, suppose it turns out: Deism is true (and so every non-design hypothesis is false). Who is more accurate?
We think: the sophisticated deist! After all the sophisticated deist has the following two advantages over the sophisticated adeist: she has a higher credence in the truth than the sophisticated adeist does, and she has less credence invested in falsehoods than the sophisticated adeist does. So what advantage does the sophisticated adeist have over the sophisticated deist? The only remaining difference between them is the way in which they distribute their confidence among the false hypotheses. But why should the way in which the adeist distributes her confidence among the various false hypotheses make her more accurate in a world in which deism is true?
From the first personal side of things: if I want to have an accurate attitude about the origin of the universe and my choices are between being a sophisticated deist or a sophisticated adeist, I’d prefer to be the sophisticated deist, in the world in which deism is true.
But, in certain situations, and given enough non-design theories, the Brier score delivers the opposite verdict. For example, let D be the design hypothesis, and suppose there are 58 non-design theories, T1, T2, ... T58. Thus our partition is
By the description of the case, the ideal credences, across this partition, are (1,0,0,0,0...0)
Suppose the sophisticated deist’s credences are
Then her Brier Score is
Suppose the sophisticated adeist’s credences are
Then her Brier Score is
That’s a small victory for the adeist, admittedly, but the point is a structural one. The adeist -- in spite of the fact that she is a good deal less confident in the truth and, overall, a good deal more confident in the false -- is more accurate than the deist, according to the Brier. (For a related point -- one which trades on this same structural feature of the Brier Score -- see Knab, “In Defense of Absolute Value.”)
That, we think, is enough of a puzzle to put some pressure on the Brier understanding of epistemic accuracy. More generally, the Brier Score fails to satisfy what looks like a plausible desideratum:
Falsity Distributions Don’t Matter: For any partition of theories: T1...Tn, a probabilistic agent’s accuracy with respect to this partition at world w should be determined solely by the amount of credence she invests in the true theory at w, and the amount of credence she invests in false theories at w. The way she distributes her credences amongst the false theories at w shouldn’t affect her accuracy.