The only symmetric inaccuracy measure is the Brier score

If you'd like a PDF of this post, see here. 

[UPDATE 1: I should have made this clear in the original post. The Normality condition makes the proof go through more easily, but it isn't really necessary. Suppose we simply assume instead that $$\mathfrak{I}(w^i, j)= \left \{ \begin{array}{ll} b & \mbox{if } i \neq j \\ a & \mbox{if } i = j \end{array} \right.$$Then we can show that, if $\mathfrak{I}$ is symmetric then, for any probabilistic credence function $p$ and any world $w_i$,$$\mathfrak{I}(p, i) = (b-a)\frac{1}{2} \left (1 - 2p_i + \sum_j p^2_j \right ) + a$$End Update 1.]

[UPDATE 2: There's something puzzling about the result below. Suppose $\mathcal{W} = \{w_1, \ldots, w_n\}$ is the set of possible worlds. And suppose $\mathcal{F}$ is the full algebra of propositions built out of those worlds. That is, $\mathcal{F}$ is the set of subsets of $\mathcal{W}$. Then there are two versions of the Brier score over a probabilistic credence function $p$ defined on $\mathcal{F}$. The first considers only the credences that $p$ assigns to the possible worlds. Thus,$$\mathfrak{B}(p, i) = \sum^n_{j=1} (w^i_j - p_j)^2 = 1 - 2p_i + \sum_j p^2_j$$But there is another that considers also the credences that $p$ assigns to the other propositions in $\mathcal{F}$. Thus,$$\mathfrak{B}^\star(p, i) = \sum_{X \in \mathcal{F}} (w_i(X) - p(X))^2$$Now, at first sight, these look related, but not very closely. However, notice that both are symmetric. Thus, by the extension of Selten's theorem below (plus update 1 above), if $\mathfrak{I}(w^i, j) = b$ for $i \neq j$ and 0 for $i = j$, then $\mathfrak{I}(p, i) = \frac{1}{2}b\mathfrak{B}(p, i)$. Now, $\mathfrak{B}(w^i, j) = 2$ for $i \neq j$, and $\mathfrak{B}(w^i, j) = 0$ for $i = j$, and so this checks out. But what about $\mathfrak{B}^\star$? Well, according to our extension of Selten's theorem, since $\mathfrak{B}^\star$ is symmetric, we can see that it is just a multiple of $\mathfrak{B}$, the factor determined by $\mathfrak{B}^\star(w^i, j)$. So what is this number? Well, it turns out that, if $i \neq j$, then$$\mathfrak{B}^\star(w^i, j) = 2\sum^{n-2}_{k=0} {n-2 \choose k}$$Thus, it follows that$$\mathfrak{B}^\star(p, i) = \sum^{n-2}_{k=0} {n-2 \choose k}\mathfrak{B}(p, i)$$And you can verify this by other means as well. This is quite a nice result independently of all this stuff about symmetry. After all, there doesn't seem any particular reason to favour $\mathfrak{B}$ over $\mathfrak{B}^\star$ or vice versa. This result shows that using one for the sorts of purposes we have in accuracy-first epistemology won't give different results from using the other. End update 2.]

So, as is probably obvious, I've been trying recently to find out what things look like in accuracy-first epistemology if you drop the assumption that the inaccuracy of a whole credal state is the sum of the inaccuracies of the individual credences that it comprises --- this assumption is sometimes called Additivity or Separability. In this post, I want to think about a result concerning additive inaccuracy measures that intrigued me in the past and on the basis of which I tried to mount an argument in favour of the Brier score. The result dates back to Reinhard Selten, the German economist who shared the 1994 Nobel prize with John Harsanyi and John Nash for his contributions to game theory. In this post, I'll show that the result goes through even if we don't assume additivity.

Suppose $\mathfrak{I}$ is an inaccuracy measure. Thus, if $c$ is a credence function defined on the full algebra built over the possible worlds $w_1, \ldots, w_n$, then $\mathfrak{I}(c, i)$ measures the inaccuracy of $c$ at world $w_i$. Then define the following function on pairs of probabilistic credence functions:$$\mathfrak{D}_\mathfrak{I}(p, q) = \sum_i p_i \mathfrak{I}(q, i) - \sum_i p_i\mathfrak{I}(p, i)$$$\mathfrak{D}_\mathfrak{I}$ measures how much more inaccurate $p$ expects $q$ to be than it expects itself to be; equivalently, how much more accurate $p$ expects itself to be than it expects $q$ to be. Now, if $\mathfrak{I}$ is strictly proper, $\mathfrak{D}_\mathfrak{I}$ is positive whenever $p$ and $q$ are different, and zero when they are the same, so in that case $\mathfrak{D}_\mathfrak{I}$ is a divergence. But we won't be assuming that here -- rather remarkably, we don't need to.

Now, it's not hard to see that $\mathfrak{D}_\mathfrak{I}$ is not necessarily symmetric. For instance, consider the log score$$\mathfrak{L}(p, i) = -\log p_i$$Then$$\mathfrak{D}_\mathfrak{L}(p, q) = p_i \log \frac{p_i}{q_i}$$This is the so-called Kullback-Leibler divergence and it is not symmetric. Nonetheless, it's equally easy to see that it is at least possible for $\mathfrak{D}_\mathfrak{I}$ to be symmetric. For instance, consider the Brier score$$\mathfrak{B}(p, i) = 1-2p_i + \sum_j p^2_j$$Then$$\mathfrak{D}_\mathfrak{B}(p, q) = \sum_i (p_i - q_i)^2$$So the natural question arises: how many inaccuracy measures are symmetric in this way? That is, how many generate symmetric divergences in the way that the Brier score does? It turns out: none, except the Brier score.

First, a quick bit of notation: Given a possible world $w_i$, we write $w^i$ for the probabilistic credence function that assigns credence 1 to world $w_i$ and 0 to any world $w_j$ with $j \neq i$. 

And two definitions:

Definition (Normal inaccuracy measure) An inaccuracy measure $\mathfrak{I}$ is normal if $$\mathfrak{I}(w^i, j) = \left \{ \begin{array}{ll} 1 & \mbox{if } i \neq j \\ 0 & \mbox{if } i = j \end{array} \right.$$

Definition (Symmetric inaccuracy measure) An inaccuracy measure is symmetric if $$\mathfrak{D}_\mathfrak{I}(p, q) = \mathfrak{D}_\mathfrak{I}(q, p)$$for all probabilistic credence functions $p$ and $q$.

Thus,  $\mathfrak{I}$ is symmetric if, for any probability functions $p$ and $q$, the loss of accuracy that $p$ expects to suffer by moving to $q$ is the same as the loss of accuracy that $q$ expects to suffer by moving to $p$.

Theorem The only normal and symmetric inaccuracy measure agrees with the Brier score for probabilistic credence functions.

Proof. (This just adapts Selten's proof in exactly the way you'd expect.) Suppose $\mathfrak{D}_\mathfrak{I}(p, q) = \mathfrak{D}_\mathfrak{I}(q, p)$ for all probabilistic $p$, $q$. Then, in particular, for any world $w_i$ and any probabilistic $p$,$$\sum_j w^i_j \mathfrak{I}(p, j) - \sum_j w^i_j \mathfrak{I}(w^i, j) = \sum_j p_j \mathfrak{I}(w^i, j) -\sum_j p_j \mathfrak{I}(p, j)$$So,$$\mathfrak{I}(p, i) = (1-p_i) - \sum_j p_j \mathfrak{I}(p, j)$$So,$$\sum_j p_j \mathfrak{I}(p, j) = 1 - \sum_j p^2_j- \sum_j p_j \mathfrak{I}(p, j)$$So,$$\sum_j p_j \mathfrak{I}(p, j) = \frac{1}{2}[1 - \sum_j p^2_j]$$So,$$\mathfrak{I}(p, i) = 1-p_i -\frac{1}{2}[1 - \sum_j p^2_j] = \frac{1}{2} \left (1 - 2p_i + \sum_j p^2_j \right )$$as required. $\Box$

There are a number of notable features of this result:

First, the theorem does not assume that the inaccuracy measure is strictly proper, but since the Brier score is strictly proper, it follows that symmetry entails strict propriety.

Second, the theorem does not assume additivity, but since the Brier score is additive, it follows that symmetry entails additivity.