Aggregating abstaining experts

In a series of posts a few months ago (here, here, and here), I explored a particular method by which we might aggregate expert credences when those credences are incoherent. The result was this paper, which is now forthcoming in Synthese. The method in question was called the coherent approximation principle (CAP), and it was introduced by Daniel Osherson and Moshe Vardi in this 2006 paper. CAP is based on what we might call the principle of minimal mutilation. We begin with a collection of credence functions, $c_1$, ..., $c_n$, one for each expert, and some of which might be incoherent. What we want at the end is a single coherent credence function $c$ that is the aggregate of $c_1$, ..., $c_n$. The principle of minimal mutilation says that $c$ should be as close as possible to the $c_i$s -- when aggregating a collection of credence functions, you should change them as little as possible to obtain your aggregate.

We can spell this out more precisely by introducing a divergence $\mathfrak{D}$. We might think of this as a measure of how far one credence function lies from another. Thus, $\mathfrak{D}(c,c^{\prime })$ measures the distance from $c$ to $c^{\prime }$. We call these measures divergences rather than distances or metrics, since they do not have the usual features that mathematicians assume of a metric: we assume $\mathfrak{D}(c,c^{\prime })\ge 0$, for any $c,c^{\prime }$, and $\mathfrak{D}(c,c^{\prime })=0$ iff $c=c^{\prime }$, but we do not assume that $\mathfrak{D}$ is symmetric nor that it satisfies the triangle inequality. In particular, we assume that $\mathfrak{D}$ is an additive Bregman divergence. The standard example of an additive Bregman divergence is squared Euclidean distance: if $c$, $c^{\prime }$ are both defined on the set of propositions $F$, then
$$\mathrm{SED}(c,c^{\prime })=\sum _{X\in F}|c(X)-c^{\prime }(X){|}^2$$In fact, $\mathrm{SED}$ is symmetric, but it does not satisfy the triangle inequality. The details of this family of divergences needn't detain us here (but see here and here for more). Indeed, we will simply use $\mathrm{SED}$ throughout. But a more general treatment would look at other additive Bregman divergences, and I hope to do this soon.

Now, suppose $c_1$, ..., $c_n$ is a set of expert credence functions. And suppose $c_i$ is defined on the set of propositions $F_i$. And suppose that $\mathfrak{D}$ is an additive Bregman divergence -- you might take it to be $\mathrm{SED}$. Then how do we define the aggregate $c$ that is obtained from $c_1$, ..., $c_n$ by a minimal mutilation? We let $c$ be the coherent credence function such that the sum of the distances from $c$ to the $c_i$s is minimal. That is,
$${\mathrm{CAP}}_{\mathfrak{D}}(c_1,\dots ,c_n)={\arg \min }_{c\in P_{F_i}}\sum _{i=1}^n\mathfrak{D}(c,c_i)$$
where $P_{F_i}$ is the set of coherent credence functions over $F_i$.

As we see in my paper linked above, if each of the credence functions are defined over the same set of propositions -- that is, if $F_i=F_j$, for all $1\le i,j,\le n$ -- then:

• if $\mathfrak{D}$ is squared Euclidean distance, then this aggregate is the straight linear pool of the original credences; if $c$ is defined on the partition $X_1$, ..., $X_m$, then the straight linear pool of $c_1$, ..., $c_n$ is this:$$c(X_j)=\frac{1}{n}{c}_1(X_j)+...+\frac{1}{n}{c}_n(X_j)$$
• if $\mathfrak{D}$ is the generalized Kullback-Leibler divergence, then the aggregate is the straight geometric pool of the originals; if $c$ is defined on the partition $X_1$, ..., $X_m$, then the straight geometric pool of $c_1$, ..., $c_n$ is this: $$c(X_j)=\frac{1}{K}(c_1(X_j{)}^{\frac{1}{n}}\times ...\times c_1(X_j{)}^{\frac{1}{n}})$$where $K$ is a normalizing factor.

(For more on these types of aggregation, see here and here).

In this post, I'm interested in cases where our agents have credences in different sets of propositions. For instance, the first agent has credences concerning the rainfall in Bristol tomorrow and the rainfall in Bath, but the second has credences concerning the rainfall in Bristol and the rainfall in Birmingham.

I want to begin by pointing to a shortcoming of CAP when it is applied to such cases. It fails to satisfy what we might think of as a basic desideratum of such procedures. To illustrate this desideratum, let's suppose that the three propositions $X_1$, $X_2$, and $X_3$ form a partition. And suppose that Amira has credences in $X_1$, $X_2$, and $X_3$, while Benito has credences only in $X_1$ and $X_2$. In particular:

• Amira's credence function is: $c_A(X_1)=0.3$, $c_A(X_2)=0.6$, $c_A(X_3)=0.1$.
• Benito's credence function is: $c_B(X_1)=0.2$, $c_B(X_2)=0.6$.

Now, notice that, while Amira's credence function is defined on the whole partition, Benito's is not. But, nonetheless, Benito's credences uniquely determine a coherent credence function on the whole partition:

• Benito's extended credence function is: $c_B^{\ast }(X_1)=0.2$, $c_B^{\ast }(X_2)=0.6$, $c_B^{\ast }(X_3)=0.2$.

Thus, we might expect our aggregation procedure to give the same result whether we aggregate Amira's credence function with Benito's or with Benito's extended credence function. That is, we might expect the same result whether we aggregate $c_A$ with $c_B$ or with $c_B^{\ast }$. After all, $c_B^{\ast }$ is in some sense implicit in $c_B$. An agent with credence function $c_B$ is committed to the credences assigned by credence function $c_B^{\ast }$.

However, CAP does not do this. As mentioned above, if you aggregate $c_A$ and $c_B^{\ast }$ using $\mathrm{SED}$, then the result is their linear pool: $\frac{1}{2}{c}_A+\frac{1}{2}{c}_B^{\ast }$. Thus, the aggregate credence in $X_1$ is $0.25$; in $X_2$ it is $0.6$; and in $X_3$ it is $0.15$. The result is different if you aggregate $c_A$ and $c_B$ using $SED$: the aggregate credence in $X_1$ is $0.2625$; in $X_2$ it is $0.6125$; in $X_3$ it is $0.125$.

Now, it is natural to think that the problem arises here because Amira's credences are getting too much say in how far a potential aggregate lies from the agents, since she has credences in three propositions, while Benito only has credences in two. And, sure enough, ${\mathrm{CAP}}_{\mathrm{SED}}(c_A,c_B)$ lies closer to $c_A$ than to $c_B$ and closer to $c_A$ than the aggregate of $c_A$ and $c_B^{\ast }$ lies. And it is equally natural to try to solve this potential bias in favour of the agent with more credences by normalising. That is, we might define a new version of CAP:
$${\mathrm{CAP}}_D^{+}(c_1,\dots ,c_n)={\arg \min }_{c^{\prime }\in P_{F_i}}\sum _{i=1}^n\frac{1}{|F_i|}D(c,c_i)$$
However, this doesn't help. Using this definition, the aggregate of Amira's credence function $c_A$ and Benito's extended credence function $c_B^{\ast }$ remains the same; but the aggregate of Amira's credence function and Benito's original credence function changes -- the aggregate credence in $X_1$ is $0.25333$; in $X_2$, it is $0.61333$; in $X_3$, it is $0.1333$. Again, the two ways of aggregating disagree.

So here is our desideratum in general:

Agreement with Coherent Commitments (ACC) Suppose $c_1$, ..., $c_n$ are coherent credence functions, with $c_i$ defined on $F_i$, for each $1\le i\le n$. And let $F=\bigcup _{i=1}^n{F}_i$. Now suppose that, for each $c_i$ defined on $F_i$, there is a unique coherent credence function $c_i^{\ast }$ defined on $F$ that extends $c_i$ -- that is, $c_i(X)=c_i^{\ast }(X)$ for all $X$ in $F_i$. Then the aggregate of $c_1$, ..., $c_n$ should be the same as the aggregate of $c_1^{\ast }$, ..., $c_n^{\ast }$.

CAP does not satisfy ACC. Is there a natural aggregation rule that does? Here's a suggestion. Suppose you wish to aggregate a set of credence functions $c_1$, ..., $c_n$, where $c_i$ is defined on $F_i$, as above. Then we proceed as follows.

1. First, let $F=\bigcup _{i=1}^n{F}_i$.
2. Second, for each $1\le i\le n$, let $$c_i^{\ast }=\{c:c\text{ is coherent \& }c\text{ is defined on }F\text{ \& }c(X)=c_i(X)\text{ for all }X\text{ in }F\}$$ That is, while $c_i$ represents a precise credal state defined on $F_i$, $c_i^{\ast }$ represents an imprecise credal state defined on $F$. It is the set of coherent credence functions on $F$ that extend $c_i$. That is, it is the set of coherent credence functions on $F$ that agree with $c_i$ on propositions in $F_i$. Thus, if, like Benito, your coherent credences on $F_i$ uniquely determine your coherent credences on $F$, then $c_i^{\ast }$ is just the singleton that contains that unique extension. But if your credences over $F_i$ do not uniquely determine your coherent credences over $F$, then $c_i^{\ast }$ will contain more coherent credence functions.
3. Finally, we take the aggregate of $c_1$, ..., $c_n$ to be the credence function $c$ that minimizes the total distance from $c$ to the $c_i^{\ast }$s. The problem is that there isn't a single natural definition of the distance from a point to a set of points, even when you have a definition of the distance between individual points. I adopt a very particular measure of such distances here; but it would be interesting to explore the alternative options in greater detail elsewhere. Suppose $c$ is a credence function and $C$ is a set of credence functions. Then $$D(c,C)=\frac{{\min }_{c^{\prime }\in C}D(c,c^{\prime })+{\max }_{c^{\prime }\in C}D(c,c^{\prime })}{2}$$ With this in hand, we can finally give our aggregation procedure:$${\mathrm{CAP}}_D^{\ast }(c_1,\dots ,c_n)={\arg \min }_{c^{\prime }\in P_F}\sum _{i=1}^{n}D(c,c_i^{\ast })$$ 

The first thing to note about CAP${}^{\ast }$ is that, unlike the original CAP, or CAP${}^{+}$, it automatically satisfies ACC.

Let's now see CAP${}^{\ast }$ in action.

• Since CAP${}^{\ast }$ satisfies ACC, the aggregate for $c_A$ and $c_B$ is the same as the aggregate for $c_A$ and $c_B^{\ast }$, which is just their straight linear pool.
• Next, suppose we wish to aggregate Amira with a third agent, Cleo, who has a credence only in $X_1$, which she assigns $0.5$ -- that is, $c_C(X_1)=0.5$. Then $F=\{X_1,X_2,X_3\}$, and  $$c_C^{\ast }=\{c:c(X_1)=0.5,c(X_2)\ge 0.5,c(X_3)=1-c(X_1)-c(X_2)\}$$ So, $${\mathrm{CAP}}_{\mathfrak{D}}^{\ast }(c_A,c_B)={\arg \min }_{c^{\prime }\in P_F}\mathfrak{D}(c^{\prime },c_A)+\mathfrak{D}(c^{\prime },c_C^{\ast })$$Working through the calculation for $\mathfrak{D}=\mathrm{SED}$, we obtain the following aggregate: $c(X_1)=0.4$, $c(X_2)=0.425$, $c(X_3)=0.175$.
• One interesting feature of CAP${}^{\ast }$ is that, unlike CAP, we can apply it to individual agents. Thus, for instance, suppose we wish to take Cleo's single credence in $X_1$ and 'fill in' her credences in $X_2$ and $X_3$. Then we can use CAP${}^{\ast }$ to do this. Her new credence function will be $$c_C^{\prime }={\mathrm{CAP}}_{\mathrm{SED}}^{\ast }(c_C)={\arg \min }_{c^{\prime }\in P_F}D(c^{\prime },c_C)$$ That is, $c_C^{\prime }(X_1)=0.5$, $c_C^{\prime }(X_2)=0.25$, $c_C^{\prime }(X_3)=0.25$. Rather unsurprisingly, $c_C^{\prime }$ is the midpoint of the line formed by the imprecise probabilities $c_C^{\ast }$. Now, notice: the aggregate of Amira and Cleo given above is just the straight linear pool of Amira's credence function $c_A$ and Cleo's 'filled in' credence function $c_C^{\prime }$. I would conjecture that this is generally true: filling in credences using CAP${}_{\mathrm{SED}}^{\ast }$ and then aggregating using straight linear pooling always agrees with aggregating using CAP${}_{\mathrm{SED}}^{\ast }$. And perhaps this generalises beyond SED.