Tuesday, 16 May 2017

The Wisdom of the Crowds: generalizing the Diversity Prediction Theorem

I've just been reading Aidan Lyon's fascinating paper, Collective Wisdom. In it, he mentions a result known as the Diversity Prediction Theorem, which is sometimes taken to explain why crowds are wiser, on average, than the individuals who compose them. The theorem was originally proved by Anders Krogh and Jesper Vedelsby, but it has entered the literature on social epistemology through the work of Scott E. Page. In this post, I'll generalize this result.

The Diversity Prediction Theorem concerns a situation in which a number of different individuals estimate a particular quantity -- in the original example, it is the weight of an ox at a local fair. Take the crowd's estimate of the quantity to be the average of the individual estimates. Then the theorem shows that the distance from the crowd's estimate to the true value is less than the average distance from the individual estimates to the true value; and, moreover, the difference between the two is always given by the average distance from the individual estimates to the crowd's estimate (which you might think of as the variance of the individual estimates).

Let's make this precise. Suppose you have a group of $n$ individuals. They each provide an estimate for a real-valued quantity. The $i^\mathrm{th}$ individual gives the prediction $q_i$. The true value of this quantity is $\tau$. And we measure the distance from one estimate of a quantity to another, or to the true value of that quantity, using squared error. Then:
  • The crowd's prediction of the quantity is $c = \frac{1}{n}\sum^n_{i=1} q_i$.
  • The crowd's distance from the true quantity is $\mathrm{SqE}(c) = (c-\tau)^2$.
  • $S_i$'s distance from the true quantity is $\mathrm{SqE}(q_i) = (q_i-\tau)^2$
  • The average individual distance from the true quantity is $\frac{1}{n} \sum^n_{i=1} \mathrm{SqE}(q_i) = \frac{1}{n} \sum^n_{i=1} (q_i - \tau)^2$.
  • The average individual distance from the crowd's estimate is $v = \frac{1}{n}\sum^n_{i=1} (q_i - c)^2$.
Given this, we have:

Diversity Prediction Theorem $$\mathrm{SqE}(c) = \frac{1}{n} \sum^n_{i=1} \mathrm{SqE}(q_i) - v$$
The theorem is easy enough to prove. You essentially just follow the algebra. However, following through the proof, you might be forgiven for thinking that the result says more about some quirk of squared error as a measure of distance than about the wisdom of crowds. And of course squared error is just one way of measuring the distance from an estimate of a quantity to the true value of that quantity, or from one estimate of a quantity to another. There are other such distance measures. So the question arises: Does the Diversity Prediction Theorem hold if we replace squared error with one of these alternative measures of distance? In particular, it is natural to take any of the so-called Bregman divergences $\mathfrak{d}$ to be a legitimate measure of distance from one estimate to another. I won't say much about Bregman divergences here, except to give their formal definition. To learn about their properties, have a look here and here. They were introduced by Bregman as a natural generalization of squared error.

Definition (Bregman divergence) A function $\mathfrak{d} : [0, \infty) \times [0, \infty) \rightarrow [0, \infty]$ is a Bregman divergence if there is a continuously differentiable, strictly convex function $\varphi : [0, \infty) \rightarrow [0, \infty)$ such that $$\mathfrak{d}(x, y) = \varphi(x) - \varphi(y) - \varphi'(y)(x-y)$$
Squared error is itself one of the Bregman divergences. It is the one generated by $\varphi(x) = x^2$. But there are many others, each generated by a different function $\varphi$.

Now, suppose we measure distance between estimates using a Bregman divergence $\mathfrak{d}$. Then:
  • The crowd's prediction of the quantity is $c = \frac{1}{n}\sum^n_{i=1} j_i$.
  • The crowd's distance from the true quantity is $\mathrm{E}(c) = \mathfrak{d}(c, \tau)$.
  • $S_i$'s distance from the true quantity is $\mathrm{E}(j_i) = \mathfrak{d}(q_i, \tau)$
  • The average individual distance from the true quantity is $\frac{1}{n} \sum^n_{i=1} \mathrm{E}(j_i) = \frac{1}{n} \sum^n_{i=1} \mathfrak{d}(q_i, \tau)$.
  • The average individual distance from the crowd's estimate is $v = \frac{1}{n}\sum^n_{i=1} \mathfrak{d}(q_i, c)$.
 Given this, we have:

Generalized Diversity Prediction Theorem $$\mathrm{E}(c) = \frac{1}{n} \sum^n_{i=1} \mathrm{E}(q_i) - v$$
& & \frac{1}{n} \sum^n_{i=1} \mathrm{E}(q_i) - v \\
& = & \frac{1}{n} \sum^n_{i=1} [ \mathfrak{d}(q_i, \tau) - \mathfrak{d}(q_i, c)] \\
& = & \frac{1}{n} \sum^n_{i=1} [\varphi(q_i) - \varphi(\tau) - \varphi'(\tau)(q_i - \tau)] - [\varphi(q_i) - \varphi(c) - \varphi'(\tau)(q_i - c)] \\
& = & \frac{1}{n} \sum^n_{i=1} [\varphi(q_i)- \varphi(\tau) - \varphi'(\tau)(q_i - \tau) - \varphi(q_i)+ \varphi(c) + \varphi'(\tau)(q_i - c)] \\
& = & - \varphi(\tau) - \varphi'(\tau)((\frac{1}{n} \sum^n_{i=1} q_i) - \tau) + \varphi(c) + \varphi'(\tau)((\frac{1}{n} \sum^n_{i=1} q_i) - c) \\
& = & - \varphi(\tau) - \varphi'(\tau)(c - \tau) + \varphi(c) + \varphi'(\tau)(c - c) \\
& = & \varphi(c) - \varphi(\tau) - \varphi'(\tau)(c - \tau) \\
& = &   \mathfrak{d}(c, \tau) \\
& = & \mathrm{E}(c)
as required.

Thursday, 11 May 2017

Reasoning Club Conference 2017

The Fifth Reasoning Club Conference will take place at the Center for Logic, Language, and Cognition in Turin on May 18-19, 2017.

The Reasoning Club is a network of institutes, centres, departments, and groups addressing research topics connected to reasoning, inference, and methodology broadly construed. It issues the monthly gazette The Reasoner. (Earlier editions of the meeting were held in Brussels, Pisa, Kent, and Manchester.)



Palazzo Badini
via Verdi 10, Torino
Sala Lauree di Psicologia (ground floor)

9:00 | welcome and coffee

9:30 | greetings
           presentation of the new editorship of The Reasoner
           (Hykel HOSNI, Milan)

Morning session – chair: Gustavo CEVOLANI (IMT Lucca)

10:00 | invited talk

Branden FITELSON (Northeastern University, Boston)

Two approaches to belief revision

In this paper, we compare and contrast two methods for the qualitative revision of (viz., full) beliefs. The first (Bayesian) method is generated by a simplistic diachronic Lockean thesis requiring coherence with the agent's posterior credences after conditionalization. The second (Logical) method is the orthodox AGM approach to belief revision. Our primary aim will be to characterize the ways in which these two approaches can disagree with each other — especially in the special case where the agent's belief set is deductively cogent.

(joint work with Ted Shear and Jonathan Weisberg)

11:00 | Ted SHEAR (Queensland) and John QUIGGIN (Queensland)
A modal logic for reasonable belief

11:45 | Nina POTH (Edinburgh) and Peter BRÖSSEL (Bochum)

Bayesian inferences and conceptual spaces: Solving the complex-first paradox

12:30 | lunch break

Afternoon session I – chair: Peter BRÖSSEL (Bochum)

13:30 | invited talk

Katya TENTORI (University of Trento)

Judging forecasting accuracy 
How human intuitions can help improving formal models

Most of the scoring rules that have been discussed and defended in the literature are not ordinally equivalent, with the consequence that, after the very same outcome has materialized, a forecast X can be evaluated as more accurate than Y according to one model but less accurate according to another. A question that naturally arises is therefore which of these models better captures people’s intuitive assessment of forecasting accuracy. To answer this question, we developed a new experimental paradigm for eliciting ordinal judgments of accuracy concerning pairs of forecasts for which various combinations of associations/dissociations between the Quadratic, Logarithmic, and Spherical scoring rules are obtained. We found that, overall, the Logarithmic model is the best predictor of people’s accuracy judgments, but also that there are cases in which these judgments — although they are normatively sound — systematically depart from what is expected by all the models. These results represent an empirical evaluation of the descriptive adequacy of the three most popular scoring rules and offer insights for the development of new formal models that might favour a more natural elicitation of truthful and informative beliefs from human forecasters.

(joint work with Vincenzo Crupi and Andrea Passerini)

14:15 | Catharine SAINT-CROIX (Michigan)

Immodesty and evaluative uncertainty

15:15 | Michael SCHIPPERS (Oldenburg), Jakob KOSCHOLKE (Hamburg)

Against relative overlap measures of coherence

16:00 | coffee break

Afternoon session II – chair: Paolo MAFFEZIOLI (Torino)

16:30 | Simon HEWITT (Leeds)

Frege's theorem in plural logic

17:15 | Lorenzo ROSSI (Salzburg) and Julien MURZI (Salzburg)

Generalized Revenge


Campus Luigi Einaudi
Lungo Dora Siena 100/A
Sala Lauree Rossa
building D1 (ground floor)

9:00 | welcome and coffee

Morning session – chair: Jan SPRENGER (Tilburg)

9:30 | invited talk

Paul EGRÉ (Institut Jean Nicod, Paris)

Logical consequence and ordinary reasoning

The notion of logical consequence has been approached from a variety of angles. Tarski famously proposed a semantic characterization (in terms of truth-preservation), but also a structural characterization (in terms of axiomatic properties including reflexivity, transitivity, monotonicity, and other features). In recent work, E. Chemla, B. Spector and I have proposed a characterization of a wider class of consequence relations than Tarskian relations, which we call "respectable" (Journal of Logic and Computation, forthcoming). The class also includes non-reflexive and nontransitive relations, which can be motivated in relation to ordinary reasoning (such as reasoning with vague predicates, see Zardini 2008, Cobreros et al. 2012, or reasoning with presuppositions, see Strawson 1952, von Fintel 1998, Sharvit 2016). Chemla et al.'s characterization is partly structural, and partly semantic, however. In this talk I will present further advances toward a purely structural characterization of such respectable consequence relations. I will discuss the significance of this research program toward bringing logic closer to ordinary reasoning.

(joint work with Emmanuel Chemla and Benjamin Spector)

10:30 | Niels SKOVGAARD-OLSEN (Freiburg)

Conditionals and multiple norm conflicts

11:15 | Luis ROSA (Munich)

Knowledge grounded on pure reasoning

12:00 | lunch break

Afternoon session I – chair: Steven HALES (Bloomsburg)

13:30 | invited talk

Leah HENDERSON (University of Groningen)

The unity of explanatory virtues

Scientific theory choice is often characterised as an Inference to the Best Explanation (IBE) in which a number of distinct explanatory virtues are combined and traded off against one another. Furthermore, the epistemic significance of each explanatory virtue is often seen as highly case-specific. But are there really so many dimensions to theory choice? By considering how IBE may be situated in a Bayesian framework, I propose a more unified picture of the virtues in scientific theory choice.

14:30 | Benjamin EVA (Munich) and Reuben STERN (Munich)

Causal explanatory power

15:15 | coffee break

Afternoon session II – chair: Jakob KOSCHOLKE (Hamburg)

16:00 | Barbara OSIMANI (Munich)

Bias, random error, and the variety of evidence thesis

16:45 | Felipe ROMERO (Tilburg) and Jan SPRENGER (Tilburg)

Scientific self-correction: The Bayesian way


Gustavo Cevolani (Torino)
Vincenzo Crupi (Torino)
Jason Konek (Kent)
Paolo Maffezioli (Torino)

For any queries please contact Vincenzo Crupi (vincenzo.crupi@unito.it) or Jason Konek (jpkonek@ksu.edu).