Introduction to Bayesian reasoning and its applications

Listen to audio

Reverend Thomas Bayes (~1701-1761), pastor of the Presbyterian Church and British mathematician, studied logic and theology at the University of Edinburgh. Various works, including an introduction to differential calculus, led to his election to the Royal Society on November 4, 1742. It was only after Bayes's death in 1761 that his friend Richard Price found in his papers an Essay on how to solve a problem in the doctrine of risks. Published by the Royal Society in 1763, this essay applies the principle of inference that we know today as " Bayes' rule ". Without denying Bayes' inventiveness, it is now generally agreed that his rule is merely a simple application of the product rule in probability theory, already known to Bernoulli and De Moivre, and whose vast range of applications Laplace (1774) was the first to perceive.

What is it all about ? The principles of Bayesian reasoning extend the principles of classical logic with discrete truth values to continuous values of plausibility. In fact, they can be deduced from the axioms that these plausibilities must verify (Jaynes, 2003). Briefly: we assume that (1) degrees of plausibility are represented by real numbers ; (2) these values follow common-sense rules, thus following Laplace's famous formula (" probability theory is basically just common sense reduced to calculus ") ; (3) none of the available data is ignored ; (4) equivalent states of knowledge have the same degree of plausibility. The Cox-Jaynes theorem shows that these rules suffice to define, to within one monotonic function, universal mathematical rules for plausibility p. These are the usual rules of probability and the fundamental rule " de Bayes " :

p(A & B) = p(A|B) p(B) = p(B|A) p(A)

Importantly, in Bayesian theory, these probabilities p are no longer interpreted as relative frequencies of events (the " frequentist " viewpoint), but as measures of the degree of subjective knowledge. This meaning is covered by the term " plausibility ", but given the equivalence we have just presented, in what follows we will also continue to use the usual term "probability".

Probability theory, as it is taught, is mainly used to calculate the probability of an observation D, given certain assumptions H about the state of the world. However, since observational data D and hypotheses H play symmetrical roles, nothing prevents us from asking the opposite question: given an observation D, how plausible is the hypothesis H ? The fundamental rule answers this question :

p(H|D) = p(D|H) p(H) / p(D) or p(H|D) α p(D|H) p(H)

In this equation, p(H) is called the a priori probability of H( note that this is not necessarily knowledge independent of experience; on the contrary,a priori knowledge is often the result of previous experience). p(H|D) is the a posteriori probability of H (this term is not necessarily to be taken in its temporal sense, but rather within a logical deduction: having become aware of data D, we revise the plausibility of H). Finally, p(D|H), considered as a function of H, is the likelihood of H.

The Bayes factor that separates two hypotheses or " models " M1 and M2, is a measure of their relative merit, the ratio of their likelihoods : p(D|M1)/p(D|M2). The decimal logarithm of this value(log likelihood ratio), measured in decibels, is an often more intelligible measure. The value log(p/(1-p)), which quantifies the likelihood of a hypothesis relative to all others, is also known as " evidence " ( weight of evidence or, for Turing, log odds). According to Jeffreys, 10 decibels, i.e. a ratio of 10 to 1, constitutes " substantial evidence ", and 20 decibels or more (at least 100 to 1) " decisive evidence ".

The principle of maximum likelihood consists in choosing the model_Mi for which the data are the most probable (p(_D|Mi) is maximum). The principle of maximum a posteriori (or MAP) consists in maximizing p(_Mi |D) - which also takes into account the a priori probabilities of the different_Mi models.

William of Ockham (1285-1347) is credited with a principle of reasoning that has actually been around since antiquity: " a plurality should not be posited without necessity entities should not be multiplied beyond the necessary or, all things being equal, simpler explanations should be preferred to more complex ones. Bayesian reasoning, as an optimal model of the " logic of science " (Jaynes, 2003), automatically includes an Ockham's razor. This is because a smaller likelihood is allocated to the weakest hypotheses, those that make the most diffuse or least specific predictions. What's more, if two models achieve the same likelihood, the one that does so with the most free parameters is automatically penalized by the Bayesian equations. Finally, it is often natural to assign a smaller a priori probability to more complex models. All these factors contribute to Ockham's rule and make Bayesian theory a plausible model of scientific reasoning (Jaynes, 2003).