Introduction to Bayesian reasoning and its applications

Importantly, in Bayesian theory, these probabilities p are no longer interpreted as relative frequencies of events (the " frequentist " viewpoint), but as measures of the degree of subjective knowledge. This meaning is covered by the term " plausibility ", but given the equivalence we have just presented, in what follows we will also continue to use the usual term "probability".

Probability theory, as it is taught, is mainly used to calculate the probability of an observation D, given certain assumptions H about the state of the world. However, since observational data D and hypotheses H play symmetrical roles, nothing prevents us from asking the opposite question: given an observation D, how plausible is the hypothesis H ? The fundamental rule answers this question :

p(H|D) = p(D|H) p(H) / p(D) or p(H|D) α p(D|H) p(H)

In this equation, p(H) is called the a priori probability of H( note that this is not necessarily knowledge independent of experience; on the contrary,a priori knowledge is often the result of previous experience). p(H|D) is the a posteriori probability of H (this term is not necessarily to be taken in its temporal sense, but rather within a logical deduction: having become aware of data D, we revise the plausibility of H). Finally, p(D|H), considered as a function of H, is the likelihood of H.

The Bayes factor that separates two hypotheses or " models " M1 and M2, is a measure of their relative merit, the ratio of their likelihoods : p(D|M1)/p(D|M2). The decimal logarithm of this value(log likelihood ratio), measured in decibels, is an often more intelligible measure. The value log(p/(1-p)), which quantifies the likelihood of a hypothesis relative to all others, is also known as " evidence " ( weight of evidence or, for Turing, log odds). According to Jeffreys, 10 decibels, i.e. a ratio of 10 to 1, constitutes " substantial evidence ", and 20 decibels or more (at least 100 to 1) " decisive evidence ".

The principle of maximum likelihood consists in choosing the model_Mi for which the data are the most probable (p(_D|Mi) is maximum). The principle of maximum a posteriori (or MAP) consists in maximizing p(_Mi |D) - which also takes into account the a priori probabilities of the different_Mi models.

William of Ockham (1285-1347) is credited with a principle of reasoning that has actually been around since antiquity: " a plurality should not be posited without necessity entities should not be multiplied beyond the necessary or, all things being equal, simpler explanations should be preferred to more complex ones. Bayesian reasoning, as an optimal model of the " logic of science " (Jaynes, 2003), automatically includes an Ockham's razor. This is because a smaller likelihood is allocated to the weakest hypotheses, those that make the most diffuse or least specific predictions. What's more, if two models achieve the same likelihood, the one that does so with the most free parameters is automatically penalized by the Bayesian equations. Finally, it is often natural to assign a smaller a priori probability to more complex models. All these factors contribute to Ockham's rule and make Bayesian theory a plausible model of scientific reasoning (Jaynes, 2003).