Abstract
Jaynes maximum entropy models link statistical physics to data modeling. A probability distribution is specified by moments (expectations of generating functions) maximizing its entropy, which amounts to making explicit the fact that we have no more information. Gibbs' theorem shows that the maximum entropy distribution is an exponential distribution, specified by Lagrange multipliers. In the case of a Gaussian distribution, the Lagrange multipliers are given by the inverse of the covariance matrix.
We demonstrate that the first and second derivatives of the log partition function (cumulant/free energy) of the Gibbs distribution can be used to calculate the expectations and covariances of the generating functions of the Gibbs distribution. We introduce the notion of conjugate duality, which establishes the link between the parameterization of a Gibbs distribution by its Lagrange multipliers and by the moments of the generating functions. This conjugate duality is calculated by the Legendre transform. The maximum entropy depends on the maximum likelihood, which is obtained by the Legendre transform of the free energy (cumulants). Lagrange multipliers can be calculated from the moments of the generating functions by likelihood maximization, using a gradient ascent algorithm.
It is shown that a probability distribution is invariant to the action of a group if and only if the moments of the generating functions are invariant to the action of the group on the data. This reduces the parameterization of the Gibbs distribution. Stationary distributions are invariant to the action of the translation group.
