The challenge now is to develop a more efficient algorithm for estimating model parameters.
We begin by reviewing the properties of the Kullback-Leibler divergence and the Pinsker inequality that relates it to total variation. An exponential family has a Gibbs energy that is linear with respect to the parameters. We demonstrate that the Kullback-Leiber divergence is a convex function of the parameters and that we can therefore calculate its minimum by gradient descent. However, the term coming from the partition function (normalization constant) takes a long time to calculate.
The score matching algorithm avoids calculating the partition function by minimizing the relative Fisher information. This depends on the score, which is defined as the gradient of the log-probability. It is the gradient that eliminates the normalization constant. We demonstrate Hyvarinen's theorem, which gives an explicit formula for Fisher's relative information, easier to minimize. We also demonstrate the consistency of parameter estimation by minimizing Fisher's relative information, and that the error asymptotically follows a normal distribution. The relative Fisher information gives a bound for the Kullback-Leibler divergence, which is multiplied by a Log-Sobolev constant. This constant can be very large when optimizing Gibbs energies that have deep local minima. In this case, although much faster to calculate, parametric estimation by Fisher information minimization can be significantly more imprecise than by Kullback-Leibler divergence minimization.
