Abstract
The lecture took up the optimization problem in the special case of neural networks. It described the phenomenon of over-parameterization, which seems to help generalization, as observed with " double descents " curves as the number of parameters increases. We then turned to the problem of function approximation and the curse of dimensionality. This began with a reminder of the approximation bounds for Lipchitzian functions, which are pessimistic in high dimension. Separability is a form of regularity that reduces the dimension of the approximation space by reducing the interactions between variables. This can considerably improve approximation speeds. This is the basis of the SIFT and MFCC descriptors used for image and sound recognition. Separability can also be achieved across scales, which can be implemented with wavelet transforms.
Symmetries known a priori are another important source of regularity. A symmetry is an operator that transforms the data while preserving the values of the function to be approximated. All symmetries together define a group. The lecture recalled the definition of a group. The reduction of the approximation dimension is achieved by observing that the data space can be quotient by the symmetry group, which defines invariants.
