Energetic Variational Inference

In this talk, I plan to highlight one component of my on-going research, energetic variational inference. Variational Inference (VI) is an important research area in the field of machine learning. Many VI approaches have been developed and widely used in machine learning and other related areas. We propose a new VI framework, called energetic variational inference, or EVI. It minimizes the VI objective function based on a prescribed energy-dissipation law. Under the EVI, we can recover many existing Particle-based Variational Inference (ParVI) methods, including the popular Stein variational gradient descent (SVGD). More importantly, many new ParVI schemes can be created under this framework. To demonstrate how to develop a new EVI method, we create a new particle-based EVI approach which performs the particle-based approximation of the density first and then uses the approximated density in the variational procedure, or “Approximation-then-Variation” for short. Thanks to this order of approximation and variation, the new scheme can maintain the variational structure at the particle level. Different divergence measures can be combined with EVI to produce different ParVI algorithms. Specially, we demonstrate the EVI methods using KL-divergence and Maximum Mean Discrepancy (MMD). The proposed methods are compared with existing ones using different examples. I conclude the talk with discussions of future research topics. If time permits, I will briefly show some of my current research projects on other directions, including experimental design for controlled experiments, uncertainty qualification, and statistical modeling for complex data, etc.

Share this Event