Interpolating between sampling and variational inference with infinite\n stochastic mixtures

Sampling and Variational Inference (VI) are two large families of methods for\napproximate inference that have complementary strengths. Sampling methods excel\nat approximating arbitrary probability distributions, but can be inefficient.\nVI methods are efficient, but may misrepresent the true distribution. Here, we\ndevelop a general framework where approximations are stochastic mixtures of\nsimple component distributions. Both sampling and VI can be seen as special\ncases: in sampling, each mixture component is a delta-function and is chosen\nstochastically, while in standard VI a single component is chosen to minimize\ndivergence. We derive a practical method that interpolates between sampling and\nVI by solving an optimization problem over a mixing distribution. Intermediate\ninference methods then arise by varying a single parameter. Our method provably\nimproves on sampling (reducing variance) and on VI (reducing bias+variance\ndespite increasing variance). We demonstrate our method's bias/variance\ntrade-off in practice on reference problems, and we compare outcomes to\ncommonly used sampling and VI methods. This work takes a step towards a highly\nflexible yet simple family of inference methods that combines the complementary\nstrengths of sampling and VI.\n