Part of Advances in Neural Information Processing Systems 12 (NIPS 1999)
Zoubin Ghahramani, Matthew Beal
We present an algorithm that infers the model structure of a mix(cid:173) ture of factor analysers using an efficient and deterministic varia(cid:173) tional approximation to full Bayesian integration over model pa(cid:173) rameters. This procedure can automatically determine the opti(cid:173) mal number of components and the local dimensionality of each component (Le. the number of factors in each factor analyser) . Alternatively it can be used to infer posterior distributions over number of components and dimensionalities. Since all parameters are integrated out the method is not prone to overfitting. Using a stochastic procedure for adding components it is possible to per(cid:173) form the variational optimisation incrementally and to avoid local maxima. Results show that the method works very well in practice and correctly infers the number and dimensionality of nontrivial synthetic examples. By importance sampling from the variational approximation we show how to obtain unbiased estimates of the true evidence, the exact predictive density, and the KL divergence between the varia(cid:173) tional posterior and the true posterior, not only in this model but for variational approximations in general.