Part of Advances in Neural Information Processing Systems 4 (NIPS 1991)
David MacKay
The Bayesian model comparison framework is reviewed, and the Bayesian Occam's razor is explained. This framework can be applied to feedforward networks, making possible (1) objective comparisons between solutions using alternative network architectures; (2) objective choice of magnitude and type of weight decay terms; (3) quantified estimates of the error bars on network parameters and on network output. The framework also gen(cid:173) erates a measure of the effective number of parameters determined by the data. The relationship of Bayesian model comparison to recent work on pre(cid:173) diction of generalisation ability (Guyon et al., 1992, Moody, 1992) is dis(cid:173) cussed.
1 BAYESIAN INFERENCE AND OCCAM'S RAZOR
In science, a central task is to develop and compare models to account for the data that are gathered. Typically, two levels of inference are involved in the task of data modelling. At the first level of inference, we assume that one of the models that we invented is true, and we fit that model to the data. Typically a model includes some free parameters; fitting the model to the data involves inferring what values those parameters should probably take, given the data. This is repeated for each model. The second level of inference is the task of model comparison. Here,
·Current address: Darwin College, Cambridge CB3 9EU, U.K.