A statistical mechanical study of Boltzmann machines

Hinton and Sejnowski (1983) have described recently a novel statistical mechanical system which they named the Boltzmann machine. The interesting property of Boltzmann machines is that they can learn to recognise the structure in a set of patterns simply by being shown an example subset of patterns. In this paper some numerical simulations of Boltzmann machines are reported. It is found that the annealing schedule proposed by Ackley, Hinton and Sejnowski (1985) is adequate to obtain a Boltzmann distribution of states, on which the key part of the algorithm depends, but it is clear that the algorithm will require massive computations for large networks. It is also found that there is a window of annealing temperatures at which learning is possible, and the sensitivity of the learning rate to temperature can be understood in terms of the density of states at low energies. Direct calculations of the partition function in small instances of Boltzmann machines are used to characterise the number of states which are thermally accessible for particular annealing schedules. Finally, since Boltzmann machines bear some resemblance to models of disordered magnetic systems, a comparison is made with results for the Sherrington-Kirkpatrick spin-glass model. Both systems support multiple metastable states (i.e. stable with respect to single spin flips), but, in contrast to the SK spin glass, Boltzmann machines exhibit a random distribution of low-energy states in terms of Hamming distance.