Hopf bifurcation and Hopf hopping in recurrent nets

Some aspects of the learning dynamics of recurrent neural networks are discussed. It is shown that as a two-unit fully recurrent network is trained to oscillate, the learning process brings the network to a point where a small change in any one of the weights can push the network through a Hopf bifurcation to create stable oscillation. As learning continues, the network indeed bifurcates to create the stable limit cycle. The limit cycle is soon destroyed and recreated several times, with the limit cycle phase becoming more dominant each time. As a result of this 'Hopf-hopping' phenomenon, it is very difficult to assess how close the network is to learning the desired periodic behavior. Eigenvalue analysis shows that the limit cycles in the later stage are more robust than the limit cycles in the earlier stage. It is shown which weights are more critical in order for the network to maintain periodic behavior.>