Reading
Introduction to the Theory of Neural Computation
Hertz et al.
A rigorous introduction to neural network theory from a statistical-mechanics perspective, treating networks as physical systems with energy landscapes and stable states. It covers Hopfield associative memory, layered-network learning and backpropagation, storage capacity, and unsupervised and competitive learning.
Key takeaways
- Approaches neural networks through statistical mechanics, treating them as physical systems with energy functions and stable states rather than purely as engineering constructs.
- Covers the Hopfield model of associative memory in depth, framing recall as a system relaxing toward stored low-energy attractor states.
- Develops the theory of learning in layered networks, including the perceptron and backpropagation, with rigor about capacity and convergence.
- Analyzes the storage capacity and error behavior of associative memories, quantifying how much a network can reliably remember.
- Treats unsupervised and competitive learning, including self-organizing feature maps, as principled ways for networks to discover structure without labels.
- Reflects the late-1980s to early-1990s theoretical foundations that predate modern deep learning, emphasizing physics-derived intuitions that still underlie current models.
Reflections
This one is a reminder that neural networks didn’t spring from nowhere in the last decade; they have a lineage that runs through physics. Hertz, Krogh, and Palmer come at the subject like statistical mechanics, energy landscapes, attractors, systems settling into stable states, and reading it changes the metaphors you reach for. I like the discipline of the older, more mathematical view because it strips away the anthropomorphic language we’ve layered onto these systems and puts something more honest underneath: not a mind, but a physical system relaxing toward a low-energy configuration. It dates itself in places, and that’s part of the value. Seeing which ideas survived and which didn’t teaches you to hold today’s certainties a little more loosely. The oldest books are often where you find the assumptions everyone else stopped questioning.