Paper #2: What AlexNet Learned, and Why It Mattered

Where we are: over the last few posts we discussed AlexNet piece by piece, from the architecture to the training recipe. This final post asks what the trained network learned, and why it mattered so much.

What the first layer learned

When the authors drew the 96 filters in the first convolutional layer as little images, they saw something familiar: edge detectors at various angles, and small blobs of color. These are the same kinds of patterns Paper #1 found when we visualized a network’s filters, oriented edges and simple color and texture detectors. A first-layer filter is tiny, just an 11-by-11 patch of weights, yet drawn out, each one clearly resembles an edge at a particular angle or a patch of a single color.

Nobody told the network to look for edges; it found, on its own, that edges are a useful first thing to measure in an image. The AlexNet paper only shows this first layer. The rest of the hierarchy Paper #1 described, edges building into textures and textures into shapes, was shown about a year later, when other researchers found ways to look inside the deeper layers.

The two halves specialized

Splitting the network across two GPUs had an odd side effect, and it shows up here. One GPU’s first-layer filters came out largely color-blind, tuned to edges and shapes; the other’s came out largely tuned to color. The authors did not plan this. It happened on its own, and it happened on every training run, no matter how the weights were first randomized.

The authors set up the conditions, two halves that rarely shared information, and training filled in the details in a way nobody had planned.

Similar meaning, not similar pixels

A third result is about the last hidden layer, the 4,096 numbers just before the final decision. Treat those numbers as a fingerprint for an image. When you compare fingerprints, images that are close together turn out to be similar in meaning: different elephants in different poses, lit differently, sit near each other, even though their raw pixels look nothing alike.

This is what Paper #1’s nearest-neighbor post said was missing. Comparing raw pixels failed because pixel similarity is not meaning similarity. AlexNet learned to turn an image into a fingerprint where closeness does mean “same kind of thing.” Those fingerprints are also why a trained network makes such a good starting point for other tasks, the transfer learning Paper #1 ended on: features general enough to reuse.

Why it mattered

A single deep network, trained on raw pixels with no hand-designed clues, beat the carefully engineered systems that had led computer vision for years, by about eleven points.

Before 2012, the strongest results in vision came from human-designed feature detectors, refined over years of research. After 2012, the winning approach was to let the network learn its own. The message the field took from the result was this: with enough data, enough compute, and the right handful of techniques, a deep network could learn better features than people could design by hand. Within a few years, nearly all of computer vision had switched to this approach, and the same recipe spread far beyond images.

That is why AlexNet comes second on this reading list. It was the clearest early evidence that a deep network, scaled up and trained carefully, could beat hand-designed vision systems, and much of what Paper #1 teaches as fundamentals today traces back to papers like it. The next paper, ResNet, picks up a problem AlexNet left open: how to make networks much deeper without training breaking down.