Paper #2: The Contest That Changed Everything
Before 2012, teaching a computer to recognize objects meant designing the clues by hand. Then a deep neural network won the hardest contest in the field, and not by a little.
Paper #2 · 2012
AlexNet
ImageNet Classification with Deep Convolutional Neural Networks
The paper that proved deep CNNs work at scale and set off the deep-learning boom.
Start readingPaper #2: The Architecture (Everything from Paper #1, Scaled Up)
AlexNet didn't invent anything new in its shape. It's the convolutional network from Paper #1, made big: five convolutional layers, three fully-connected ones, and sixty million weights.
Paper #2: ReLU, the Small Change That Made Big Networks Trainable
AlexNet's shape is familiar from Paper #1. What we skipped is how the activation function controls training speed. Picking a fast one, ReLU, is the first thing that made training a network as big as AlexNet possible.
Paper #2: Splitting One Network Across Two Graphics Cards
ReLU made training fast enough. But the network was too big to fit on a single graphics card, so the authors did something unusual: they cut it in half.
Paper #2: Fighting Overfitting at Scale
With sixty million weights and only a million images, AlexNet could have memorized its way to a useless answer. Two defenses stopped it: more data, made for free, and dropout.
Paper #2: The Finer Points (Normalization, Pooling, and the Training Recipe)
The big ideas are in place. This post gathers the smaller tricks AlexNet used, including one now considered obsolete, and the plain recipe for actually training it.
Paper #2: What AlexNet Learned, and Why It Mattered
With AlexNet trained, this last post looks at what the network learned, and why this paper mattered.