Paper #2: The Contest That Changed Everything
Before 2012, teaching a computer to recognize objects meant designing the clues by hand. Then a deep neural network won the hardest contest in the field, and not by a little.
Where we are: In Paper #1 we worked through CS231n, going from raw pixels up to how a trained convolutional network labels images. Now we read AlexNet, the 2012 paper that put those ideas together and won the hardest contest in computer vision.
Why it matters: before AlexNet, people decided what a computer should look for in an image and wrote code to find it. AlexNet learned what to look for from labeled examples, and won the 2012 ImageNet contest with 15.3% top-5 error, against 26.2% for the best hand-designed entry. After that result, the field switched to networks that learn their own features.
Paper #1 was a course: it taught how the pieces work. Paper #2 is a real research paper. It is where those pieces came together and beat every hand-built entry in the 2012 contest, by a far wider margin than any previous year’s improvement.
Before 2012: computer vision done by hand
For decades, from the field’s start in the 1960s until about 2012, getting a computer to recognize objects meant designing the clues by hand. A person decided, ahead of time, what to measure in a picture, and wrote code to measure it.
Say you wanted to find faces. You might write code that looks for two small dark circles side by side and decide those are probably eyes. You, the human, picked that clue. The computer just ran your code and reported what it found. Real systems used more careful hand-designed detectors that measured things like edges at particular angles and sharp corners. A person designed the detectors; the computer never worked out on its own what was worth looking for. Only the last step, a simple classifier that sorted images using those hand-picked measurements, did any learning.
This is the opposite of what we discussed in Paper #1. A convolutional network is not told what to look for. It starts with random filters and, as it trains, those filters slowly change until they detect whatever helps label the image: first simple edges, then textures, then whole shapes. In Paper #1’s post on peeking inside a CNN, we even looked at the filters a trained network had learned and saw edge- and color-detectors the network came up with on its own, not ones anyone wrote code to specify.
The hand-built approach worked, a little. But progress was slow, and the hardest test in the field, a dataset called ImageNet, stumped even the best of these systems.
ImageNet: over a million photos, a thousand categories
ImageNet is a giant collection of labeled photos pulled from the web, millions of them, sorted into categories by human labelers. (The full dataset holds over 15 million images across roughly 22,000 categories, each tagged by people paid small amounts through Amazon’s Mechanical Turk.) Each year, a slice of it powered a contest, the ImageNet Large Scale Visual Recognition Challenge: about 1.2 million training photos spread across 1,000 categories, from “golden retriever” to “container ship.”
A thousand categories is a lot to tell apart. Many look alike (think of all the dog breeds), and the same object shows up in endless variety: different angles, lighting, and backgrounds. That is exactly the set of challenges Paper #1 kept running into, now at full scale.
Scoring it: top-5 error
With 1,000 possible labels, demanding the single exact answer every time would be harsh. So the contest used top-5 error: a guess counts as correct if the right label is among the model’s five most confident answers. Top-5 error is the fraction of test images it still gets wrong even by that generous measure. (The stricter version, top-1 error, counts only whether your single best guess was right.) Lower is better.
Before 2012, the best systems, those hand-built feature pipelines, sat around 26% top-5 error on the contest, and that number was barely moving: the winning error had crept from about 28% in 2010 to about 26% in 2011, roughly two points in a year.
2012: a neural network wins, and not by a little
Then a deep convolutional network, later nicknamed AlexNet after its lead author Alex Krizhevsky, entered the 2012 contest and scored 15.3% top-5 error. The runner-up, still built on hand-designed clues, managed 26.2%, almost exactly where hand-built systems had been stuck for years.
That eleven-point jump was several times any gain the contest had seen before. By 2014, nearly every serious entry in the contest was a deep neural network, and hand-built features stopped winning.
What made it work
AlexNet did not invent the convolutional network. The idea went back to the 1980s. Almost every piece, convolutions, pooling, ReLU, dropout, softmax, was already known, and most of it we covered in Paper #1.
What AlexNet did was put those pieces together, make the network big, and actually train it on all 1.2 million images without it either taking forever or just memorizing them. Training at that scale, without either failure, was the hard part. The rest of this series covers the specific choices that made it possible: a faster activation function, a way to split the model across two graphics cards, and the methods they used to stop the network from memorizing. We start with the one change that, more than any other, made training a network this large feasible at all.