Paper #2: Splitting One Network Across Two Graphics Cards
ReLU made training fast enough. But the network was too big to fit on a single graphics card, so the authors did something unusual: they cut it in half.
Archive
All posts
45 posts, newest first. Page 3 of 5.
Paper #2: Fighting Overfitting at Scale
With sixty million weights and only a million images, AlexNet could have memorized its way to a useless answer. Two defenses stopped it: more data, made for free, and dropout.
Paper #2: The Finer Points (Normalization, Pooling, and the Training Recipe)
The big ideas are in place. This post gathers the smaller tricks AlexNet used, including one now considered obsolete, and the plain recipe for actually training it.
Paper #2: What AlexNet Learned, and Why It Mattered
With AlexNet trained, this last post looks at what the network learned, and why this paper mattered.
Paper #3: Deeper Networks That Did Worse
After AlexNet, adding layers kept helping, up to a point. Past about twenty layers, plain networks started scoring worse, even on the images they were trained on. ResNet starts from that puzzle.
Paper #3: Not a Fading Signal (Batch Normalization)
The usual suspect when deep networks fail is a learning signal that fades as it travels back through the layers. By 2015 a technique called batch normalization had largely dealt with that, so ResNet's problem had to be something else.
Paper #3: Learning the Change Instead of the Whole Thing
ResNet's fix is one addition: let each block of layers learn only a change to its input, and add the input back at the end. Then doing nothing becomes easy.
Paper #3: The Evidence (18 Layers vs. 34)
The same two networks from the first post, rebuilt with shortcuts. Without them, going from 18 to 34 layers made error worse. With them, it made error better by about three points.
Paper #3: Going to 152 Layers (The Bottleneck Block)
To go past 34 layers without the cost growing too fast, ResNet changed the shape of its blocks: shrink the data, do the main work, then expand it again. ResNet-152 ended up cheaper to run than the 19-layer VGG.
Paper #3: What ResNet Won, and Where It Stopped
ResNet won the 2015 ImageNet contest with 3.57% top-5 error. On a small dataset, a 1,202-layer version did worse than a 110-layer one, and the paper says why.