Paper #3: Deeper Networks That Did Worse
After AlexNet, adding layers kept helping, up to a point. Past about twenty layers, plain networks started scoring worse, even on the images they were trained on. ResNet starts from that puzzle.
Paper #3 · 2015
ResNet
Deep Residual Learning for Image Recognition
Skip connections that let networks go hundreds of layers deep without breaking.
Start readingPaper #3: Not a Fading Signal (Batch Normalization)
The usual suspect when deep networks fail is a learning signal that fades as it travels back through the layers. By 2015 a technique called batch normalization had largely dealt with that, so ResNet's problem had to be something else.
Paper #3: Learning the Change Instead of the Whole Thing
ResNet's fix is one addition: let each block of layers learn only a change to its input, and add the input back at the end. Then doing nothing becomes easy.
Paper #3: The Evidence (18 Layers vs. 34)
The same two networks from the first post, rebuilt with shortcuts. Without them, going from 18 to 34 layers made error worse. With them, it made error better by about three points.
Paper #3: Going to 152 Layers (The Bottleneck Block)
To go past 34 layers without the cost growing too fast, ResNet changed the shape of its blocks: shrink the data, do the main work, then expand it again. ResNet-152 ended up cheaper to run than the 19-layer VGG.
Paper #3: What ResNet Won, and Where It Stopped
ResNet won the 2015 ImageNet contest with 3.57% top-5 error. On a small dataset, a 1,202-layer version did worse than a 110-layer one, and the paper says why.