Paper #4 · 2016
Identity Mappings in Deep Residual Networks
A closer look at why skip connections work as well as they do.
Start readingResNet showed that shortcuts work but offered only a hypothesis about why. A year later, the same authors followed the signal through the shortcuts, and found one step in the original block that got in the way.
If the shortcut and the step after the addition both do nothing, any block's output is an earlier block's output plus every change in between. Going backward, the same fact gives the correction a route that does not fade.
The authors tried scaling, gating, a 1×1 convolution, and dropout on the shortcuts of a 110-layer ResNet. Every one did worse than a shortcut that passes its input through unchanged.
With the shortcut left alone, one step still sat on the path between blocks: the ReLU after the addition. Moving batch normalization and ReLU to before each weight layer removed it, and error went down.
With the original blocks, a 1,001-layer network did worse than a 110-layer one. With pre-activation blocks, it reached 4.92% error on CIFAR-10, the best in the paper. The authors give two reasons.