Paper #4: 1,001 Layers, and Why It Helped
With the original blocks, a 1,001-layer network did worse than a 110-layer one. With pre-activation blocks, it reached 4.92% error on CIFAR-10, the best in the paper. The authors give two reasons.
Tag
identity mappings
Paper #4: Moving the Activation (Pre-activation)
With the shortcut left alone, one step still sat on the path between blocks: the ReLU after the addition. Moving batch normalization and ReLU to before each weight layer removed it, and error went down.
Paper #4: Leave the Shortcut Alone
The authors tried scaling, gating, a 1×1 convolution, and dropout on the shortcuts of a 110-layer ResNet. Every one did worse than a shortcut that passes its input through unchanged.
Paper #4: A Path Straight Through
If the shortcut and the step after the addition both do nothing, any block's output is an earlier block's output plus every change in between. Going backward, the same fact gives the correction a route that does not fade.
Paper #4: The Question ResNet Left Open
ResNet showed that shortcuts work but offered only a hypothesis about why. A year later, the same authors followed the signal through the shortcuts, and found one step in the original block that got in the way.