Paper #4: 1,001 Layers, and Why It Helped
With the original blocks, a 1,001-layer network did worse than a 110-layer one. With pre-activation blocks, it reached 4.92% error on CIFAR-10, the best in the paper. The authors give two reasons.
Tag
With the original blocks, a 1,001-layer network did worse than a 110-layer one. With pre-activation blocks, it reached 4.92% error on CIFAR-10, the best in the paper. The authors give two reasons.
With the shortcut left alone, one step still sat on the path between blocks: the ReLU after the addition. Moving batch normalization and ReLU to before each weight layer removed it, and error went down.