Tag
ResNet showed that shortcuts work but offered only a hypothesis about why. A year later, the same authors followed the signal through the shortcuts, and found one step in the original block that got in the way.
The same two networks from the first post, rebuilt with shortcuts. Without them, going from 18 to 34 layers made error worse. With them, it made error better by about three points.
ResNet's fix is one addition: let each block of layers learn only a change to its input, and add the input back at the end. Then doing nothing becomes easy.