Paper #3: Learning the Change Instead of the Whole Thing

Where we are: deeper plain networks trained worse even though their learning signal was healthy. A deeper network should at least be able to copy a shallower one and pass the rest through unchanged, but training could not find that. This post is ResNet’s answer.

The idea first- a metaphor

Picture a line of artists drawing a cat, each one there to improve the picture a little. Before ResNet, networks worked like a line where each artist looks at the drawing they receive, but has to make their version on a fresh sheet, copying from the one in front of them. To keep what the earlier artists drew, they have to copy all of it exactly, then add their own improvement. ResNet’s change was to hand the drawing itself down the line, so each artist only makes their change on top: adding, erasing, or adjusting. An artist with nothing useful to add just hands it on.

The paper suggests that copying exactly is hard for a plain network’s layers, so in practice their copy of the earlier work comes out a little off. A deeper network has more layers that each have to copy, so it has more chances to distort the earlier work. That fits what the paper measured: the 34-layer plain network did worse than the 18-layer one, even on its training images. ResNet removes the need to copy: the earlier work goes forward on its own, and each layer only adds its change to it.

Same layers, a different job

ResNet treats a network as a series of blocks, each made of a few consecutive convolutional layers, usually 2. ResNet-34, for example, is 1 first layer, 16 blocks of 2 layers, and 1 final layer that gives the answer. In the artist metaphor above, each artist is one block, not one layer.

Training works the same way as before: same data, same backpropagation, same gradient descent. What changes is the wiring around each block, and with it, what the block’s layers have to produce.

Call a block’s input x, and call the output the block should ideally produce H(x). In a plain network, the block’s layers have to turn x into H(x) directly.

In ResNet, a path carries x around the block and adds it to whatever the layers produce:

output = (layers' result) + x

Since x is added at the end regardless, the layers only have to produce the part x doesn’t already supply. Call that part F(x):

F(x) = H(x) − x

Nothing is subtracted inside the network; that formula only describes what the layers have to learn. For example, with made-up numbers: if 5.0 comes in and the right output is 5.2, the path delivers the 5.0, and the layers only have to produce 0.2.

When the layers learn F well, the output is H(x), the same thing the plain block was aiming for. The difference F, 0.2 in the example, is called the residual, the part left over, which gives the paper its name: residual learning.

Why that should help

The authors’ reason for expecting the fix to help comes from the identity idea in the first post, because passing the input through unchanged was the specific thing plain networks couldn’t learn:

• If the best a block can do is nothing (identity), a residual block reaches it by outputting 0.
• If the best is close to nothing (a small change), a residual block only has to find that small change, starting from identity.

With many blocks sharing the job, each block’s part of it is usually small, even when the total change across all of them is large.

In a plain block, doing nothing takes an exact pattern of weights. Take a layer with 3×3 filters, 64 feature maps in, and 64 out. Its 64 filters hold 36,864 weights (3 × 3 × 64 × 64). To pass its input through unchanged, each filter has to copy exactly one incoming map and ignore the rest: 64 particular weights must be exactly 1, and the other 36,800 exactly 0. Both layers in the block need that pattern, and the batch-normalization and ReLU steps between them have to leave the values alone too.

In a residual block, doing nothing means the layers output 0. The input is added back at the end, so if 5.0 comes in and 5.0 is the best output, the layers only have to produce 0. The paper says they can get there by pushing their weights toward zero. No weight has to land on an exact value like 1. The authors’ hypothesis is that this makes doing nothing the easy default, instead of a precise setting the layers have to find.

They present it as a hypothesis, not a proof: “We hypothesize that it is easier to optimize the residual mapping than to optimize the original, unreferenced mapping.” They also note that doing exactly nothing is unlikely to be the best choice in practice.

The authors also tried shortcuts around single layers and “have not observed advantages”: with only one layer inside, the layer’s result plus the input is no different from what a single plain layer can already produce. So their blocks use two or more layers.

The shortcut

The path that carries x around the block and adds it back is called a shortcut connection, or skip connection, because it skips over the layers.

Two details about the shortcut. First, it has no weights of its own. It passes x through unchanged and adds it, so a residual network has the same number of weights as a plain network with the same layers. (When a block changes the size of its data, the shortcut needs a small adjustment; that comes up two posts from now.) Second, in the paper’s design, the ReLU comes after the addition, so the layers’ output and the added input are combined before the activation.

A residual network is a stack of these blocks, each one adding its own learned change to what the blocks before it produced.

What’s next

This is a hypothesis about why the shortcut helps. The next post looks at whether it worked: the same 18- and 34-layer networks from the first post, rebuilt with shortcuts.