AI Like I'm Five

Search AI Like I'm Five

By Richie Thomas in Vision5 min read

Paper #4: The Question ResNet Left Open

ResNet showed that shortcuts work but offered only a hypothesis about why. A year later, the same authors followed the signal through the shortcuts, and found one step in the original block that got in the way.

ResNetidentity mappingsresidual connections

Share

Where we are: Paper #3, ResNet, added shortcuts that let networks train at 152 layers. Paper #4 is “Identity Mappings in Deep Residual Networks,” by the same four authors (Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun), published at the European Conference on Computer Vision (ECCV) in 2016, one year after ResNet.

Why it matters: ResNet showed that shortcuts work. This paper showed how to arrange each block so the signal passes from block to block untouched. With that change, a 1,001-layer network trained well and reached 4.92% error on CIFAR-10, where the original design got 7.61%.

The idea first- a metaphor

Paper #3 described ResNet as a line of artists: each artist is one block, and each one receives the drawing, makes a change on top, and hands the drawing on.

This paper looks at the handing-on. In ResNet’s original design, between one artist and the next, the drawing passes through one more step that can alter it. With a handful of artists that barely matters. With hundreds, the paper finds, it starts to get in the way. The paper’s main change is to move that step off the path between artists, so the drawing reaches each artist exactly as the last one left it.

What ResNet left open

ResNet showed that shortcuts work, but its explanation was a hypothesis: that learning a small change to the input is easier than learning the whole output. Its last post also ended on a result that did not fit neatly. On CIFAR-10, a 1,202-layer ResNet had 7.93% error, worse than the 110-layer version’s 6.43%. The authors suspected overfitting.

This paper takes a closer look. Instead of only training networks and comparing scores, the authors write down how a signal moves through a stack of residual blocks, forward (from the image to the answer) and backward (the correction from backpropagation), and ask what the path between blocks needs so that nothing changes the signal on its way through except the blocks’ own changes.

Their summary, from the abstract: “the forward and backward signals can be directly propagated from one block to any other block, when using identity mappings as the skip connections and after-addition activation.” So the signal passes straight through only when two things are true. Both are about leaving the signal alone:

  1. The shortcut is an identity mapping. It carries x, the block’s input, forward with nothing multiplied or added.
  2. Nothing after the addition changes the result. Whatever comes out of “x plus the layers’ change” goes to the next block as it is.

ResNet’s original block meets the 1st condition but not the 2nd.

The original block, step by step

Here is one block from the original ResNet, in the order its steps run. The weight layers are 3×3 convolutions, BN is batch normalization, and ReLU is the activation that turns every negative value into 0.

A single residual block from the original ResNet. Input x goes down through weight layer, BN, ReLU, weight layer, and BN, labeled F(x). A blue shortcut carries x around those steps, unchanged, to a plus sign, giving x + F(x). A final ReLU after the plus sign is highlighted, with the note that it acts on the sum, so it also acts on x. Its result is the next block's x.

The highlighted ReLU is the step the metaphor described. The ReLU after the addition applies to the sum, so it applies to what the shortcut carried too. When the sum is 0 or more, it changes nothing. x itself comes out of an earlier ReLU and is never negative, but the layers’ change can be, and then the sum can drop below 0:

x = 5.0,  F(x) = −0.2   →  sum 4.8   →  ReLU  →  4.8   (unchanged)
x = 0.3,  F(x) = −0.5   →  sum −0.2  →  ReLU  →  0     (changed)

In the 2nd case the next block receives 0, not −0.2. The block’s result should have been −0.2, and the step after the addition replaced it with 0. Each of these cases is small, but a 1,001-layer network has 333 blocks, and every value passes through that ReLU 333 times on its way to the answer.

Two questions

The paper asks the same question of both pieces on the path from one block to the next: should this piece do anything at all?

  • The shortcut. ResNet chose a shortcut that passes x unchanged, but other designs had put gates and weights on their shortcuts. Would any of those do better?
  • The step after the addition. Can the block be rearranged so that nothing comes after the addition?

The paper’s answers: none of the other shortcuts it tried did better than the one that passes x unchanged, and yes, the block can be rearranged so that nothing comes after the addition. In both cases, the version that left the signal alone did best. The next post shows why, by following the signal through a stack of blocks where both pieces do nothing.

Here is the rearranged block, from later in the paper. Nothing is deleted: it still has two BN steps and two ReLUs. The BN and ReLU that ended the original block now start it, so each BN-and-ReLU pair runs before its weight layer instead of after. That takes the last ReLU off the shortcut’s path; it now acts only on the layers’ path:

The rearranged pre-activation block. Input x goes down through BN, ReLU, weight layer, BN, ReLU, and weight layer, labeled F(x). The first BN and ReLU are highlighted, with the note: moved here from the end of the original block. A blue shortcut carries x around the layers, unchanged, to a plus sign, giving x + F(x). Below the plus sign, a note says nothing after the addition, and the result is the next block's x.

What’s next

The next post works through the paper’s two formulas with small numbers: one for the forward signal and one for the backward correction, and why a shortcut that does nothing keeps the correction from fading.

Keep reading