AI Like I'm Five

Search AI Like I'm Five

By Richie Thomas in Vision5 min read

Paper #4: Moving the Activation (Pre-activation)

With the shortcut left alone, one step still sat on the path between blocks: the ReLU after the addition. Moving batch normalization and ReLU to before each weight layer removed it, and error went down.

ResNetidentity mappingspre-activation

Share

Where we are: the last post showed that a shortcut works best when it does nothing. The 2nd condition from the 1st post is that nothing comes after the addition either. In the original block, a ReLU does.

What is still on the path

Here is the original block again, from the 1st post:

A single residual block from the original ResNet. Input x goes down through weight layer, BN, ReLU, weight layer, and BN, labeled F(x). A blue shortcut carries x around those steps, unchanged, to a plus sign, giving x + F(x). A final ReLU after the plus sign is highlighted, with the note that it acts on the sum, so it also acts on x. Its result is the next block's x.

That last ReLU turns any negative value in the sum into 0. When the layers’ change is negative enough to pull the sum below 0, the next block receives 0 instead of what the layers produced. The authors wanted this step to do nothing, like the shortcut, and tried four ways of rearranging the same pieces.

Four rearrangements

The same experiment setup as the last post: ResNet-110 on CIFAR-10, original error 6.61%. They also ran a 164-layer version built from bottleneck blocks, original error 5.93%.

1. Batch normalization after the addition. Move the last BN from inside the layers’ path to after the addition. Worse: 8.17% (110 layers) and 6.50% (164). Now BN sits on the shortcut path and rescales everything that passes through it. In the paper’s words, it “alters the signal that passes through the shortcut and impedes information propagation.”

Rearrangement 1, BN after the addition. The layers' path runs weight layer, BN, ReLU, weight layer, then the shortcut x is added. After the addition come a highlighted BN, labeled moved here from the layers' path, and a ReLU. A note says both act on the sum, so both act on x.

2. ReLU before the addition. Move the last ReLU into the layers’ path, so it runs before the addition and nothing runs after it. Also worse: 7.84% and 6.14%. The problem is that the change F(x) now comes out of a ReLU, so it can never be negative. A block can only add to x, never subtract. The paper: “intuitively a ‘residual’ function should take values in (−∞, +∞),” meaning it should be able to come out positive or negative. A block whose best output is 4.8 when 5.0 comes in has no way to get there.

Rearrangement 2, ReLU before the addition. The layers' path runs weight layer, BN, ReLU, weight layer, BN, and a highlighted ReLU labeled moved here from after the addition, so F(x) can't be negative. Then the shortcut x is added, with nothing after the addition.

3. ReLU-only pre-activation. Put the ReLU at the start of the layers’ path, before the 1st weight layer, and keep BN where it was. Nothing comes after the addition. About the same as the original: 6.71% and 5.91%. The paper’s guess is that this ReLU, with no BN right before it, “may not enjoy the benefits of BN.”

Rearrangement 3, ReLU-only pre-activation. The layers' path starts with a highlighted ReLU, labeled moved here from after the addition, with no BN before it, then weight layer, BN, ReLU, weight layer, BN. Then the shortcut x is added, with nothing after the addition.

4. Full pre-activation. Put both BN and ReLU before each weight layer, so the layers’ path runs BN, ReLU, weight, BN, ReLU, weight, and nothing comes after the addition. Better than the original: 6.37% (110 layers) and 5.46% (164 layers). This is the block the paper recommends, called full pre-activation. “Activation” means the BN-and-ReLU step, and “pre” means it now comes before each weight layer instead of after.

Rearrangement 4, full pre-activation. The layers' path starts with a highlighted BN and ReLU, labeled moved here from the end of the original block, then weight layer, BN, ReLU, weight layer. Then the shortcut x is added, with nothing after the addition.

Why the order can move at all

It might seem that moving BN and ReLU to the front of the block changes what the network computes. In a plain network, it would not. The paper points out that in a plain stack of layers, the BN and ReLU between two weight layers come after one layer and before the next, so it “does not matter whether we think of them as post- or pre-activations.”

The addition is what makes the position matter. The ReLU after the addition is also on the shortcut path; the same ReLU at the start of the layers’ path is only on the layers’ path. Moving it from one to the other removes it from the shortcut’s route without removing it from the network. The layers still get their ReLUs, and the shortcut’s x arrives at the next block untouched.

In the artist picture, where each artist is one block, the step between artists is gone. Each artist works on the drawing exactly as the last one left it.

A pre-activation block, with numbers

The example from the 1st post, where the original block’s ReLU overrode the layers’ change:

original:        x = 0.3,  F(x) = −0.5  →  sum −0.2  →  ReLU  →  0
pre-activation:  x = 0.3,  F(x) = −0.5  →  sum −0.2  →  (nothing)  →  −0.2

The next block now receives −0.2, the value the layers asked for. Negative values can travel between blocks, and the forward sum from the last two posts holds exactly: any block’s input is an earlier block’s input plus every change in between.

What’s next

On 110 layers, pre-activation lowered error from 6.61% to 6.37%, and on 164 layers from 5.93% to 5.46%. The final post covers the 1,001-layer test, where the difference was much larger, and the paper’s two explanations for why it helped.

Keep reading