Paper #4: Leave the Shortcut Alone

Where we are: the last post showed that when the shortcut passes x through unchanged, the correction from backpropagation reaches early blocks multiplied by “1 + something,” and the 1 cannot fade. This post covers the experiments that test it by putting something on the shortcut.

What these experiments test

The paper’s central hypothesis is that a signal travels best through a residual network when the path between blocks leaves it alone: the shortcut passes x unchanged, and nothing comes after the addition. To test the 1st half of that, the authors changed the shortcut in several ways and checked whether training got harder.

The last post’s formulas say it should: if the shortcut multiplied x by something, the 1 would become a product of those numbers across every block, which could shrink toward zero or grow very large.

The authors tried several changes, which were not arbitrary:

• Gates came from highway networks, a design published around the same time as ResNet that put learned gates on its shortcuts.
• A 1×1 convolution on the shortcut was ResNet’s own option C, which had done slightly better than identity at 34 layers.
• Scaling and dropout are the simplest ways to multiply the shortcut by something other than 1.

Two of these, the shortcut-only gate and the 1×1 convolution, can do everything the identity shortcut does: they have extra weights, and they could learn to pass x through unchanged. In the paper’s words, they “should have stronger representational abilities than identity shortcuts.” So if they still did worse, the problem would be training, not what the network was able to represent.

The test network was ResNet-110 on CIFAR-10, built from “54 two-layer Residual Units,” meaning 54 blocks. With the original identity shortcut, it had 6.61% test error. Each experiment changed only the shortcut, the same way in all 54 blocks.

Scaling the shortcut by 0.5

The simplest change: multiply x by 0.5 on its way around each block. The paper tried this two ways.

• Shortcut scaled, layers not: the network “does not converge well.” The paper marks it as a failure, meaning test error above 20%.
• Both scaled by 0.5: it trains, but reaches 12.35%, almost twice the original error.

The last post’s formula shows why. With the shortcut scaled, the 1 in “1 + something” becomes 0.5 at every block it passes. Going back across all 54 blocks, the shortcut route carries:

0.5 × 0.5 × ... (54 times) = 0.5⁵⁴ ≈ 0.00000000000000006

That is the photocopy problem again, this time on the path that was supposed to avoid it. The paper says the same about any scale other than 1: below 1, the factor “can be exponentially small and vanish, which blocks the backpropagated signal from the shortcut,” and above 1 it “can be exponentially large.” A scale of 2 would give 2⁵⁴, about 18 quadrillion.

The scaled networks also had higher error on their own training images than the original. A network that is overfitting does well on its training images and worse on new ones, so this was a different problem: training could not find good settings for the weights.

Gates

Highway networks put a learned gatea learned number between 0 and 1 that scales how much of a signal passes through; highway networks put gates on their shortcuts. on the shortcut: a number between 0 and 1 that sets how much of x passes through. The authors tried two versions.

• Exclusive gating: the shortcut gets the gate’s opposite (1 − g) and the layers get the gate (g). The best result was 8.70%.
• Shortcut-only gating: only the shortcut is gated. When the gate started out at about 0.5, the error was 12.86%. When it started out almost fully open, passing about 99.75% of x, the error was 6.91%, close to the original 6.61%.

The pattern runs in one direction. The closer the gate stayed to passing everything, the closer the result came to the plain identity shortcut, and none of them beat it.

A 1×1 convolution

In ResNet, a 1×1 convolution on every shortcut (option C) had done slightly better than identity, 24.19% top-1 error on ImageNet against 25.03%, at 34 layers with 16 blocks. At 110 layers it did much worse: 12.22%. The authors’ explanation: “When stacking so many Residual Units (54 for ResNet-110), even the shortest path may still impede signal propagation.” A 1×1 convolution multiplies x by learned weights, so the shortcut route no longer carries a plain 1.

Dropout

Last, they applied dropout to the shortcut, randomly zeroing half its values during training. The network “fails to converge to a good solution.” On average, dropping half the values scales the shortcut by 0.5, so this is the scaling experiment again in a random form.

All the results together

Test error on CIFAR-10 with ResNet-110, lower is better:

• identity shortcut (original): 6.61%
• shortcut-only gate, started almost fully open: 6.91%
• exclusive gate, best setting: 8.70%
• 1×1 convolution: 12.22%
• both paths scaled by 0.5: 12.35%
• shortcut scaled by 0.5, layers not: failed (above 20%)
• dropout on the shortcut: failed (above 20%)

Both the shortcut-only gate and the 1×1 convolution had higher training error than the identity shortcut, which the authors read as “caused by optimization issues, instead of representational abilities.” Every change to the shortcut is repeated in all 54 blocks, so even a small change multiplies up. A shortcut that does nothing is the only one whose effect stays 1 no matter how many blocks there are.

What’s next

That is the paper’s evidence for the 1st condition: the shortcut should pass x through unchanged. The 2nd is the ReLU that sits after the addition in the original block. The next post covers how the authors rearranged the block to get rid of it.