Paper #3: Not a Fading Signal (Batch Normalization)
The usual suspect when deep networks fail is a learning signal that fades as it travels back through the layers. By 2015 a technique called batch normalization had largely dealt with that, so ResNet's problem had to be something else.
Where we are: the last post found that plain networks got worse as they got deeper, even on their own training images. Before we discuss ResNet’s fix, it helps to rule out the most common explanation.
The usual suspect
The architecture post described one way depth can break training. The “how wrong you were” signal from backpropagation shrinks a little at each layer it passes back through, until the earliest layers barely change. The ReLU post showed one cause: S-shaped activation functions that flatten out. The usual name for this is the vanishing gradient problem, and it had been an obstacle for many years before 2012.
It would be natural to assume a fading signal is also what broke the deeper networks in the last post. The ResNet authors say it is not, because their networks used a technique, published early in 2015, that keeps the signal from fading.
Batch normalization
Back in Paper #1, the post on preparing data normalized the inputs before training: shift each input so its average is 0, and scale it so its standard deviation is 1, so no input dominates just because its numbers are bigger. Batch normalization applies the same idea inside the network, after a layer, while training runs.
It works on the small, randomly chosen batch of images the network sees in each training step. For each of a layer’s feature maps, it takes all of that map’s values, at every position and across every image in the batch, subtracts their average, and divides by their standard deviation. Then it multiplies each result by one new number, called a scale, and adds a 2nd new number, called a shift.
Each feature map has its own pair of these numbers. They usually start at 1 and 0, which leave the normalized values unchanged, and training adjusts them the same way it adjusts the filter weights, through gradient descent. So a layer can still produce larger values, smaller values, or a different average when that lowers the error. The ResNet authors put a batch-normalization step right after every convolutional layer, before the ReLU.
The effect is that the numbers flowing through the network tend to stay in a steady range, layer after layer, instead of shrinking toward zero or growing out of control. Together with better rules for choosing the starting weights, the authors write, this let networks with tens of layers “start converging,” meaning their training error started going down.
What the authors checked
The plain networks from the last post were trained with batch normalization, which makes it far less likely that the forward signals shrink toward zero. The authors also checked the backward signals, the gradients, and found that they “exhibit healthy norms,” meaning they had not shrunk away. Their conclusion: “neither forward nor backward signals vanish.”
So the deeper networks were getting a learning signal at every layer, and they still trained worse. Whatever degradation is, it is not the signal fading out.
So what is it?
The paper does not claim to know. The authors offer a guess: very deep plain networks “may have exponentially low convergence rates.” Roughly, training might make progress so slowly that, in any reasonable amount of time, it looks stuck. They also write that “the reason for such optimization difficulties will be studied in the future.”
So ResNet does not start from a full explanation of the problem. It starts from two things: a measured fact, that deeper plain networks train worse even with healthy signals, and the argument from the last post, that a deeper network should be able to copy a shallower one and do no worse. The fix in the next post is built directly on that argument.
The fading signal described in the architecture post was an obstacle for networks around 2012. By the time of ResNet, batch normalization and better starting weights had largely handled it, and the problem left over is the one this paper takes on.
What’s next
If deeper networks have trouble learning to do nothing with their extra layers, one fix is to make doing nothing the easy default. The next post covers how ResNet does that.