Paper #2: ReLU, the Small Change That Made Big Networks Trainable

Where we are: we’ve seen that AlexNet’s shape is an ordinary convolutional network, just large. The open question is how something that big could be trained at all. This post covers the first change that made it possible.

In Paper #1 we met ReLU as one activation function among a few. It is very simple: for each neuron, it takes the weighted sum coming in, keeps it if it is positive, and outputs zero if it is negative. AlexNet’s contribution here was not the idea. It was showing, at scale, how much that simple choice matters.

The old default, and why it dragged

Before ReLU, the usual activation functions were tanh and the sigmoid, both smooth S-shaped curves. They share a problem: they saturate, meaning that past a certain point they stop responding to their input.

Take the sigmoid as an example. Its input is the weighted sum a neuron receives, and its output is the number it passes on to the next layer. Here is what it outputs as the input climbs by the same amount each time:

input   output    change from the row above
  0     0.50
  2     0.88      +0.38
  4     0.98      +0.10
  6     0.998     +0.015
  8     0.9997    +0.002

Each step adds the same 2 to the input, but the output moves less and less. Past about 4 the curve is almost flat: a big change going in produces almost no change coming out. That flatness is saturation, and tanh does the same thing at both ends.

A flat curve is a problem because of how training finds its way. Training improves a weight by checking whether a small nudge to it makes the network’s final answer better or worse. That nudge reaches the final answer only through each neuron’s output. If the output barely moves, the nudge barely shows up, and training cannot tell which direction helped. Paper #1’s post on gradient descent described it as a walk downhill in fog, feeling the slope underfoot to choose a direction; a saturated neuron is a flat plateau, with no slope to feel.

It gets worse in a deep network. Training adjusts each weight using its gradient, the slope of the loss with respect to that weight, and that slope has to travel back through every neuron’s activation on the way. Each flattened activation shrinks it a little more, so the earliest layers can end up with almost nothing to learn from. Across a deep, large network, learning becomes very slow.

Why does a flat curve make training slow? Each training step moves a weight by an amount that depends on its slope, so a tiny slope means a tiny move. A network full of saturated neurons needs many more steps to reach the same accuracy, and every step costs time on the hardware. How much more time, and why that was a serious problem rather than an inconvenience, comes at the end of this post.

ReLU keeps the signal alive

ReLU does not flatten on the positive side. For any positive input, its slope stays exactly 1. The learning signal passes straight through instead of being squeezed toward zero, so weights keep moving and training keeps making progress.

The paper measured the difference. In a small test on the CIFAR-10 dataset, a four-layer network using ReLU reached a given training-error target about six times faster than the same network using tanh.

ReLU is not flawless. A neuron whose input stays negative always outputs zero, and zero has zero slope, so that neuron can get stuck and stop learning, a quirk people call the “dying ReLU.” In 2012 the speed was well worth that risk, and later variants soften the flat side to avoid it.

If S-curves flatten out, why were they the default? They had advantages at the time. Their output stays in a fixed range, so numbers never grow out of control. Saturation also hurt much less in the shallow networks common before 2010, since the problem compounds with depth. And ReLU only began catching on in deep networks a couple of years before AlexNet; measurements like the paper’s six-times speedup helped move the field.

S-curves did not disappear, either. A network that answers yes or no still ends with a sigmoid, because its 0-to-1 output can be used directly as a probability, which a loss like the one in Paper #1’s post on loss functions requires. Both also appear inside LSTMs, a later paper in this series.

Why six times faster mattered so much

On its own, “trains faster” sounds like a nice-to-have. At AlexNet’s size it was the difference between possible and not.

The network had 60 million weights and 1.2 million images to learn from, and even with ReLU the training took five to six days. A slower activation would have stretched each run from days toward weeks, and AlexNet needed many runs: the winning entry alone averaged seven separately trained networks. Trained one after another at weeks each, those seven alone would take months. You cannot iterate on a design when a single attempt takes that long. The authors wrote that without faster nonlinearities, they “would not have been able to experiment with such large neural networks.”

So the smallest-looking choice in the paper, replacing a smooth curve with a bent line, ended up mattering a great deal. It did not make the network smarter; it made training it fast enough to try at all.

What’s next

ReLU solved the speed problem. But AlexNet hit a second limit, this one about hardware rather than math: the network was simply too big to fit on a single graphics card. The next post is the odd, clever way they worked around that.