Paper #3: Deeper Networks That Did Worse
After AlexNet, adding layers kept helping, up to a point. Past about twenty layers, plain networks started scoring worse, even on the images they were trained on. ResNet starts from that puzzle.
Where we are: AlexNet showed that a deep network could beat hand-designed vision systems. Paper #3 is ResNet, the 2015 paper “Deep Residual Learning for Image Recognition” by Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun of Microsoft Research. It starts from a problem that showed up when people tried to go much deeper.
Why it matters: before ResNet, making a plain network much deeper could make it worse; a 34-layer network did worse than an 18-layer one. ResNet’s shortcuts let networks keep improving as they got deeper, up to 152 layers, and won the 2015 ImageNet contest with 3.57% top-5 error, down from 6.66% for the previous year’s winner.
Depth helped, at first
AlexNet had eight layers with weights. Two years later, a network from Oxford called VGG, built in versions with 16 and 19 layers of small 3×3 filters, reached 7.32% top-5 error in the 2014 contest, about half of AlexNet’s 15.3%. The architecture post showed the same effect on a small scale: removing one of AlexNet’s middle layers cost about two points of accuracy. Each extra layer can build on the patterns found by the one before it, from edges to textures to object parts, so a deeper network can describe more complicated things.
So the next step was to keep stacking layers.
The ResNet authors built their plain networks the VGG way, with two simple rules: use small 3×3 filters, and whenever the feature maps are shrunk to half their width and height, double the number of filters, so each layer does about the same amount of work. Making a network deeper is then a matter of adding more of these layers.
Then depth hurt
The ResNet authors compared two of their own plain networks, built like VGG with their layers simply placed one after another: one with 18 layers and one with 34. The deeper network scored worse. On ImageNet, the 18-layer network had 27.94% top-1 error on the validation images (the share of images where its single best guess was wrong), and the 34-layer network had 28.54%. Sixteen more layers made it slightly worse, not better. Top-1 is a stricter measure than the top-5 numbers above; measured the same way, VGG-16 had 28.07%, so these networks are in the same range as VGG. They saw the same pattern on CIFAR-10, the small 10-category dataset from Paper #1’s post on nearest neighbor, where a 56-layer plain network did worse than a 20-layer one.
Not overfitting
The first guess would be overfitting: a bigger network memorizing its training images. But overfitting has a signature. The network does well on the images it trained on and badly on new ones. Here, the deeper network did worse on its own training images too. The authors write that the 34-layer plain network “has higher training error throughout the whole training procedure.” A network that fits its training data worse than a smaller one is not overfitting.
They also checked whether more training time would fix it. Training three times as long still showed the same problem.
A deeper network should never be worse
Maybe a 34-layer network is just unable to do as well as an 18-layer one. The authors show that it can, with a thought experiment. Take the trained 18-layer network. Build a 34-layer network by copying those 18 layers and adding 16 more after them that pass their input through unchanged. That deeper network computes exactly what the shallow one does, so its error is exactly the same. A layer that passes its input through unchanged is called an identity layer, because what comes out is identical to what went in.
So, according to the thought experiment, a 34-layer network at least as good as the 18-layer one exists. If that’s the case, the standard training process of the time should have eventually ended up with a 34-layer network that scored at least as well. It didn’t. In the paper’s words, the training methods “are unable to find solutions that are comparably good or better than the constructed solution.” The paper calls this problem degradation: accuracy that gets worse as plain networks get deeper, even on their training data.
So the problem is not overfitting, and not too little training time. A stack of layers trained from scratch apparently has trouble learning to do nothing, i.e. learning to simply pass its input through unchanged, even when doing nothing would be the best choice.
What’s next
A natural suspect is the fading learning signal described in the architecture post. The paper argues that this was not the cause by 2015, and the next post explains why, along with the tool that had dealt with that signal by then: batch normalization.