Tag

training

3 posts

Paper #3: Not a Fading Signal (Batch Normalization)

The usual suspect when deep networks fail is a learning signal that fades as it travels back through the layers. By 2015 a technique called batch normalization had largely dealt with that, so ResNet's problem had to be something else.

Sep 28, 20233 min read

Paper #2: The Finer Points (Normalization, Pooling, and the Training Recipe)

The big ideas are in place. This post gathers the smaller tricks AlexNet used, including one now considered obsolete, and the plain recipe for actually training it.

Sep 25, 20233 min read

Paper #2: Splitting One Network Across Two Graphics Cards

ReLU made training fast enough. But the network was too big to fit on a single graphics card, so the authors did something unusual: they cut it in half.

Sep 23, 20234 min read

All tags