Paper #6: Looking Inside, and What It Left Open
A few of a trained RNN's memory values track things nobody told it about, such as whether it's inside a quote. Karpathy also names what RNNs do poorly, and points to attention as the next step.
Author
Richie Thomas
Engineer · 45 posts
Paper #6: What It Learned to Write
Karpathy trained the same kind of network on 6 texts, from Paul Graham's essays to the Linux source code. It learned spelling, play formatting, nested markup, and balanced brackets, and it lost track of things that span long stretches.
Paper #6: Predicting the Next Character
Karpathy trains RNNs on one task: given the characters so far, score each possible next character. One word, "hello," holds 4 training examples, and its 2 l's show why the network needs a memory.
Paper #6: How an RNN Remembers
An RNN's memory is a list of numbers. At each step it mixes that list with the new input to make an updated list, using the same weights every time. The network repeats this once for each character it reads.
Paper #6: From Fixed Inputs to Sequences
Every network in Chapter 1 takes in a whole image at once and answers in one pass. Paper #6 is about recurrent networks, which read a sequence one piece at a time and carry a memory of what they've read.
Paper #5: What It Achieved, and What It Left Open
The context module raised accuracy on everyday photos and on street scenes from car cameras. The bigger result was the argument behind it: a network that has to label every pixel can keep its maps large all the way through.
Paper #5: The Front End and the Context Module
The authors removed VGG-16's last two shrinking steps and beat earlier segmentation networks by more than 5 points. Then they added a small stack of dilated layers, started so each one passes its input through unchanged, and gained more.
Paper #5: Dilated Convolutions, a Filter with Gaps
A dilated convolution spaces out the positions a filter reads, so the same few weights cover a wider area. Stack a few with growing gaps, and the area each output sees grows exponentially, while the map never shrinks.
Paper #5: Why Classification Networks Shrink the Image
Classification networks shrink their feature maps on purpose: it lets later layers see wide areas cheaply. For one label that costs nothing. For a label at every pixel, it throws away the detail the answer needs.
Paper #5: From One Label to Every Pixel
Papers #2 to #4 improved the process of identifying a single thing in an image. Paper #5 labels every pixel instead: road here, pedestrian there, sky above. That needs a network that keeps detail while still seeing a wide area.
Paper #4: 1,001 Layers, and Why It Helped
With the original blocks, a 1,001-layer network did worse than a 110-layer one. With pre-activation blocks, it reached 4.92% error on CIFAR-10, the best in the paper. The authors give two reasons.
Paper #4: Moving the Activation (Pre-activation)
With the shortcut left alone, one step still sat on the path between blocks: the ReLU after the addition. Moving batch normalization and ReLU to before each weight layer removed it, and error went down.
Paper #4: Leave the Shortcut Alone
The authors tried scaling, gating, a 1×1 convolution, and dropout on the shortcuts of a 110-layer ResNet. Every one did worse than a shortcut that passes its input through unchanged.
Paper #4: A Path Straight Through
If the shortcut and the step after the addition both do nothing, any block's output is an earlier block's output plus every change in between. Going backward, the same fact gives the correction a route that does not fade.
Paper #4: The Question ResNet Left Open
ResNet showed that shortcuts work but offered only a hypothesis about why. A year later, the same authors followed the signal through the shortcuts, and found one step in the original block that got in the way.
Paper #3: What ResNet Won, and Where It Stopped
ResNet won the 2015 ImageNet contest with 3.57% top-5 error. On a small dataset, a 1,202-layer version did worse than a 110-layer one, and the paper says why.
Paper #3: Going to 152 Layers (The Bottleneck Block)
To go past 34 layers without the cost growing too fast, ResNet changed the shape of its blocks: shrink the data, do the main work, then expand it again. ResNet-152 ended up cheaper to run than the 19-layer VGG.
Paper #3: The Evidence (18 Layers vs. 34)
The same two networks from the first post, rebuilt with shortcuts. Without them, going from 18 to 34 layers made error worse. With them, it made error better by about three points.
Paper #3: Learning the Change Instead of the Whole Thing
ResNet's fix is one addition: let each block of layers learn only a change to its input, and add the input back at the end. Then doing nothing becomes easy.
Paper #3: Not a Fading Signal (Batch Normalization)
The usual suspect when deep networks fail is a learning signal that fades as it travels back through the layers. By 2015 a technique called batch normalization had largely dealt with that, so ResNet's problem had to be something else.
Paper #3: Deeper Networks That Did Worse
After AlexNet, adding layers kept helping, up to a point. Past about twenty layers, plain networks started scoring worse, even on the images they were trained on. ResNet starts from that puzzle.
Paper #2: What AlexNet Learned, and Why It Mattered
With AlexNet trained, this last post looks at what the network learned, and why this paper mattered.
Paper #2: The Finer Points (Normalization, Pooling, and the Training Recipe)
The big ideas are in place. This post gathers the smaller tricks AlexNet used, including one now considered obsolete, and the plain recipe for actually training it.
Paper #2: Fighting Overfitting at Scale
With sixty million weights and only a million images, AlexNet could have memorized its way to a useless answer. Two defenses stopped it: more data, made for free, and dropout.
Paper #2: Splitting One Network Across Two Graphics Cards
ReLU made training fast enough. But the network was too big to fit on a single graphics card, so the authors did something unusual: they cut it in half.
Paper #2: ReLU, the Small Change That Made Big Networks Trainable
AlexNet's shape is familiar from Paper #1. What we skipped is how the activation function controls training speed. Picking a fast one, ReLU, is the first thing that made training a network as big as AlexNet possible.
Paper #2: The Architecture (Everything from Paper #1, Scaled Up)
AlexNet didn't invent anything new in its shape. It's the convolutional network from Paper #1, made big: five convolutional layers, three fully-connected ones, and sixty million weights.
Paper #2: The Contest That Changed Everything
Before 2012, teaching a computer to recognize objects meant designing the clues by hand. Then a deep neural network won the hardest contest in the field, and not by a little.
Paper #1, Part 16: Putting It Together (What CS231n Actually Teaches)
Fifteen concepts. One goal: from raw pixels to classification. This is how it all fits together, and why CS231n became foundational reading.
Paper #1, Part 15: Why Train From Scratch? Transfer Learning
Training a large CNN from scratch requires massive amounts of data and compute. Transfer learning reuses a network trained on one task as a head start on another.
Paper #1, Part 14: Peeking Inside the Black Box (What Is a CNN Looking At?)
Neural networks learn patterns, but we can't see them directly. Visualization techniques let us peek inside and understand what features a network has learned.
Paper #1, Part 13: Pooling and Stacking (From Edges to Objects)
Convolutions find local patterns. Pooling zooms out. Stack them together and you build a hierarchy: edges → textures → shapes → objects.
Paper #1, Part 12: Convolutions (Spotting Patterns Anywhere in a Picture)
A fully-connected network treats every pixel equally. Convolutions exploit one fact: important patterns are local. So a convolution scans the image in small patches, looking for a pattern wherever it appears.
Paper #1, Part 11: Memorizing vs. Learning (Overfitting and Regularization)
A network can memorize training data perfectly. But memorized answers don't generalize. Overfitting is the enemy, and regularization is how you fight it.
Paper #1, Part 10: Garbage In, Garbage Out (Why Data Prep Matters)
Before a network can learn from pixels, those pixels need to be put on a consistent scale. It's a small step that makes training much more stable.
Paper #1, Part 9: Neurons and Networks (Stacking Simple Decisions Into Something Smart)
A single linear classifier is rigid. But what if you stack multiple layers, each adding non-linearity? That gives you a neural network.
Paper #1, Part 8: Backpropagation (How Blame Gets Passed Backward)
Gradient descent needs gradients. But computing them directly would be impossibly slow. Backpropagation is the trick that makes it fast.
Paper #1, Part 7: Learning by Trial and Error (Gradient Descent)
We have scores. We have loss. Now we need to adjust weights to reduce that loss. Gradient descent is the algorithm that does this, and it works by walking downhill in the dark.
Why Do Hinge Loss and Softmax Look the Way They Do?
Part 6 takes hinge loss and softmax as given. But why these formulas and not some other arbitrary ones? Why the max? Why the '+1'? Why raise e to a power instead of just taking absolute values? Here's where they come from.
Paper #1, Part 6: How Does the Computer Know It's Wrong? Loss Functions
A classifier can guess. But how do you measure how badly it guessed? Loss functions turn 'wrongness' into a number, and that number is what drives learning.
Paper #1, Part 5: Giving the Computer a Scorecard (Linear Classifiers)
We've tried memorizing examples (nearest neighbor). Now let's try something smarter: score each image and pick the highest score. It's simple, interpretable, and it teaches us how learning actually works.
Paper #1, Part 4: How to Test a Model Without Cheating
Before you can tell whether your classifier works, you have to measure it. The obvious way to measure has a trap in it, one the CS231n notes warn about. Here's the trap and the fix the course recommends.
Paper #1, Part 3: The Simplest Way to Guess (Nearest Neighbor)
Before we build anything clever, let's try the dumbest possible approach to image classification. It's simpler than it sounds, and it teaches us something crucial about why this problem is hard.
Paper #1, Part 2: The Problem of Classification
Given a grid of numbers, name what the picture shows. It sounds simple. Seven specific things make it one of the hardest problems computers ever faced.
Paper #1, Part 1: What Does It Mean for a Computer to 'See' a Picture?
Before we build anything, we need one idea: to a computer, a picture is a grid of numbers. A small cartoon dog, saved as a text file, shows this directly. Every one of its numbers is readable.