Paper #6: Looking Inside, and What It Left Open
A few of a trained RNN's memory values track things nobody told it about, such as whether it's inside a quote. Karpathy also names what RNNs do poorly, and points to attention as the next step.
Tag
RNNs
Paper #6: What It Learned to Write
Karpathy trained the same kind of network on 6 texts, from Paul Graham's essays to the Linux source code. It learned spelling, play formatting, nested markup, and balanced brackets, and it lost track of things that span long stretches.
Paper #6: Predicting the Next Character
Karpathy trains RNNs on one task: given the characters so far, score each possible next character. One word, "hello," holds 4 training examples, and its 2 l's show why the network needs a memory.
Paper #6: How an RNN Remembers
An RNN's memory is a list of numbers. At each step it mixes that list with the new input to make an updated list, using the same weights every time. The network repeats this once for each character it reads.
Paper #6: From Fixed Inputs to Sequences
Every network in Chapter 1 takes in a whole image at once and answers in one pass. Paper #6 is about recurrent networks, which read a sequence one piece at a time and carry a memory of what they've read.