Paper #6: From Fixed Inputs to Sequences
Every network in Chapter 1 takes in a whole image at once and answers in one pass. Paper #6 is about recurrent networks, which read a sequence one piece at a time and carry a memory of what they've read.
Paper #6 · 2015
RNNs
The Unreasonable Effectiveness of Recurrent Neural Networks
Networks that read and write sequences one step at a time.
Start readingPaper #6: How an RNN Remembers
An RNN's memory is a list of numbers. At each step it mixes that list with the new input to make an updated list, using the same weights every time. The network repeats this once for each character it reads.
Paper #6: Predicting the Next Character
Karpathy trains RNNs on one task: given the characters so far, score each possible next character. One word, "hello," holds 4 training examples, and its 2 l's show why the network needs a memory.
Paper #6: What It Learned to Write
Karpathy trained the same kind of network on 6 texts, from Paul Graham's essays to the Linux source code. It learned spelling, play formatting, nested markup, and balanced brackets, and it lost track of things that span long stretches.
Paper #6: Looking Inside, and What It Left Open
A few of a trained RNN's memory values track things nobody told it about, such as whether it's inside a quote. Karpathy also names what RNNs do poorly, and points to attention as the next step.