AI Like I'm Five

Search AI Like I'm Five

The series

The papers

This blog works through 30papers.com, the reading list Ilya Sutskever reportedly gave John Carmack. The order below is a planned path, built so each paper leans on the ones before it. Completed papers link to their full series.

6 of 27 papers45 posts published

01

Seeing

How machines learn to recognize images: convolutional neural networks.

  1. 01CS231n · Stanford courseA Stanford course that builds computer vision from a single pixel up to deep convolutional networks.17 posts
  2. 02AlexNet · 2012The paper that proved deep CNNs work at scale and set off the deep-learning boom.7 posts
  3. 03ResNet · 2015Skip connections that let networks go hundreds of layers deep without breaking.6 posts
  4. 04Identity Mappings · 2016A closer look at why skip connections work as well as they do.5 posts
  5. 05Dilated Convolutions · 2016Seeing wider context in an image without throwing away resolution.5 posts
02

Remembering

Networks that handle sequences and hold on to context: RNNs and LSTMs.

  1. 06RNNs · 2015Networks that read and write sequences one step at a time.5 posts
  2. 07LSTMs · 2015The gated memory cell that made recurrent networks practical.Up next
  3. 08RNN Regularization · 2014Keeping big recurrent networks from memorizing instead of learning.Planned
  4. 09Deep Speech 2 · 2015Turning raw audio into text with one big network.Planned
03

Attention and Transformers

The road to large language models, from the first attention mechanism onward.

  1. 10Attention · 2014The paper where the attention mechanism is born.Planned
  2. 11Pointer Networks · 2015Letting a network point back at positions in its own input.Planned
  3. 12Order Matters · 2015When the order you feed inputs and read outputs changes the answer.Planned
  4. 13Transformer · 2017The Transformer: the architecture behind modern large language models.Planned
  5. 14Annotated Transformer · 2018The Transformer walked through line by line, in code.Planned
04

Memory, reasoning, structure

Giving networks an external memory, relations, and graphs to reason over.

  1. 15Neural Turing Machines · 2014A network with an external memory it learns to read from and write to.Planned
  2. 16Relational Memory · 2018Memory that can relate its own stored pieces to each other.Planned
  3. 17Relational Reasoning · 2017A building block for reasoning about how objects relate.Planned
  4. 18Graph Networks · 2017Learning on graphs by passing messages between connected nodes.Planned
05

Scale and systems

What changes when the models, and the machines training them, get big.

  1. 19Scaling Laws · 2020The math behind why bigger models keep getting better.Planned
  2. 20GPipe · 2018Training models too big to fit on a single chip.Planned
06

The deep why

Information, compression, and complexity: why learning works at all.

  1. 21VLAE · 2016A bridge from practice to the theory of what a model should throw away.Planned
  2. 22Keeping Networks Simple · 1993Overfitting seen through the lens of information and compression.Planned
  3. 23MDL Principle · 2004Learning as finding the shortest description of the data.Planned
  4. 24Kolmogorov ComplexityThe shortest program that could produce a given output.Planned
  5. 25Complexodynamics · 2011Why complexity tends to rise and then fall over time.Planned
  6. 26Coffee Automaton · 2014Measuring how complexity rises and falls, with a cup of coffee.Planned
  7. 27Machine Super Intelligence · 2008A formal attempt to define intelligence itself. The capstone of the list.Planned