Tag

activation functions

2 posts

Paper #2: ReLU, the Small Change That Made Big Networks Trainable

AlexNet's shape is familiar from Paper #1. What we skipped is how the activation function controls training speed. Picking a fast one, ReLU, is the first thing that made training a network as big as AlexNet possible.

Sep 22, 20235 min read

Paper #1, Part 9: Neurons and Networks (Stacking Simple Decisions Into Something Smart)

A single linear classifier is rigid. But what if you stack multiple layers, each adding non-linearity? That gives you a neural network.

Sep 12, 20238 min read

All tags