Paper #6: Predicting the Next Character
Karpathy trains RNNs on one task: given the characters so far, score each possible next character. One word, "hello," holds 4 training examples, and its 2 l's show why the network needs a memory.
Tag
softmax
Why Do Hinge Loss and Softmax Look the Way They Do?
Part 6 takes hinge loss and softmax as given. But why these formulas and not some other arbitrary ones? Why the max? Why the '+1'? Why raise e to a power instead of just taking absolute values? Here's where they come from.
Paper #1, Part 6: How Does the Computer Know It's Wrong? Loss Functions
A classifier can guess. But how do you measure how badly it guessed? Loss functions turn 'wrongness' into a number, and that number is what drives learning.