Paper #1, Part 15: Why Train From Scratch? Transfer Learning
Training a large CNN from scratch requires massive amounts of data and compute. Transfer learning reuses a network trained on one task as a head start on another.
Where we are: we can build, train, and inspect convolutional networks (Parts 12 to 14). Training one from scratch is expensive, though, so this post reuses a network someone else already trained.
Training a CNN from scratch is expensive. You need:
- Hundreds of thousands of images
- Weeks of GPU compute
- Careful hyperparameter tuning
If you’re building a classifier for a niche task (detecting disease in medical scans), you might not have that data.
Transfer learning solves this. Reuse a network trained on a large, public dataset (like ImageNet, which has over a million images across a thousand categories) as a starting point. Adapt it to your task.
How It Works
Pre-trained networks have learned general visual features: edges, textures, shapes, objects. These features are useful for almost any vision task.
Take a pre-trained network. Remove its final classification layer, the one that predicts the original task’s categories (for example, ImageNet categories, if ImageNet is what it was trained on). Replace it with a new classification layer for your task (for example, telling cats from dogs, or disease from healthy).
Now you have two choices:
Option 1: Fine-tuning
Train the entire network (including pre-trained layers) on your small dataset, but with a low learning rate. The pre-trained weights are a good starting point; small adjustments adapt them to your task.
This works well when you have a fair amount of data: enough examples that adjusting all the weights improves the fit instead of just overfitting.
Option 2: Feature extraction
Freeze the pre-trained layers (don’t update their weights). Only train the new final layer.
This is faster and needs less data. You’re treating the pre-trained network as a feature extractor: it converts your images to high-level features, and you just train a simple classifier (often a single new layer) on top.
This works well when your data is scarce, too little to safely retrain the whole network without overfitting. It relies on the borrowed features being relevant, so it works best when your task resembles the everyday objects ImageNet was trained on; the more your images differ from those, the less the frozen features help.
The Hiring Analogy
Imagine hiring someone with years of general experience in the field versus a total beginner. The experienced hire still has to learn your specific job, but they pick it up far faster because they already have the foundation. Transfer learning is similar.
A network trained on ImageNet already understands vision in general. Adapting it to your specific task is much faster than teaching it to see from zero.
Impact
Transfer learning is what puts state-of-the-art CNNs within reach of people without giant datasets or compute budgets. Instead of the hundreds of thousands of images and weeks of GPU time that training from scratch demands, you can build a competitive system with modest data and a pre-trained network.
What’s Next
We’ve covered:
- The fundamentals of neural networks
- How convolutions exploit local structure
- How to interpret what networks learn
- How to reuse trained networks
Part 16 ties everything together: a complete walkthrough of how a CNN classifies an image from start to finish.