AdvancedTopic 96 of 104Software & Computing
Overfitting and regularisation
Models memorising training data fail on new data; regularisation helps.
In depth
Overfitting happens when a model memorises noise and generalises poorly. Remedies: more data, simpler models, regularisation (L1/L2, dropout, early stopping, data augmentation) and cross-validation. Underfitting is the opposite. Learning curves diagnose the issue.
Key points
- Training vs validation error
- Weight decay
- Ensembling
ExampleA model with 100% training but 70% test accuracy is overfitted.
What is Overfitting and regularisation?
Models memorising training data fail on new data; regularisation helps.
Why is it important to understand overfitting and regularisation?
Overfitting makes a model look accurate on training data but fail in real use, so checking performance on unseen data is essential.
How do I detect overfitting?
Compare the error on training data with the error on a separate validation set, and a large gap signals overfitting.
What is regularisation?
Methods such as weight penalties or dropout that discourage overly complex models and improve performance on new data.
How is Overfitting and regularisation used in practice?
Typical applications include finding that a model predicts training failures well but misses new ones, using a validation set to choose the complexity of a model and adding dropout or weight penalties to make a network generalise better.
What should I learn before Overfitting and regularisation?
Study "Neural networks" first (topic 95 of 104 in Software & Computing). It covers the groundwork this topic builds on.
What are the key points of Overfitting and regularisation?
Training vs validation error. Weight decay. Ensembling.
Can you give an example of Overfitting and regularisation?
A model with 100% training but 70% test accuracy is overfitted.
What comes after Overfitting and regularisation?
The next topic is "Deep learning", which builds directly on this one.
How difficult is Overfitting and regularisation?
It is classed as Advanced in Software & Computing (topic 96 of 104). It is an advanced topic. It assumes you are comfortable with the intermediate topics in Software & Computing.