AdvancedTopic 95 of 104Software & Computing
Neural networks
Layers of weighted units trained by gradient descent.
In depth
Neural networks stack layers of weighted sums and non-linear activations (ReLU, sigmoid); training uses backpropagation with gradient descent (SGD, Adam) to minimise a loss. Capacity grows with depth and width. Initialisation, learning rate and normalisation determine training success.
Key points
- Forward and backward pass
- Loss functions
- Universal approximation
ExampleA network with 2 hidden layers can classify handwritten digits.
What is Neural networks?
Layers of weighted units trained by gradient descent.
Why is it important to understand neural networks?
Neural networks are flexible function approximators that learn from data, so they are behind many modern applications of machine learning.
What does a neuron in a network do?
It takes a weighted sum of its inputs, adds a bias and applies an activation function to produce an output.
How are weights learned?
Gradient descent adjusts the weights to reduce a loss function, using backpropagation to compute the gradients.
How is Neural networks used in practice?
Typical applications include recognising handwritten digits from scanned forms, predicting the next value in a time series from past readings and estimating the output of a process from measured inputs.
What should I learn before Neural networks?
Study "Supervised and unsupervised learning" first (topic 94 of 104 in Software & Computing). It covers the groundwork this topic builds on.
What are the key points of Neural networks?
Forward and backward pass. Loss functions. Universal approximation.
Can you give an example of Neural networks?
A network with 2 hidden layers can classify handwritten digits.
What comes after Neural networks?
The next topic is "Overfitting and regularisation", which builds directly on this one.
How difficult is Neural networks?
It is classed as Advanced in Software & Computing (topic 95 of 104). It is an advanced topic. It assumes you are comfortable with the intermediate topics in Software & Computing.