MLPs and Backpropagation

How does an MLP Learn?

We saw how each layer works:

$$\begin{array}{}\text{(1)}& \left[\begin{array}{c}{a}_{12}\\ a_{22}\\ a_{32}\end{array}\right]=sigmoid(\left[\begin{array}{ccc}{W}_{11}^2& W_{21}^2& W_{31}^2\\ W_{12}^2& W_{22}^2& W_{32}^2\\ W_{13}^2& W_{23}^2& W_{33}^2\end{array}\right]\ast \left[\begin{array}{c}{a}_{11}\\ a_{21}\\ a_{31}\end{array}\right]+\left[\begin{array}{c}{b}_{12}\\ b_{22}\\ b_{32}\end{array}\right])\end{array}$$

and:

$$\begin{array}{}\text{(2)}& A^l=\sigma (W^l{A}^{l-1}+B^l)\end{array}$$

See how the connections between neurons are marked by weights: these multiply the signal from the previous neuron. The multiplied/weighted products are added up in the neuron, and the sum is given to the activation block therein.

So learning?

The only controllable variables in a neural network are these weights! So learning involves adapting these weights so that they can perform a useful function.

What is the Learning Process?

The process of adapting the weights of a neural network can be described in the following steps:

• Training Set: Training is over several known input-output pairs (“training data”)
• Training Epoch: For each input, the signals propagate forward until we have an output
• Error Calculation: Output is compared with desired output, to calculate error
• Backpropagation: Each neuron (and its weights) need to be told what is their share of the error! Errors therefore need to be sent backward from the output to input, unravelling them from layer $l$ to layer $l-1$. (like apportioning blame !!).
• Error-to-Cost: How does error at any given neuron relate to the idea of an overall Cost function? Is the Cost function also apportioned in the same way?
• Differentiate: Evaluate the effect of each weight/bias on the (apportioned) error overall Cost. (Slope!!)
• Gradient Descent: Adapt the weights/biases with a small step in the opposite direction to the slope.

There.

What is the Output Error?

If $d(k)$ are the desired outputs of the NN (over an entire training set), and $y(k)$ are the outputs of the output layer, then we calculate the error at the outputs of the NN as:

$$\begin{array}{}\text{(3)}& e(k)=a(k)-d(k)\end{array}$$

This error is calculated at each output for each training epoch/sample/batch. (More about the batch-mode in a bit.)

What is the Cost Function?

We define the cost or objective function as the squared error averaged over all neurons:

$$\begin{array}{}\text{(4)}& \begin{array}{rl}C(W,b)& =\frac{1}{2n}\sum _{i=1}^{nneurons}{e}^2(i)\\ \\ & =\frac{1}{2n}\sum _{k=1}^{nneurons}(a_i-d_i{)}^2\end{array}\end{array}$$

The $a_i$s are the outputs of $n$ neurons and $d_i$ are the desired outputs for each of the training samples.

The Cost Function is of course dependent upon the Weights and the biases, and is to be minimized by adapting these. Using the sum of squared errors, along with the linear operations in the NN guarantees that the Cost Function (usually) has one global, minimum.

What is Backpropagation of Error?

As we stated earlier, error is calculated at the output. In order to adapt all weights, we need to send error proportionately back along the network, towards the input. This proportional error will give us a basis to adapt the individual weights anywhere in the network.

What does “proportional” mean here? Consider the diagram below:

$$\begin{array}{r}{e}_{11}\sim \sim e_{12}\ast \frac{W_{11}}{SumofWeightsto{h}_1{h}_1}\\ e_{21}\sim \sim e_{12}\ast \frac{W_{21}}{SumofWeightsto{h}_1{h}_1}\\ e_{31}\sim \sim e_{12}\ast \frac{W_{31}}{SumofWeightsto{h}_1{h}_1}\end{array}$$

$$\begin{array}{r}{e}_{11}\sim \sim e_{12}\ast \frac{W_{11}}{W_{11}+W_{21}+W_{31}{W}_{11}+W_{21}+W_{31}}\\ e_{21}\sim \sim e_{12}\ast \frac{W_{21}}{W_{11}+W_{21}+W_{31}{W}_{11}+W_{21}+W_{31}}\\ e_{31}\sim \sim e_{12}\ast \frac{W_{31}}{W_{11}+W_{21}+W_{31}{W}_{11}+W_{21}+W_{31}}\end{array}$$

These are the contributions of the error $e_{12}$ to each of the previous neurons.

Another way of looking at this:

$$\begin{array}{r}{e}_{11}=e_{12}\ast \frac{W_{11}}{Sumofweightsto{h}_1{h}_1}\\ +e_{22}\ast \frac{W_{21}}{SumofWeightsto{h}_2{h}_2}\\ +e_{32}\ast \frac{W_{31}}{SumofWeightsto{h}_3{h}_3}\end{array}$$

$$\begin{array}{r}{e}_{11}=e_{12}\ast \frac{W_{11}}{W_{11}+W_{21}+W_{31}{W}_{11}+W_{21}+W_{31}}\\ +e_{22}\ast \frac{W_{12}}{W_{12}+W_{22}+W_{32}{W}_{12}+W_{22}+W_{32}}\\ +e_{32}\ast \frac{W_{13}}{W_{13}+W_{23}+W_{33}{W}_{13}+W_{23}+W_{33}}\end{array}$$

$$\begin{array}{r}{e}_{21}=similarexpression!!\\ e_{31}=similarexpression!!\end{array}$$

Equation corrected by Gayatri Jadhav, April 2025

This is the total error at $e_{11}$ from all the three output errors. So:

• We have taken each output error, $e_{\ast 2}$ and parcelled it back to the preceding neurons in proportion to the connecting Weight. This makes intuitive sense; we are making those neurons put their money where their mouth is. As Nassim Nicholas Taleb says, people (and neurons!) need to pay for their opinions, especially when things go wrong!
• The accumulated error at each neuron in layer $l-1$ is the weighted sum of back-propagated error contributions from all layer $l$ neurons to which we are connected.
• So we can compactly write the relationships above as:

$$\left[\begin{array}{c}{e}_{11}\\ e_{21}\\ e_{31}\end{array}\right]=(\left[\begin{array}{ccc}\frac{W_{11}}{D_{11}}& \frac{W_{12}}{D_{12}}& \frac{W_{13}}{D_{13}}\\ \frac{W_{21}}{D_{21}}& \frac{W_{22}}{D_{22}}& \frac{W_{23}}{D_{23}}\\ \frac{W_{31}}{D_{31}}& \frac{W_{32}}{D_{32}}& \frac{W_{33}}{D_{33}}\end{array}\right]\ast \left[\begin{array}{c}{e}_{12}\\ e_{22}\\ e_{32}\end{array}\right])$$

The denominators make things look complicated! But if we are able to simply ignore them for a moment, then we see a very interesting thing:

$$\left[\begin{array}{c}{e}_{11}\\ e_{21}\\ e_{31}\end{array}\right]\sim \sim \left[\begin{array}{ccc}{W}_{11}& W_{12}& W_{13}\\ W_{21}& W_{22}& W_{23}\\ W_{31}& W_{32}& W_{33}\end{array}\right]\ast \left[\begin{array}{c}{e}_{12}\\ e_{22}\\ e_{32}\end{array}\right]$$

This new approximate matrix is the tranpose of our original Weight matrix from Equation 1! The rows there have become columns here!! That makes intuitive sense: in the forward information direction, we were accounting for information from the point of view of the destinations; in the reverse error backpropagation direction, we are accounting for information from the point of view of the sources.

Writing this equation in a compact way:

$$\begin{array}{}\text{(5)}& e^{l-1}\sim \sim {W^l}^{TT}\ast e^l\end{array}$$

This is our equation for backpropagation of error.

Why is ignoring all those individual denominators justified? Let us park that question until we have understood the one last step in NN training, the Gradient Descent.

Here Comes the Rain Maths Again!

Now, we are ready (maybe?) to watch these two very beautifully made videos on Backpropagation. One is of course from Dan Shiffman, and the other from Grant Sanderson a.ka. 3Blue1Brown.

Backpropagation in Code

Using p5.js

Using R

Using torch.

References

1. Tariq Rashid. Make your own Neural Network. PDF Online
2. Mathoverflow. Intuitive Crutches for Higher Dimensional Thinking. https://mathoverflow.net/questions/25983/intuitive-crutches-for-higher-dimensional-thinking
3. Interactive Backpropagation Explainer https://xnought.github.io/backprop-explainer/