The goal of backpropagation is to compute the partial derivatives b in the network.
we use the quadratic cost function
two assumptions :
1: The first assumption we need is that the cost function can be written as an average
(case for the quadratic cost function)
The reason we need this assumption is because what backpropagation actually lets us do is compute the partial derivatives
by averaging over training examples. In
subscript, writing the
back in, but for now it's a notational nuisance that is better left implicit.
2: The cost function can be written as a function of the outputs from the neural network
the Hadamard product
(s⊙t)j=sjtj
The four fundamental equations behind backpropagation
BP1
:the error in the jth neuron in the lth layer
You might wonder why the demon is changing the weighted input . Surely it'd be more natural to imagine the demon changing
as our measure of error. In fact, if you do this things work out quite
similarly to the discussion below. But it turns out to make the presentation of backpropagation a little more algebraically complicated.
as our measure of error.
An equation for the error in the output layer, are given by
it's easy to rewrite the equation in a matrix-based form, as
BP2
BP3
BP4
The backpropagation algorithm
Of course, to implement stochastic gradient descent in practice you also need an outer loop generating mini-batches
of training examples, and an outer loop stepping through multiple epochs of training. I've omitted those for simplicity.
reference: http://neuralnetworksanddeeplearning.com/chap2.html