Validating analytic gradient for a Neural Network

When you are building a neural network from scratch, without using any high-level framework. Then calculating the gradient while backpropagation might be tricky and complex, and the problem gets worse as the neural network goes deeper. This tutorial will help you know if your gradient expression is correct. I assume that you are already familiar with neural networks.

For this example, we are taking the following 2 layers feed-forward network. This network has 3 input neurons, 4 hidden neurons, and 1 output neuron. And has sigmoid activation for the hidden layer.

2 layers feed-forward neural network

Here is the code for writing the gradient expression for this network.