Handwriting by humans may seem as a very easy task but when done by a machine
is a very complex one. It is impossible for humans to expend so much of their
time to interpret characters apart from them analyzing the collected data. Our
main focus should be on analyzing the data rather than trying to interpret the
data in the first place. Apart from this, the manual interpretation of data may
not yield the right results since it may wary from person to person and hence it
is not accurate to a great extent and may take a lot of time and energy.
Algorithms using neural networks have made this task a lot easier and more
accurate. Therefore neural networks have been utilized with an aim to determine
the characters by recognizing those using algorithms, which use various rules
for prediction of handwritten characters. In this paper we discuss the
recognition of handwritten digits taken from the MNIST data set and calculate
the efficiency of our algorithm. This is done by training a neural network
using stochastic gradient descent and backpropagation.
Digit recognition, Backpropagation, Mini batch
Handwriting is a form
of writing peculiar to a person with it varying in size, shape of letters,
spacing between letters. There are different styles of handwriting including
cursive, block letters, calligraphy, signature etc. It makes the task of
recognizing handwritten characters a bit complex for neural networks since it
has to predict the characters based on its ability to learn rather than being
explicitly programmed. According to Tom Mitchell “A computer program is
said to learn from experience E with respect to some class of tasks T and
performance measure P, if its performance at tasks in T, as measured by P,
improves with experience E.”link
A neural network
consists of neurons which are simple processing units and there are directed,
weighted connections between these neurons.
For a neuron j,
propagation function receives the outputs of other neurons and transforms them
in consideration of the weights into the network input that can be further
processed by the activation function.link
Mini batch gradient
descent used in the paper is a combination of batch gradient descent and
stochastic gradient descent algorithms. It calculates model error rate by
splitting data set into small batches.
algorithm used in this paper is used for adjusting the weights in the neural
network. The algorithm works by comparing the actual output and the desired output
for a given input and calculates error value. The weights are adjusted based on
the error value. The error is first calculated at the output layer and then
distributed for the other layers.
Digit recognition is done by training a multi-layer
feedforward neural network by using mini batch stochastic gradient descent and
The MNIST data set obtained from link1 contains a
modified version of the original training set of 60,000 images. The original
training set is split into a training set with 50,000 examples and a validation
set with 10,000 examples. This set is then used to train the neural network. Each image is represented
as numpy 1-dimensional array of 784 float values between 0 and 1. The labels
are numbers between 0 and 9 indicating which digit the image represents. link1
artificial neural network with sigmoid neurons is implemented. Therefore, the output
of each neuron is calculated using the sigmoid function.
The output of each neuron is given as. Where, w is the
weight, b is the bias and x is the input.
Initially, the weights and biases of the neural
network are initialized randomly using Gaussian distribution. They are later
adjusted by applying mini batch stochastic gradient descent and backpropagation.
The training data is split into a number of mini
batches. In each epoch, the training data is shuffled and split into mini
batches of a fixed size and gradient descent is applied. The neural network is
trained for a number of epochs. The gradient of the cost function is calculated
by using the backpropagation algorithm. This calculated gradient is then used
to update the weights and biases of the neural network. Starting from the
output layer and moving backwards, the biases and weights between connections
After training the network during each epoch, the
trained network is tested using the 10,000 test images.