In addition to the seminar paper, I used a simple two-layer perceptron trained with gradient descent and error backpropagation (see the seminar paper for details) to recognize handwritten digits based on the MNIST dataset. The implementation was done in MatLab. I plan to publish the code on GitHub and give a short introduction in a separate article. The code and some of the results can be found in my seminar paper.
Update. The LaTeX source of the seminar paper can now be found on GitHub.
Update. The MatLab code is now available on GitHub. A short introduction can be found in my article "Recognizing Handwritten Digits using a Two-Layer Perceptron and the MNIST Dataset".
In this seminar paper we study artificial neural networks, their training and application to pattern recognition. We start by giving a general definition of artificial neural networks and introduce both the single-layer and the multilayer perceptron. After considering several activation functions we discuss network topology and the expressive power of multilayer perceptrons. The second section introduces supervised network training. Therefore, we discuss gradient descent and Newton's method for parameter optimization. We derive the error backpropagation algorithm for evaluating the gradient of the error function and extend this approach to evaluate its hessian. In addition, the concept of regularization will be introduced. The third section introduces pattern classification. Using maximum likelihood estimation we derive the cross-entropy error function. As application, we train a two-layer perceptron to recognize handwritten digits based on the MNIST dataset.
Table of Contentsseminar_toc
-  David Stutz. Introduction to Neural Networks. Seminar on Selected Topics in Human Language Technology and Pattern Recognition, 2014. PDF