# DAVIDSTUTZ

## Illustrating (Convolutional) Neural Networks in LaTeX with TikZ

Many papers and theses provide high-level overviews of the proposed methods. Nowadays, in computer vision, natural language processing or similar research areas strongly driven by deep learning, these illustrations commonly include architectures of the used (convolutional) neural network. In this article, I want to provide a collection of examples using LaTeX and TikZ to produce nice figures of (convolutional) neural networks. All the discussed examples can also be found on GitHub.

Recently, in computer vision as well as other research areas, used (convolutional) neural networks are becoming even deeper, and include a more and more diverse set of operations — including variants of convolutional layers, various normalization and pooling layers, or layers for specific tasks, such as object detection. Thus, it is increasingly important to include concise and easy-to-understand descriptions of these networks. As a result, most papers include high-level overviews of the used architecture, as for example done in the original AlexNet paper shown in Figure 1.

This article includes several examples of nice illustrations of neural network architectures in LaTeX and TikZ. These examples can easily be adapted or included in larger overview figures and address different levels of abstraction. They can be found in the following GitHub repository:

Most of these figures are taken from my PUBLICATIONS , THESES or SEMINAR PAPERS .

### Single Neuron and Perceptron

An individual neuron usually takes multiple values $x_1, \ldots, x_D$ as input, potentially also a bias term $w_0$ and computes and output $y = f(z)$. This can easily be illustrated as shown in Figure 1, using a simple node in TikZ and some input and output arrows/lines. Figure 2 shows how these neurons can be assembled into a perceptron — a linear model of the form $y_i = w^T x$ where $x_0 = 0$ may be zero to include a bias term $w_0$, often also denoted $b$. In this case, the input variables $x_1, \ldots, x_D$ are illustrates similarly to the actual neurons as separate nodes in TikZ, as well. In multi-layer perceptrons as shown in the next section, the input values could also be omitted for simplicity. I want to note that these examples merely include basic TikZ elements — nodes and lines/arrows.

### Multi-Layer Perceptrons

Figure 3 shows a full-fledged multi-layer perceptron, meaning a fully-connected network with $L$ hidden layers. The input layer, the first hidden layer as well as the last two layers are shown fully; the layers in between are omitted but implicitly indicated through TikZ fadings library. Due to the included variable names, the illustration can be used to illustrate the operations within a fully-connected network including back-propagation as shown below.

#### Backpropagation

These simple illustrations of individual neurons, or small-scale neural networks also allow to illustrate the computations required for backpropagation, see Figure 4. Here $\delta_i^{(l + 1)}$ corresponds to the errors that are propagated back from layer $(l + 1)$ to layer $(l)$.

### Convolutional Neural Networks

Convolutional neural networks are more difficult to illustrate in details due to the convolutional structure and many additional layers such as local contrast normalization/batch normalization or pooling layers. Thus, a high-level illustration might be more appropriate, as shown in Figure 5.

### Auto-Encoders and Variational Auto-Encoders

The layers used in state-of-the-art convolutional neural networks can also made explicit, including the used hyper-parameters, as shown for the (variational) auto-encoders shown in Figures 8 and 9. Here, corresponding to many recent framework such as Tensorflow and PyTorch, the layers are organized in a computational graph.