A simple implementation of batch normalization in Tensorflow. Note that the implementation is meant for training — i.e. for testing on individual samples, the mean and variance should be fixed in advance. Also note that
tf.nn.moments is replaced by a custom method for computing mean and variance to avoid the problem described here.
In specific cases,
tf.nn.moments cannot be run on the GPU (see here). This is problematic when training (convolutional) neural networks where moments are part of the computation graph (e.g. for normalization). This snippet is a simple work around, computing mean and variance along the provided dimensions manually.
Following the Theano documentation, this snippet illustrates the creation of a new Theano type, namely the Double type. Based on this type, the add operation is implemented. Originally, I intended this as a quick tutorial on how to define more complex types with differentiable operations. However, as also discussed here, this turned out to be more involved than expected.
Following the PyTorch documentation, this snippet illustrates how to extend PyTorch by manually adding a linear neural network module. The example includes the linear module as discussed in the documentation and an example application on linearly separable data.
Slightly adapted example for adding new operations in Tensorflow taken from the official documentation. The files should be copied to
tensorflow/core/user_ops. The new operation is compiled using
bazel build -c opt //tensorflow/core/user_ops:zero_out.so from the Tensorflow root. The generated
.so file can usually be found by searching
bazel-bin. This code does not include the corresponding gradient function yet.
In this article, I discuss a simple Tensorflow operation implemented in C++. While the example mostly builds upon the official documentation, it includes trainable parameters and the gradient computation is implemented in C++, as well. As such, the example is slightly more complex compared to the simple
ZeroOut operation discussed in the documentation.
Sphinx is a Python documentation tool that allows to automatically create clear documentation by parsing Python docstrings. The documentation can further be complemented using reStructuredText — a markup language similar to Markdown. This article gives a brief overview of setting up Sphinx on Ubuntu.