Many recent deep learning frameworks such as Tensorflow, PyTorch, Theano or Torch are based on dense tensors. However, deep learning on non-tensor data structures is also interesting – especially for sparse, three-dimensional data. This article summarizes some of my experiences regarding deep learning on custom data structures in the mentioned libraries.
This blog series collects useful insights for working with Ubuntu. In this article, I describe how to avoid building problems when building Tensorflow on an NFS-mounted
$HOME directory and how to install CUDA 8.0 without a manual driver installation.
A simple implementation of batch normalization in Tensorflow. Note that the implementation is meant for training — i.e. for testing on individual samples, the mean and variance should be fixed in advance. Also note that
tf.nn.moments is replaced by a custom method for computing mean and variance to avoid the problem described here.
In specific cases,
tf.nn.moments cannot be run on the GPU (see here). This is problematic when training (convolutional) neural networks where moments are part of the computation graph (e.g. for normalization). This snippet is a simple work around, computing mean and variance along the provided dimensions manually.
Slightly adapted example for adding new operations in Tensorflow taken from the official documentation. The files should be copied to
tensorflow/core/user_ops. The new operation is compiled using
bazel build -c opt //tensorflow/core/user_ops:zero_out.so from the Tensorflow root. The generated
.so file can usually be found by searching
bazel-bin. This code does not include the corresponding gradient function yet.
In this article, I discuss a simple Tensorflow operation implemented in C++. While the example mostly builds upon the official documentation, it includes trainable parameters and the gradient computation is implemented in C++, as well. As such, the example is slightly more complex compared to the simple
ZeroOut operation discussed in the documentation.