Nar et al. show that minimizing the cross entropy loss does not result in the optimal margin solution for linearly separable datasets. In fact, the margin obtained can be arbitrarily bad, assuming that the training point slie on an affine subspace of the input space. If the dimensionality of this subspace becomes small relative to the dimensionality of the input space, poor margins become more likely. As solution, the authors propose differential training, i.e. minimizing
$\sum_i \sum_j \log(1 + e^{-w^T(x_i – y_j)})$
where $x_i$ are positive examples and $y_j$ are negative examples. Unfortunately, the authors do not provide a multi-class formulation of this loss.
What is your opinion on this article? Let me know your thoughts on Twitter @davidstutz92 or LinkedIn in/davidstutz92.
Nar et al. show that minimizing the cross entropy loss does not result in the optimal margin solution for linearly separable datasets. In fact, the margin obtained can be arbitrarily bad, assuming that the training point slie on an affine subspace of the input space. If the dimensionality of this subspace becomes small relative to the dimensionality of the input space, poor margins become more likely. As solution, the authors propose differential training, i.e. minimizing
$\sum_i \sum_j \log(1 + e^{-w^T(x_i – y_j)})$
where $x_i$ are positive examples and $y_j$ are negative examples. Unfortunately, the authors do not provide a multi-class formulation of this loss.