Carlini and Wagner study the effectiveness of adversarial example detectors as defense strategy and show that most of them can by bypassed easily by known attacks. Specifically, they consider a set of adversarial example detection schemes, including neural networks as detectors and statistical tests. After extensive experiments, the authors provide a set of lessons which include:

• Randomization is by far the most effective defense (e.g. dropout).
• Defenses seem to be dataset-specific. There is a discrepancy between defenses working well on MNIST and on CIFAR.
• Detection neural networks can easily be bypassed.

Additionally, they provide a set of recommendations for future work:

• For developing defense mechanism, we always need to consider strong white-box attacks (i.e. attackers that are informed about the defense mechanisms).
• Reporting accuracy only is not meaningful; instead, false positives and negatives should be reported.
• Simple datasets such as MNIST and CIFAR are not enough for evaluation.