Adversarial examples in random neural networks with general activations
Andrea Montanari
Stanford University, USAYuchen Wu
Stanford University, USA
![Adversarial examples in random neural networks with general activations cover](/_next/image?url=https%3A%2F%2Fcontent.ems.press%2Fassets%2Fpublic%2Fimages%2Fserial-issues%2Fcover-msl-volume-6-issue-1.png&w=3840&q=90)
Abstract
A substantial body of empirical work documents the lack of robustness in deep learning models to adversarial examples. Recent theoretical work proved that adversarial examples are ubiquitous in two-layers networks with sub-exponential width and ReLU or smooth activations, and multi-layer ReLU networks with sub-exponential width. We present a result of the same type, with no restriction on width and for general locally Lipschitz continuous activations.
More precisely, given a neural network with random weights , and feature vector , we show that an adversarial example can be found with high probability along the direction of the gradient . Our proof is based on a Gaussian conditioning technique. Instead of proving that is approximately linear in a neighborhood of , we characterize the joint distribution of and for , where for some positive step size .
Cite this article
Andrea Montanari, Yuchen Wu, Adversarial examples in random neural networks with general activations. Math. Stat. Learn. 6 (2023), no. 1/2, pp. 143–200
DOI 10.4171/MSL/41