Approximation by superpositions of a sigmoidal function
George Cybenko1989Mathematics of Control, Signals and Systems
Read it on doi.org(opens in a new tab)Why this one
Read this before Understanding deep learning requires rethinking generalization. Cybenko gives you the clean version of a claim people repeat badly: a single hidden layer with a sigmoidal activation can approximate any continuous function on a compact domain, if you allow enough units. That is a statement about expressiveness, not training, sample efficiency, robustness, depth, or whether SGD will find the weights. The useful lesson for an engineer is the separation of questions. Can this architecture represent the mapping? Can my optimizer find it? Can my data identify it? Will it generalize? Those are different problems. This paper is worth an evening because it gives you the first answer with real mathematical discipline, then leaves the other three unanswered on purpose.
What to take away
- Universal approximation is about existence of weights, not an algorithm for finding them.
- The compact domain assumption matters, the theorem is not a free pass for arbitrary inputs.
- Width can buy representation, but the theorem says nothing about efficiency or generalization.
Reads with
- Understanding deep learning requires rethinking generalization
shows why expressiveness alone does not explain why deep nets generalize
- Neural Tangent Kernel: Convergence and Generalization in Neural Networks
turns overparameterized networks into a trainability and generalization question
- Understanding the difficulty of training deep feedforward neural networks
moves from what networks can represent to why gradient training can fail