227-0423-00L Neural Network Theory

Abstract	The class focuses on fundamental mathematical aspects of neural networks with an emphasis on deep networks: Universal approximation theorems, capacity of separating surfaces, generalization, reproducing Kernel Hilbert spaces, support vector machines, fundamental limits of deep neural network learning, dimension measures, feature extraction with scattering networks
Learning objective	After attending this lecture, participating in the exercise sessions, and working on the homework problem sets, students will have acquired a working knowledge of the mathematical foundations of neural networks.
Content	1. Universal approximation with single- and multi-layer networks 2. Geometry of decision surfaces 3. Separating capacity of nonlinear decision surfaces 4. Generalization 5. Reproducing Kernel Hilbert Spaces, support vector machines 6. Deep neural network approximation theory: Fundamental limits on compressibility of signal classes, Kolmogorov epsilon-entropy of signal classes, covering numbers, fundamental limits of deep neural network learning 7. Learning of real-valued functions: Pseudo-dimension, fat-shattering dimension, Vapnik-Chervonenkis dimension 8. Scattering networks
Lecture notes	Detailed lecture notes will be provided as we go along.
Prerequisites / Notice	This course is aimed at students with a strong mathematical background in general, and in linear algebra, analysis, and probability theory in particular.