392132 Theory of Deep Neural Networks (V) (WiSe 2023/2024)

This lecture offers a journey through recent approaches towards a deeper theo-retical understanding of deep neural networks, focussing on questions such as properties of deep nets near initialization, mean field theory, complexity results for deep networks, properties of loss landscapes and gradient learning, infinite width limit of learning, neural tangent kernel, and renormalization group ap-proaches for the non-gaussian regime. Many results will be seen to stem from an analysis of certain limiting cases of neural network architectures (linear, vanishing depth/width ratio) that provide insights about properties of real net-works that are "close" to such cases.

The lecture will proceed along a "theory backbone" provided in the recent book "The principles of deep learning theory" by Roberts, Yaida and Hanin (2021), exploiting methods from theoretical physics for analyzing properties of deep networks which reveal striking analogies with phenomena in physical many-particle systems. This backbone will be complemented with selected papers that deepen the topics listed above further.

Requirements for participation, required level

Prerequisites are a solid background in linear algebra, multivariate calculus and rudimentary probability theory.