Neural networks: Evaluation of kernel spectra through random matrix theory

VIENNA PROBABILITY SEMINAR

In the first part of the talk we give an overview of recent mathematical results on scaling limits of neural networks with random weights. In particular we introduce the neural tangent kernel (NTK) and the conjugate kernel (CK) and explain connections to the double descent phenomenon observed in generalisation performance. In the second part of the talk we report on recent work [Piccolo, S. NeurIPS 2021] on the asymptotic spectrum of the CK in the linear width scaling regime. Our result indicates that in the case of an additive bias it is impossible to choose an activation function leaving the asymptotic CK spectrum invariant throughout multiple layers, in sharp contrast to the bias-free case where a simple integral constraint is sufficient for this type of invariance.