Spectral Analysis of Deep Neural Network Weight Matrices Using Random Matrix Theory

Abstract

This paper establishes a framework for systematic spectral analysis of weight matrices in deep neural networks, grounded in Random Matrix Theory. For various architectures, such as multilayer perceptrons, convolutional networks (ResNet-18, VGG-16), and transformers trained on standard benchmarks, the eigenvalue distributions of weight matrices are studied. Well-trained networks deviate significantly from the predictions made by Marchenko-Pasture and are found with eigenvalue spectra that are heavy-tailed and have power-law exponents α ∈ [0.7, 0.9]. We demonstrate that spectral properties and generalization performance are significantly correlated (p < 0.05). For instance, while training, there is a 30.7% improvement in the stable rank and a -0.87 Pearson correlation coefficient (p < 0.001) between the stable rank and the generalization gap. Looking at the layers separately, the first few keep a very random spectrum, but as we go deeper, we start to see a more structured spectrum that starts to diverge from the random matrix models. Spectral parameters of outlier eigenvalues increase by factors ranging from 2.3 to 12.9 during training. These results show how spectral metrics can be used in real life to check the health of a network and how link optimization dynamics, implicit organization, and the quality of learned representation are connected. The findings show that spectral metrics can be used to check the health of a network in real life. They also show a connection between link optimization dynamics, implicit organization, and the quality of learned representations.