Neural Collapse in Deep Net Training

Modern deep neural networks are structurally complex and massively over-parameterized. One might then expect these networks to exhibit many particularities with little regularity across architectures and applications. However, through extensive experiments on contest winning deep nets and across benchmark datasets, this talk reveals that a common Neural Collapse phenomenon prevalently emerges in the training of deep classification networks. During Neural Collapse, penultimate-layer features and classifiers of networks collapse to a simple geometric structure called a Simplex Equiangular Tight Frame (ETF), where all vectors are equi-length and maximally separated in the ambient space. Analyzing deep nets through the geometric lens of Neural Collapse has led to insights into important aspects of the modern deep learning such as fairness, adversarial robustness, and generalization; it has also led to the invention of new principles for neural network design.



I frame this work as part of an emerging research interface between machine learning and optimization, driven by the identification and modeling of pervasive phenomena discovered in large-scale experiments. My own research contributions have delivered both real-world solutions as well as intellectual insights. In addition to Neural Collapse, they include the invention of the Survey Descent method for nonsmooth optimization, and collaborations with the Frick Art Reference Library in NYC and the Veolia North America Utilities company.

Share this Event