Survey Descent: A Multipoint Generalization of Gradient Descent for Nonsmooth Optimization

First-order minimization algorithms, which rely only on blackbox (sub)gradient oracles, have always played a central role in the field of optimization, and are now even more prominent due to widespread application in machine learning. However, a gap persists between theoretical understanding of such methods and empirical observation: frequent experimental observations of fast, linear convergence is supported in classical theory only for smooth, strongly convex objectives. On nonsmooth objectives, traditional approaches like the subgradient method, mirror descent, and bundle methods enjoy only sublinear guarantees, even assuming strong convexity. Motivated by this gap, this talk proposes a multipoint generalization of the classical gradient descent method, called Survey Descent. Linear convergence is guaranteed on “max-of-smooth” objectives, behavior that experiments suggest is a broader phenomenon.
I frame this work as part of an emerging research interface between machine learning and optimization, driven by the identification and modeling of pervasive phenomena discovered in large-scale experiments. My own research contributions have delivered both real-world solutions as well as intellectual insights. In addition to Survey Descent, they include the discovery of the now-widely-studied Neural Collapse phenomenon in deep net training, and collaborations with the Frick Art Reference Library in NYC and the Veolia North America Utilities company.

Share this Event