Universality of regularized regression estimators in high dimensions

The study of regularized regression estimators is of fundamental importance in high dimensional
inference. From the canonical linear model to many of its extensions, a by now well-established type of
analysis has generated a remarkable suite of results on risk and generalization bounds, support recovery,
oracle inequalities, and many others. However, this type of analysis typically has a worst-case/minimax
flavor and is usually only accurate up to multiplicative constants, thereby still falling short of capturing
the precise stochastic behavior of various statistical estimators. In contrast, a recent line of work has
focused on pointwise exact analysis, which leads to an exact characterization of key properties of an
estimator, such as its risk or distribution. Unfortunately, this second type of analysis has thus far been
mostly limited to the setting of a Gaussian design matrix, because it relies on Gaussian-specific
techniques such as the Convex Gaussian Min-Max Theorem. In this talk, focusing on the high
dimensional linear model, I will establish a universality framework that enables us to extend pointwise
exact analysis to the setting of a non- Gaussian design matrix. The key ingredient is a Gaussian
approximation argument for the objective function associated with the estimator, after which existing
Gaussian analysis based on the Convex Gaussian Min-Max Theorem can be applied. I will then apply this
new framework to well-known regularized estimators, such as the (debiased) lasso, ridge, and robust
regression, in order to obtain guarantees on the risk, distribution, confidence intervals, residuals, and
other key quantities of interest. This is based on joint work with Qiyang Han at Rutgers University.

Share this Event