Back to Upcoming Events
FDS Colloquium

High-dimensional Optimization with Applications to Compute-Optimal Neural Scaling Laws

Speaker: Elliot Paquette (McGill)

Associate Professor
Mathematics

McGill University

Wednesday, April 16, 2025

11:30AM - 1:00PM

Lunch at 11:30am in room 1307
Talk at 12:00pm in room 1327

Location: Yale Institute for Foundations of Data Science, Kline Tower 13th Floor, Room 1327, New Haven, CT 06511 and via Webcast: https://yale.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=10b62899-39a8-4750-b55d-b2a400f4c19c

Abstract: Given the massive scale of modern ML models, we now only get a single shot to train them effectively. This restricts our ability to test multiple architectures and hyper-parameter configurations. Instead, we need to understand how these models scale, allowing us to experiment with smaller problems and then apply those insights to larger-scale models. In this talk, I will present a framework for analyzing scaling laws in stochastic learning algorithms using a power-law random features model, leveraging high-dimensional probability and random matrix theory. I will then use this scaling law to address the compute-optimal question: How should we choose model size and hyper-parameters to achieve the best possible performance in the most compute-efficient manner?

Based on joint work with Courtney Paquette (McGill & Google Deepmind), Jeffrey Pennington (Google Deepmind), and Lechao Xiao (Google Deepmind).

Speaker Bio: Elliot Paquette is a researcher in probability theory, and especially theory of high dimensional optimization and machine learning.  He received his PhD in 2013 at the University of Washington, in random matrix theory.  He was an NSF postdoctoral researcher at Weizmann Institute of Science, working at the interface of random matrix theory and branching processes.  In 2016, he joined Ohio State University as an assistant professor.  In 2020, he moved to McGill University, department of Mathematics and Statistics, where he is now associate professor.

elliotpaquette.github.io

Add To: Google Calendar | Outlook | iCal File

Submit an Event

Interested in creating your own event, or have an event to share? Please fill the form if you’d like to send us an event you’d like to have added to the calendar.

Submit an Event

Share your event ideas with us using the form below.

"*" indicates required fields

MM slash DD slash YYYY
Start Time*
:
End Time*
: