Events
FDS Colloquium
High-dimensional Optimization with Applications to Compute-Optimal Neural Scaling Laws
 |
Speaker: Elliot Paquette (McGill) Associate Professor McGill University Wednesday, April 16, 2025 11:30AM - 1:00PM Lunch at 11:30am in room 1307
Talk at 12:00pm in room 1327 Location: Yale Institute for Foundations of Data Science, Kline Tower 13th Floor, Room 1327, New Haven, CT 06511 and via Webcast: https://yale.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=60cf5f95-dedb-4a90-b2fd-b2c000ea1bd9 |
Abstract: Given the massive scale of modern ML models, we now only get a single shot to train them effectively. This restricts our ability to test multiple architectures and hyper-parameter configurations. Instead, we need to understand how these models scale, allowing us to experiment with smaller problems and then apply those insights to larger-scale models. In this talk, I will present a framework for analyzing scaling laws in stochastic learning algorithms using a power-law random features model, leveraging high-dimensional probability and random matrix theory. I will then use this scaling law to address the compute-optimal question: How should we choose model size and hyper-parameters to achieve the best possible performance in the most compute-efficient manner?
Based on joint work with Courtney Paquette (McGill & Google Deepmind), Jeffrey Pennington (Google Deepmind), and Lechao Xiao (Google Deepmind).
Speaker Bio: Elliot Paquette is a researcher in probability theory, and especially theory of high dimensional optimization and machine learning. He received his PhD in 2013 at the University of Washington, in random matrix theory. He was an NSF postdoctoral researcher at Weizmann Institute of Science, working at the interface of random matrix theory and branching processes. In 2016, he joined Ohio State University as an assistant professor. In 2020, he moved to McGill University, department of Mathematics and Statistics, where he is now associate professor.
Add To: Google Calendar | Outlook | iCal File
Submit an Event
Interested in creating your own event, or have an event to share? Please fill the form if you’d like to send us an event you’d like to have added to the calendar.