Events
FDS Seminar Series
An information theoretic view of machine learning
 |
Speaker: Akhil Premkumar (UChicago) Postdoc at the Kavli Institute for Cosmological Physics University of Chicago Thursday, June 26, 2025 1:00PM - 2:00PM Location: Yale Institute for Foundations of Data Science & Webcast, 219 Prospect Street, New Haven, CT 06511 and via Webcast: https://yale.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=84717762-1e59-442e-a09f-b30200e3998a |
Abstract: Diffusion models serve as a bridge between generative AI and information theory. These models have demonstrated a remarkable ability to learn high-dimensional continuous distributions, like images and video, from relatively small training datasets. They can do this because they learn the ensemble statistics of the entire dataset, allowing them to identify long wavelength correlations between the given samples. In other words, these models are extremely good at compression. Drawing on ideas from thermodynamics and information theory, we introduce a method to measure the information content of a neural network, called neural entropy. Measurements of neural entropy in diffusion models reveal that the amount of new information the model learns from each additional sample diminishes with the total number of training samples. We will also draw some connections between probabilistic modeling and games of chance.
Based on: https://arxiv.org/abs/2409.03817v2 and ongoing work.
Speaker Bio: I am a postdoc at the Kavli Institute for Cosmological Physics at the University of Chicago. I did my PhD in theoretical physics at University of California San Diego, where I studied stochastic processes that emerge from combining quantum mechanics and gravity.
Hosted by John Sous.
Add To: Google Calendar | Outlook | iCal File
Submit an Event
Interested in creating your own event, or have an event to share? Please fill the form if you’d like to send us an event you’d like to have added to the calendar.