BEGIN:VCALENDAR
VERSION:2.0
PRODID:-//wp-events-plugin.com//7.4.0.1//EN
TZID:America/New_York
X-WR-TIMEZONE:America/New_York
BEGIN:VEVENT
UID:853@fds.yale.edu
DTSTART;TZID=America/New_York:20250328T113000
DTEND;TZID=America/New_York:20250328T130000
DTSTAMP:20250916T142148Z
URL:https://fds.yale.edu/events/sds-seminar-blake-bordelon-harvard-scaling
 -limits-and-scaling-laws-of-deep-learning/
SUMMARY:SDS Seminar: Blake Bordelon (Harvard)\, "Scaling Limits and Scaling
  Laws of Deep Learning"
DESCRIPTION:\n Abstract: Scaling up the size and training horizon of deep 
 learning models has enabled breakthroughs in computer vision and natural l
 anguage processing. Empirical evidence suggests that these neural network 
 models are described by regular scaling laws where performance of finite p
 arameter models improves as model size increases\, eventually approaching 
 a limit described by the performance of an infinite parameter model. In th
 is talk\, we will first examine certain infinite parameter limits of deep 
 neural networks which preserve representation learning and then describe h
 ow quickly finite models converge to these limits. Using dynamical mean fi
 eld theory methods\, we provide an asymptotic description of the learning 
 dynamics of randomly initialized infinite width and depth networks. Next\,
  we will empirically investigate how close the training dynamics of finite
  networks are to these idealized limits. Lastly\, we will provide a theore
 tical model of neural scaling laws which describes how generalization depe
 nds on three computational resources: training time\, model size and data 
 quantity. This theory allows analysis of compute optimal scaling strategie
 s and predicts how model size and training time should be scaled together 
 in terms of spectral properties of the limiting kernel. The theory also pr
 edicts how representation learning can improve neural scaling laws in cert
 ain regimes. For very hard tasks\, the theory predicts that representation
  learning can approximately double the training-time exponent compared to 
 the static kernel limit.\n\n\n\nBio: Blake Bordelon is a PhD student in A
 pplied Math at Harvard University where he researches the theory of natura
 l and artificial neural networks. \n
CATEGORIES:FDS Events,Statistics & Data Science Seminar
LOCATION:Yale Institute for Foundations of Data Science\, Kline Tower 13th 
 Floor\, Room 1327\, New Haven\, CT\, 06511\, United States
X-APPLE-STRUCTURED-LOCATION;VALUE=URI;X-ADDRESS=Kline Tower 13th Floor\, Ro
 om 1327\, New Haven\, CT\, 06511\, United States;X-APPLE-RADIUS=100;X-TITL
 E=Yale Institute for Foundations of Data Science:geo:0,0
END:VEVENT
BEGIN:VTIMEZONE
TZID:America/New_York
X-LIC-LOCATION:America/New_York
BEGIN:DAYLIGHT
DTSTART:20250309T030000
TZOFFSETFROM:-0500
TZOFFSETTO:-0400
TZNAME:EDT
END:DAYLIGHT
END:VTIMEZONE
END:VCALENDAR