|
Speaker: Yury Polyanskiy (MIT) “Clustering and phase transitions in transformers” Monday, October 14, 2024 3:30 PM – Pre-talk meet and greet teatime at 219 Prospect Street, 13th floor. There will be light snacks and beverages in the kitchen area. |
Add To: Google Calendar | Outlook | iCal File
Abstract: Most of the recent advances in AI are based around a neural architecture known as Transformer. We describe the process of evolution of representations through the layers of Transformer as an interacting particle system. This dynamical system numerically exhibits an unusual two time-scales behavior:
in the initial (short) phase particles coalesce to form meta-stable clusters, which continue to slowly move, occasionally merging, until eventually collapsing to a single point. We will present rigorous results confirming this behavior in various degrees of generality. Joint work with B. Geshkovski, N. Karagodin, C. Letrouit and P. Rigollet.
Bio: Yury Polyanskiy is a Professor of Electrical Engineering and Computer Science and a member of IDSS and LIDS at MIT. Yury received M.S. degree in applied mathematics and physics from the Moscow Institute of Physics and Technology, Moscow, Russia in 2005 and Ph.D. degree in electrical engineering from Princeton University, Princeton, NJ in 2010. His research interests are in information theory, statistical learning and error-correcting codes.
Dr. Polyanskiy was elected IEEE Fellow (2024), and won the 2020 IEEE Information Theory Society James Massey Award, 2013 NSF CAREER award and 2011 IEEE Information Theory Society Paper Award.
