Events
Colloquium
"Fisher information and Neural Tangent Kernels"
 |
Speaker: Jun'ichi Takeuchi (Kyushu) Professor Kyushu University Wednesday, October 15, 2025 11:30AM - 1:00PM Lunch will be served in 1307 at 11:30am
Talk will be in 1327 from 12:00-1:00pm Location: Yale Institute for Foundations of Data Science, Kline Tower 13th Floor, Room 1327, New Haven, CT 06511 and via Webcast: https://yale.hosted.panopto.com/Panopto/Pages/Viewer.aspx?id=9684b07d-d72b-4a7c-9dd0-b37200f78723 |
Abstract: We argue relation between neural tangent kernels (NTK) and Fisher information matrices of neural networks. For the Fisher information matrices of two layer ReLU neural networks with random hidden weights, we demonstrated their approximate spectral decomposition, whose eigenvalue distribution highly concentrates (Takeishi et al. 2023). In particular, the sum of the top 3 eigenvalues (they have multiplicity) accounts for 97.7% of the sum of all the eigenvalues no matter how large the number of parameters is. This means that the effective number of parameters is almost dominated by the dimension of eigenspaces of the top 3 eigenvalues. In fact, we derived a tight risk bound (Takeishi and Takeuchi 2024) using Barron and Cover theory for two-stage codes in the minimum description length (MDL) principle. Concerning our decomposition of Fisher information matrices, we recently found that the functions specified by the eigenvectors with top 3 eigenvalues converge to spherical harmonic functions of orders up to 2 as the number of parameters goes to infinity (Ho et al. 2025). This result reveals a certain detailed aspect of the relation between NTK and Fisher information matrices, since the NTK has a decomposition based on spherical harmonic functions shown by Biettio and Mairal (2019). Based on the decomposition, they gave a nice kernel regression scenario in over-parameterization situation. We discuss the relation of Fisher information and NTK of the two layer networks based on these spectral decompositions. Finally, we show a vision to extend our result to more general feedforward neural networks.
Speaker bio: Jun’ichi Takeuchi was born in Tokyo, Japan in 1964. He graduated from the University of Tokyo in majoring physics in 1989. He received the Dr. Eng. degree in mathematical engineering from the University of Tokyo in 1996. From 1989 to 2006, he worked for NEC Corporation, Japan. In 2006, he moved to Kyushu University, Fukuoka, Japan, where he is a Professor. From 1996 to 1997 he was a Visiting Research Scholar at Department of Statistics, Yale University, New Haven, CT, USA. He was involved in the start of the annual workshop series on Information-Based Induction Sciences (IBIS, Japan domestic) in 1998. His research interest includes information theory and machine learning. He is a member of IEEE, IEICE, and JSIAM.
Add To: Google Calendar | Outlook | iCal File
- Colloquium
Submit an Event
Interested in creating your own event, or have an event to share? Please fill the form if you’d like to send us an event you’d like to have added to the calendar.