, 11:30–12:30
Why Adam Works in Optimizing Language Models: Heavy-Tailed Data Imbalance
Frederik Kunstner (INRIA) on why plain gradient descent struggles to train language models while Adam does not — tracing the gap to Zipf-law imbalance in language data, with empirical results and a proof on a linear bigram model.
- Dates
- Time
- 11:30–12:30
- City
- Zurich
- Venue
- ETH AI Center, OAT X 11, Andreasstrasse 5, Oerlikon
- Organiser
- ETH AI Center
- Format
- Meetup
- Cost
- Free (registration required)
- Topics
- LLMs, Machine learning, AI research
- Verified via
- ai.ethz.ch