About the Course
In this course, we will delve into the mechanisms of neural networks through a combination of experimental observations and theoretical analyses. Specifically, we will examine significant experimental findings that have shaped theoretical advancements in machine learning and present the theoretical frameworks that explain these observations.
The image below summarizes the topics we plan to cover. We will begin with the central topic and navigate between theory and observation as the course progresses. Starting with shallow neural networks with a single layer, we will advance to discussions on deep convolutional networks and transformers.
Note: This course is not designed to enhance implementation skills but rather to introduce open-ended research questions in the understanding of neural networks.
Requirements: Students are expected to have a foundational understanding of linear algebra, statistics, probability theory, and calculus and can program in Python. While required materials will be reviewed during the course, these prerequisites are essential for successful participation and comprehension.
Course Schedule and Notes
- Shallow neural networks: Bridging random features, kernel methods and neural networks
- Universal random features (slides, in-class task, solution for in-class tasks, take-home exercise )
- Single-layer neural networks
- Deep neural networks: generalization and overfitting
- Optimizatio challenges: non-convexity and stochastic gradient approximation
- Bridgining neural networks and Turing machine: In-context learning from practice to theory
- Transformers: esssence of normalization, residual connections and attention layers
- Generative models and saddle point optimization
- Additional topics: adversarial examples, reinforcement learning for langauge models