Mathematical Methods for Supervised Learning M1 SID (2024-2025)

Lecturer: Clément Lalanne

Overview

This course introduces mathematical methods essential for supervised learning. Topics include linear models, optimization techniques, and advanced methods like kernel and sparse methods. Students are expected to have basic knowledge of linear algebra, calculus, and probability.

Evaluation

The class will be evaluated by a final exam (70%) and a homework (30%)

Lectures

Lecture 1 (21-01-2025 15h45->17h45): Introduction to Supervised Learning.
Lecture 2 (24-01-2025 7h45->9h45): Linear Models.
Lecture 3 (28-01-2025 7h45->9h45): Elements of Optimization.
Lecture 4 (30-01-2025 15h45->17h45): Validation, Model Selection and Sparse Methods.
Lecture 5 (04-02-2025 7h45->9h45): Classification and Regression Trees + Boosting.
Lecture 6 (07-02-2025 13h30->15h30): Kernel Methods (Jean-Philippe Vert's proof of Aronszajn's theorem).
Lecture 7 (11-02-2025 15h45->17h45):Neural Networks.
Lecture 8 (17-02-2025 10h00->12h00): Questions / Answers before the exam : To maximize efficiency, prepare your questions before the class !

TDs

TD 1 (Date TBD): Elements of Probability and Statistics (solution).
TD 2 (Date TBD): Logistic Regression (solution).
TD 3 (Date TBD): Trees (solution).
TD 4 (Date TBD): Kernel Methods (solution).

Some of the original material was made by François Bachoc and by Adrien Mazoyer.

TPs

TP 1 (Date TBD): Linear Regression, Ridge Regression, and (Cross-)Validation (solution)
TP 2 (Date TBD): Kernel SVM.
TP 3 (Date TBD): TBD

References and External Resources

Machine Learning and Learning Theory

Learning Theory from First Principles by Francis Bach, 2024: Main reference for the course.
Alice's Adventures in a Differentiable Wonderland -- Volume I, A Tour of the Land by Simone Scardapane, 2024: A very accessible hands on presentation of modern machine learning methods.
Foundations of Machine Learning by Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar, 2018.

Optimization for Machine Learning

Convex Optimization by Stephen Boyd and Lieven Vandenberghe, Cambridge University Press, 2012.
First-Order Optimization Methods by Amir Beck, 2017.
Acceleration Methods by Alexandre d’Aspremont, Damien Scieur and Adrien Taylor, 2024.

Measure and Probability Theory

Intégration, Probabilités et Processus Aléatoires by Jean-François Le Gall, 2006: An excellent introduction to measure theory with elements of stochastic processes and conditional expectation.
Probabilités 2 by Jean-Yves Ouvrard, 2009: A detailed treatment of advanced topics in probability theory.
Concentration Inequalities: A Nonasymptotic Theory of Independence by Stéphane Boucheron, Gábor Lugosi, and Pascal Massart, 2013.

Analysis

Cours d'analyse by Jean-Michel Bony, 2001.