Mathematics of Machine Learning M2 MAPI3 (Paul Sabatier, 2025-2026)

Lecturers: Clément Lalanne, Valentin Lafargue

Overview

This class aims to introduce the main theoretical foundations of modern machine learning, with a focus on supervised learning. Basic knowledge of linear algebra and probability theory (including measure theory) is required. For the last lectures, basic knowledge of functional analysis is recommended.

Evaluation

For all students, the final grade will consist of an 80% weight from a 2-hour final exam and 20% from in-class work. For part-time working students (étudiants en alternance), the in-class grade will be based on three MCQs (QCMs en Français). For other students, the in-class grade will be split equally: 50% from MCQs and 50% from the projects.

Survival Kit for the Exam

For the exam, you should be familiar with the following concepts and techniques:

Decomposing risk as an estimation error (considering the uniform worst-case error over the predictor set), approximation error, and potential other error sources (e.g., optimization errors). You should be able to recognize the different bias and variance terms in this decomposition.
Basic linear algebra (vectors, linear maps, rank, matrices) and bilinear algebra (eigenvectors, eigenvalues, spectral theorem, basic matrix decompositions).
Multivariable calculus (Gradients, Hessians, Taylor expansions, higher-order derivative tensors), Convexity, First and Second Order optimality conditions, KKT conditions.
Basic probability theory in general probability spaces, conditional laws, conditional expectations, and independence.
Common inequalities: Triangle inequality, convexity, AM-GM inequality, Cauchy-Schwarz, Jensen, Hölder, Minkowski.
Concentration inequalities: Markov, Bienaymé-Tchebychev and McDarmid's inequality.
Rademacher complexity: Definition, upper bounding the sup deviations (each side by two times the Rademacher complexity), Lipschitz contraction principle, and typical use cases.
Kernel methods: Aronszajn's theorem (equivalence between positive kernels and Hilbert space dot products), representer theorem, kernel trick, operations on kernels, and Bochner's theorem.
Optimization: general optimization priciples and usual regularity assumptions.
More to be continued...

Lectures

Lecture 1
Lecture 2
Lecture 3
Lecture 4
Lecture 5
Lecture 6
Lecture 7
Lecture 8
Lecture 9

TDs / TPs

TD/TP 1
TD/TP 2
TD/TP 3
TD/TP 4
TD/TP 5
TD/TP 6

References and External Resources

Machine Learning and Learning Theory

Learning Theory from First Principles by Francis Bach, 2024: Main reference for the course.
Alice's Adventures in a Differentiable Wonderland -- Volume I, A Tour of the Land by Simone Scardapane, 2024: A very accessible hands on presentation of modern machine learning methods.
Foundations of Machine Learning by Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar, 2018.
Aurélien Garivier's ML M2 class at ENS Lyon: Complementary topics to those presented in this course.

Generative AI

Optimization for Machine Learning

Convex Optimization by Stephen Boyd and Lieven Vandenberghe, Cambridge University Press, 2012.
First-Order Optimization Methods by Amir Beck, 2017.
Acceleration Methods by Alexandre d’Aspremont, Damien Scieur and Adrien Taylor, 2024.

Measure and Probability Theory

Intégration, Probabilités et Processus Aléatoires by Jean-François Le Gall, 2006: An excellent introduction to measure theory with elements of stochastic processes and conditional expectation.
Probabilités 2 by Jean-Yves Ouvrard, 2009: A detailed treatment of advanced topics in probability theory.
Concentration Inequalities: A Nonasymptotic Theory of Independence by Stéphane Boucheron, Gábor Lugosi, and Pascal Massart, 2013.

Analysis

Cours d'analyse by Jean-Michel Bony, 2001.