Mathematics of Machine Learning M2 MAPI3 (University of Toulouse, 2026-2027)

Lecturers: Clément Lalanne, Valentin Lafargue

⚠ Séance annulée le lundi 12 octobre Malheureusement, la séance du lundi 12 octobre est annulée en raison du mouvement de mobilisation étudiante en France. En particulier, le TP avec QCM noté (QCM 1) prévu ce jour-là est reporté à une date ultérieure, qui vous sera communiquée. Nous verrons comment rattraper cette séance.

Overview

This class aims to introduce the main theoretical foundations of modern machine learning, with a focus on supervised learning. Basic knowledge of linear algebra and probability theory (including measure theory) is required. For the last lectures, basic knowledge of functional analysis is recommended.

Evaluation

For all students, the final grade will consist of 80% from a 2-hour final exam and 20% from in-class work. For part-time working students (étudiants en alternance), the in-class grade will be based on three MCQs (QCMs en Français). For other students, the in-class grade will be split equally: 50% from MCQs and 50% from the projects.

QCM 1 : ~~12 octobre 2026~~ reporté à une date ultérieure, TPs et méthodes expérimentales (TP1 au programme).
QCM 2 : 4 novembre 2026, cours jusqu'aux réseaux de neurones.

Survival Kit for the Exam

For the exam, you should be familiar with the following concepts and techniques:

Decomposing risk as an estimation error (considering the uniform worst-case error over the predictor set), approximation error, and potential other error sources (e.g., optimization errors). You should be able to recognize the different bias and variance terms in this decomposition.
Basic linear algebra (vectors, linear maps, rank, matrices) and bilinear algebra (eigenvectors, eigenvalues, spectral theorem, basic matrix decompositions).
Multivariable calculus (Gradients, Hessians, Taylor expansions, higher-order derivative tensors), Convexity, First and Second Order optimality conditions, KKT conditions.
Basic probability theory in general probability spaces, conditional laws, conditional expectations, and independence.
Common inequalities: Triangle inequality, convexity, AM-GM inequality, Cauchy-Schwarz, Jensen, Hölder, Minkowski.
Concentration inequalities: Markov, Bienaymé-Tchebychev and McDiarmid's inequality.
Rademacher complexity: Definition, upper bounding the sup deviations (each side by two times the Rademacher complexity), Lipschitz contraction principle, and typical use cases.
Kernel methods: Aronszajn's theorem (equivalence between positive kernels and Hilbert space dot products), representer theorem, kernel trick, operations on kernels, and Bochner's theorem.
Optimization: general optimization principles and usual regularity assumptions.

Lectures

Lecture 1 Introduction to supervised learning.
Lecture 2 Introduction to supervised learning continued.
Lecture 3 Linear regression and Ridge regression.
Lecture 4 Kernel methods (Jean-Philippe Vert's proof of Aronszajn's theorem).
Lecture 5 ERM 1 (Concentration inequalities intro).
Lecture 6 ERM 2.
Lecture 7 Optimization for Machine Learning and neural networks.
Lecture 8
Lecture 9

TDs / TPs

TD/TP 1 Linear Regression, Ridge Regression, and (Cross-)Validation (solution).
TD/TP 2 Kernel Methods TP (solution).
TD/TP 3 Optimization and Chain Rule (solution).
TD/TP 4 Logistic Regression (solution).
TD/TP 5 Kernel Methods (solution).
TD/TP 6 PyTorch (solution).

References and External Resources

Machine Learning and Learning Theory

Learning Theory from First Principles by Francis Bach, 2024: Main reference for the course.
Alice's Adventures in a Differentiable Wonderland -- Volume I, A Tour of the Land by Simone Scardapane, 2024: A very accessible hands-on presentation of modern machine learning methods.
Foundations of Machine Learning by Mehryar Mohri, Afshin Rostamizadeh, and Ameet Talwalkar, 2018.
Aurélien Garivier's ML M2 class at ENS Lyon: Complementary topics to those presented in this course.

Generative AI

Deepia's YouTube channel.
Tutorial on Diffusion Models for Imaging and Vision by Stanley Chan, 2024.
What are Diffusion Models? by Lilian Weng, 2021.
A Visual Dive into Conditional Flow Matching by Gagneux et al., 2025.

Optimization for Machine Learning

Convex Optimization by Stephen Boyd and Lieven Vandenberghe, Cambridge University Press, 2012.
First-Order Optimization Methods by Amir Beck, 2017.
Acceleration Methods by Alexandre d’Aspremont, Damien Scieur and Adrien Taylor, 2024.

Measure and Probability Theory

Intégration, Probabilités et Processus Aléatoires by Jean-François Le Gall, 2006: An excellent introduction to measure theory with elements of stochastic processes and conditional expectation.
Probabilités 2 by Jean-Yves Ouvrard, 2009: A detailed treatment of advanced topics in probability theory.
Concentration Inequalities: A Nonasymptotic Theory of Independence by Stéphane Boucheron, Gábor Lugosi, and Pascal Massart, 2013.

Analysis

Cours d'analyse by Jean-Michel Bony, 2001.