Mathematics of Machine Learning Projects M2 MAPI3 (Paul Sabatier, 2024-2025)

Important

• Send me an email with your chosen project and the members of your group (one email per group) with the following object : [MAPI3 2024 ML PROJECTS]

Evaluation

• Practical Projects: Choose a challenge from Challenge Data and propose a solution using machine learning techniques. Evaluation will be based on:
  • Your 15 minutes presentation AND code.
  • Problem-Solving Skills: Effectiveness in addressing the chosen challenge.
  • Technical Implementation: Quality and rigor of the machine learning methods used.
  • Presentation Quality: Clarity, organization, and accessibility of the presentation to classmates.
  • Engagement: Ability to engage the audience and facilitate understanding.
  • Mathematical Highlight: Include at least one mathematical highlight to emphasize important mathematical concepts or techniques relevant to the project.
• Theoretical Projects: Explore an advanced ML topic, analyze recent research, and present your findings. Evaluation will be based on:
  • Your 15 minutes presentation and other resources that you wish to share (such as code).
  • Depth of Understanding: Thoroughness in exploring the topic and current research.
  • Critical Analysis: Quality of analysis and critique of different approaches.
  • Clarity of Presentation: Logical structure, use of visual aids, and accessibility to classmates.
  • Presentation Quality: Ability to explain complex concepts in an understandable manner.
  • Engagement: Effectiveness in engaging the audience and fostering interest in the topic.
  • Mathematical Highlight: Include at least one mathematical highlight to emphasize important mathematical concepts or techniques relevant to the topic.

In Class Sessions

• 22-10-2024 : 13h30 -> 16h30 in room U6-402

Students - Projects mapping

• Football : Qui va gagner ? : Alexandre Hervoit, Iyad Zekhnini and Thibault Margelin
• Electricity price predictions : Jean-Pierre Mansour, Paul Pasquet and Racha Rochdi
• Electricity price predictions V2 : Ahmed BA DHAWI, Duc-Trong HOANG and Johan PEYRANNE
• Multimodal Learning : Haritsiky Rajaonah-Rabesoa, Naomi Albukerque and Yan Larrouquere
• Logistic Regression : Arman Sarukhanyan, Kacimi Walid and Makas Yusuf
• Robustness and Adversarial Attacks : Bourgeois Nicolas and Bruisson Mathieu-Anthony
• Anticipez l'affluence au sein des gares SNCF-Transilien : Benjamin Gransard, David Milioutine and Luca Petibout
• Foundation Models and Large Language Models (LLMs) : Noura Osman, Néo De Sa Moreira and Satya Capblancq

Theoretical Projects

• Meta-Learning
  Motivation: Improves learning efficiency by enabling models to learn how to learn from previous tasks, useful for few-shot and fast adaptation scenarios.
  * Finn, C. et al. "Model-agnostic meta-learning for fast adaptation of deep networks." ICML, 2017.
  * Thrun, S., Pratt, L. "Learning to learn." Springer, 1998.
  * Vinyals, O. et al. "Matching networks for one-shot learning." NeurIPS, 2016.
• Self-Supervised Learning
  Motivation: Learns from unlabeled data by creating surrogate tasks, offering a powerful way to scale learning when labeled data is scarce.
  * Doersch, C. et al. "Unsupervised visual representation learning by context prediction." ICCV, 2015.
  * Chen, X. et al. "Big self-supervised models are strong semi-supervised learners." NeurIPS, 2020.
  * Grill, J. B. et al. "Bootstrap your own latent: A new approach to self-supervised learning." NeurIPS, 2020.
• Uncertainty Estimation in ML
  Motivation: Quantifies the uncertainty in model predictions, essential for safety-critical applications and improving model reliability.
  * Gal, Y., Ghahramani, Z. "Dropout as a Bayesian approximation: Representing model uncertainty in deep learning." ICML, 2016.
  * Lakshminarayanan, B. et al. "Simple and scalable predictive uncertainty estimation using deep ensembles." NeurIPS, 2017.
  * Kendall, A. et al. "What uncertainties do we need in Bayesian deep learning for computer vision?" NeurIPS, 2017.
• Causal Inference in ML
  Motivation: Enables models to learn cause-effect relationships from data, important for tasks like decision-making, policy evaluation, and understanding model behavior.
  * Pearl, J. "Causality: Models, reasoning, and inference." Cambridge University Press, 2009.
  * Peters, J. et al. "Elements of causal inference: Foundations and learning algorithms." MIT Press, 2017.
  * Spirtes, P. et al. "Causation, prediction, and search." MIT Press, 2000.
• PAC-Bayesian Generalization Framework
  Motivation: Provides a probabilistic framework to analyze the generalization of machine learning models by connecting prior knowledge with empirical evidence, leading to tighter bounds on generalization error.
  * McAllester, D. A. "Some PAC-Bayesian theorems." COLT, 1998.
  * Germain, P. et al. "PAC-Bayesian theory meets Bayesian inference." NeurIPS, 2016.
  * Catoni, O. "PAC-Bayesian Supervised Classification: The Thermodynamics of Statistical Learning." IMS Lecture Notes, 2007.
• Algorithmic Stability Generalization Framework
  Motivation: Analyzes the sensitivity of a learning algorithm to small changes in the training data, providing a way to establish generalization guarantees based on the stability of the algorithm.
  * Bousquet, O., Elisseeff, A. "Stability and generalization." JMLR, 2002.
  * Hardt, M. et al. "Train faster, generalize better: Stability of stochastic gradient descent." ICML, 2016.
  * Mukherjee, S. et al. "Learning theory: Stability is sufficient for generalization and necessary and sufficient for consistency of empirical risk minimization." Adv. Comput. Math, 2006.
• Foundation Models and Large Language Models (LLMs)
  Motivation: Large pre-trained models, such as GPT and BERT, are increasingly used as foundation models across many domains due to their ability to generalize across tasks with minimal fine-tuning.
  * Brown, T. et al. "Language models are few-shot learners." NeurIPS, 2020.
  * Bommasani, R. et al. "On the opportunities and risks of foundation models." arXiv, 2021.
  * Devlin, J. et al. "BERT: Pre-training of deep bidirectional transformers for language understanding." NAACL-HLT, 2019.
• Differential Privacy in ML
  Motivation: Protects the privacy of individual data points in training sets while allowing models to learn meaningful patterns. It is crucial for legal and ethical compliance in sensitive domains.
  * Dwork, C. et al. "Calibrating noise to sensitivity in private data analysis." TCC, 2006.
  * Abadi, M. et al. "Deep learning with differential privacy." CCS, 2016.
  * Bassily, R. et al. "Private Empirical Risk Minimization." NeurIPS, 2014.
• Robustness and Adversarial Attacks
  Motivation: Ensures models perform reliably under adversarial conditions, such as attacks designed to fool the model into making incorrect predictions.
  * Goodfellow, I. et al. "Explaining and harnessing adversarial examples." ICLR, 2015.
  * Madry, A. et al. "Towards deep learning models resistant to adversarial attacks." ICLR, 2018.
  * Szegedy, C. et al. "Intriguing properties of neural networks." ICLR, 2014.
• Multimodal Learning
  Motivation: Integrates information from multiple data sources or modalities (e.g., text, images, and audio) to create more robust and informed models.
  * Ngiam, J. et al. "Multimodal deep learning." ICML, 2011.
  * Baltrušaitis, T. et al. "Multimodal machine learning: A survey and taxonomy." IEEE TPAMI, 2018.
  * Radford, A. et al. "Learning transferable visual models from natural language supervision." ICML, 2021.
• Fairness in Machine Learning
  Motivation: Develops methods to ensure that machine learning models make unbiased decisions and avoid discriminatory outcomes.
  * Hardt, M. et al. "Equality of opportunity in supervised learning." NeurIPS, 2016.
  * Barocas, S. et al. "Fairness and machine learning." fairmlbook.org, 2019.
  * Binns, R. "Fairness in machine learning: Lessons from political philosophy." Proceedings of the 2018 Conference on Fairness, Accountability, and Transparency, 2018.
• Reinforcement Learning with Function Approximation
  Motivation: Extends reinforcement learning to large-scale or continuous environments using function approximation methods like deep learning.
  * Mnih, V. et al. "Human-level control through deep reinforcement learning." Nature, 2015.
  * Sutton, R. S. et al. "Generalization in reinforcement learning: Successful examples using sparse coarse coding." NeurIPS, 1996.
  * Lillicrap, T. P. et al. "Continuous control with deep reinforcement learning." ICLR, 2016.
• Graph Neural Networks (GNNs)
  Motivation: Models data structured as graphs, allowing for the incorporation of relational and spatial information, useful for many domains like social networks and molecular biology.
  * Kipf, T. N., Welling, M. "Semi-supervised classification with graph convolutional networks." ICLR, 2017.
  * Scarselli, F. et al. "The graph neural network model." IEEE Transactions on Neural Networks, 2008.
  * Hamilton, W. et al. "Inductive representation learning on large graphs." NeurIPS, 2017.
• Nonconvex Optimization in ML
  Motivation: Explores optimization methods for nonconvex functions, which arise frequently in deep learning and other complex machine learning models.
  * Shalev-Shwartz, S. et al. "Stochastic dual coordinate ascent methods for regularized loss." JMLR, 2013.
  * Kingma, D. P., Ba, J. "Adam: A method for stochastic optimization." ICLR, 2015.
  * Bottou, L. et al. "Optimization methods for large-scale machine learning." SIAM Review, 2018.
• Bayesian Deep Learning
  Motivation: Combines the uncertainty quantification benefits of Bayesian methods with the expressive power of deep learning, providing robust uncertainty estimates.
  * Blundell, C. et al. "Weight uncertainty in neural networks." ICML, 2015.
  * Neal, R. "Bayesian learning for neural networks." Springer, 2012.
  * Gal, Y., Ghahramani, Z. "Bayesian convolutional neural networks with Bernoulli approximate variational inference." ICLR, 2016.
• AutoML (Automated Machine Learning)
  Motivation: Automates the process of model selection, hyperparameter tuning, and feature engineering, making ML more accessible and efficient.
  * Zoph, B., Le, Q. V. "Neural architecture search with reinforcement learning." ICLR, 2017.
  * Feurer, M. et al. "Auto-sklearn: Efficient and robust automated machine learning." NeurIPS, 2015.
  * Hutter, F. et al. "Automated machine learning: Methods, systems, challenges." Springer, 2019.
• Active Learning
  Motivation: Selectively queries the most informative data points for labeling, reducing the amount of labeled data needed to train models.
  * Settles, B. "Active learning literature survey." University of Wisconsin-Madison, 2010.
  * Cohn, D. A. et al. "Active learning with statistical models." JMLR, 1996.
  * Sener, O., Savarese, S. "Active learning for convolutional neural networks: A core-set approach." ICLR, 2018.
• Sparse Learning
  Motivation: Introduces sparsity in model weights or data, leading to more interpretable models and reducing computational complexity.
  * Tibshirani, R. "Regression shrinkage and selection via the lasso." J. Royal Stat. Soc., 1996.
  * Lee, N. et al. "SNIP: Single-shot network pruning based on connection sensitivity." ICLR, 2019.
  * Zhang, T. et al. "Sparse coding and its applications in computer vision." Journal of Visual Communication and Image Representation, 2010.
• Contrastive Learning
  Motivation: Learns useful representations by maximizing similarity between positive pairs while minimizing it between negative pairs, often in self-supervised settings.
  * Hadsell, R. et al. "Dimensionality reduction by learning an invariant mapping." CVPR, 2006.
  * Chen, T. et al. "A simple framework for contrastive learning of visual representations." ICML, 2020.
  * He, K. et al. "Momentum contrast for unsupervised visual representation learning." CVPR, 2020.
• Energy-Based Models
  Motivation: Uses energy functions to model complex dependencies between variables, allowing for a unified view of various ML tasks like classification and generation.
  * LeCun, Y. et al. "A tutorial on energy-based learning." Predicting Structured Data, 2006.
  * Ng, A. Y., Jordan, M. I. "On discriminative vs. generative classifiers: A comparison of logistic regression and naive Bayes." NeurIPS, 2002.
  * Hinton, G. E. "Training products of experts by minimizing contrastive divergence." Neural Computation, 2002.
• Few-Shot Learning
  Motivation: Addresses the problem of learning from only a few labeled examples, making it highly valuable in settings where data collection is expensive or time-consuming.
  * Lake, B. M. et al. "Human-level concept learning through probabilistic program induction." Science, 2015.
  * Vinyals, O. et al. "Matching networks for one-shot learning." NeurIPS, 2016.
  * Snell, J. et al. "Prototypical networks for few-shot learning." NeurIPS, 2017.
• Neural Architecture Search (NAS)
  Motivation: Automates the design of neural network architectures, often outperforming manually crafted models and leading to more efficient architectures.
  * Zoph, B., Le, Q. V. "Neural architecture search with reinforcement learning." ICLR, 2017.
  * Liu, H. et al. "DARTS: Differentiable architecture search." ICLR, 2019.
  * Real, E. et al. "Regularized evolution for image classifier architecture search." AAAI, 2019.
• Imitation Learning
  Motivation: Enables agents to learn behaviors by imitating expert demonstrations, useful in robotics, gaming, and autonomous systems.
  * Ross, S. et al. "A reduction of imitation learning and structured prediction to no-regret online learning." AISTATS, 2011.
  * Ho, J., Ermon, S. "Generative adversarial imitation learning." NeurIPS, 2016.
  * Pomerleau, D. A. "Efficient training of artificial neural networks for autonomous navigation." Neural Computation, 1991.
• Invariant Representation Learning
  Motivation: Learns representations that are invariant to specific transformations (e.g., rotation, scaling), improving model generalization across tasks.
  * Lenc, K., Vedaldi, A. "Understanding image representations by measuring their equivariance and equivalence." CVPR, 2015.
  * Wang, X. et al. "Invariant information clustering for unsupervised image classification and segmentation." ICCV, 2019.
  * Cohen, T. S., Welling, M. "Group equivariant convolutional networks." ICML, 2016.
• Transfer Learning
  Motivation: Reuses a pre-trained model for a new, related task, significantly reducing the need for large amounts of data and computational resources.
  * Pan, S. J., Yang, Q. "A survey on transfer learning." IEEE TKDE, 2010.
  * Yosinski, J. et al. "How transferable are features in deep neural networks?" NeurIPS, 2014.
  * Bengio, Y. "Deep learning of representations for unsupervised and transfer learning." ICML Workshop on Unsupervised and Transfer Learning, 2012.
• Neural Ordinary Differential Equations (Neural ODEs)
  Motivation: Models continuous-time dynamics with neural networks, enabling the solution of complex dynamical systems and memory-efficient learning.
  * Chen, T. Q. et al. "Neural ordinary differential equations." NeurIPS, 2018.
  * Dupont, E. et al. "Augmented neural ODEs." NeurIPS, 2019.
  * Rubanova, Y. et al. "Latent ordinary differential equations for irregularly-sampled time series." NeurIPS, 2019.
• Deep Reinforcement Learning in Continuous Control
  Motivation: Applies deep reinforcement learning to environments with continuous action spaces, which is crucial for applications like robotics and autonomous driving.
  * Lillicrap, T. P. et al. "Continuous control with deep reinforcement learning." ICLR, 2016.
  * Silver, D. et al. "Deterministic policy gradient algorithms." ICML, 2014.
  * Haarnoja, T. et al. "Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor." ICML, 2018.
• Continual Learning
  Motivation: Allows models to learn continuously from a stream of data, enabling them to retain and integrate knowledge without forgetting past information.
  * Kirkpatrick, J. et al. "Overcoming catastrophic forgetting in neural networks." PNAS, 2017.
  * Lopez-Paz, D., Ranzato, M. "Gradient episodic memory for continual learning." NeurIPS, 2017.
  * Parisi, G. I. et al. "Continual lifelong learning with neural networks: A review." Neural Networks, 2019.
• Representation Learning for Graphs
  Motivation: Focuses on learning useful embeddings for graph-structured data, crucial for applications like social networks, biological networks, and knowledge graphs.
  * Hamilton, W. et al. "Inductive representation learning on large graphs." NeurIPS, 2017.
  * Kipf, T. N., Welling, M. "Variational graph auto-encoders." NeurIPS, 2016.
  * Grover, A., Leskovec, J. "node2vec: Scalable feature learning for networks." KDD, 2016.
• Federated Learning
  Motivation: Trains models across decentralized devices or servers holding local data samples without sharing them, enhancing privacy while still enabling collaborative learning.
  * McMahan, B. et al. "Communication-efficient learning of deep networks from decentralized data." AISTATS, 2017.
  * Konecny, J. et al. "Federated learning: Strategies for improving communication efficiency." NeurIPS Workshop, 2016.
  * Yang, Q. et al. "Federated learning: Challenges, methods, and future directions." IEEE SPM, 2019.
• Neuro-Symbolic AI
  Motivation: Combines neural networks with symbolic reasoning to integrate learning and logic, aiming for explainability and robustness in AI systems.
  * Mao, J. et al. "The neuro-symbolic concept learner: Interpreting scenes, words, and sentences from natural supervision." ICLR, 2019.
  * Garcez, A. S. d. et al. "Neural-symbolic learning and reasoning: Contributions and challenges." AAAI, 2015.
  * Besold, T. R. et al. "Neural-symbolic learning and reasoning: A survey and interpretation." Frontiers in AI, 2017.
• Prompt Engineering
  Motivation: With the rise of large language models, prompt engineering has become crucial for guiding these models to perform specific tasks effectively.
  * Liu, P. et al. "Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing." arXiv, 2021.
  * Reynolds, L., McDonell, K. "Prompt programming for large language models: Beyond the few-shot paradigm." arXiv, 2021.
  * Schick, T., Schütze, H. "Exploiting cloze-questions for few-shot text classification and natural language inference." EACL, 2021.
• Explainable AI (XAI)
  Motivation: As machine learning models, especially deep neural networks, become more complex, explainability is crucial for ensuring transparency, trust, and adoption in real-world applications.
  * Ribeiro, M. T. et al. "Why should I trust you? Explaining the predictions of any classifier." KDD, 2016.
  * Samek, W. et al. "Explainable AI: Interpreting, explaining and visualizing deep learning models." ITU Journal, 2017.
  * Doshi-Velez, F., Kim, B. "Towards a rigorous science of interpretable machine learning." arXiv, 2017.
• AI Alignment and Ethics
  Motivation: AI alignment ensures that powerful AI systems act in ways that are beneficial to humans, addressing safety and ethical concerns as these technologies become more integrated into society.
  * Russell, S. "Human compatible: Artificial intelligence and the problem of control." Viking, 2019.
  * Bostrom, N. "Superintelligence: Paths, dangers, strategies." Oxford University Press, 2014.
  * Gabriel, I. "Artificial intelligence, values, and alignment." Minds and Machines, 2020.
• Neuromorphic Computing
  Motivation: Inspired by the structure of the human brain, neuromorphic computing aims to develop hardware that mimics neural processes, enabling more efficient AI with low energy consumption.
  * Indiveri, G. et al. "Neuromorphic silicon neurons: A survey of models, learning methods, and applications." IEEE, 2011.
  * Davies, M. et al. "Loihi: A neuromorphic manycore processor with on-chip learning." IEEE Micro, 2018.
  * Merolla, P. A. et al. "A million spiking-neuron integrated circuit with a scalable communication network and interface." Science, 2014.
• Quantum Machine Learning
  Motivation: Quantum computing promises exponential speedups for certain types of computations, which could revolutionize machine learning by solving problems intractable for classical computers.
  * Biamonte, J. et al. "Quantum machine learning." Nature, 2017.
  * Schuld, M., Petruccione, F. "Supervised learning with quantum computers." Springer, 2018.
  * Havlíček, V. et al. "Supervised learning with quantum-enhanced feature spaces." Nature, 2019.