09-25, 14:40–15:15 (Europe/Amsterdam), Nebula
In today’s financial sector, the continuous accuracy and reliability of machine learning models are crucial for operational efficiency and effective risk management. With the rise of MLOps (Machine Learning Operations), automating monitoring mechanisms has become essential to ensure model performance and compliance with regulations. This presentation introduces a method for continuous monitoring of model drift, highlighting the benefits of automation within the MLOps framework. This topic is particularly interesting because it addresses a common challenge in maintaining model performance over time and demonstrates a practical solution that has been successfully implemented in the bank.
This talk is aimed at data scientists, machine learning engineers, and MLOps practitioners who are interested in automating the monitoring of machine learning models. Attendees will be guided on how to continuous monitor model drift within the MLOps framework. They will understand the benefits of automation in this context, and gain insights into MLOps best practices. A basic understanding of MLOps principles, and statistical techniques for model evaluation will be helpful but not strictly needed.
The presentation will be an informative talk with a focus on the design and implementation. It will include some mathematical concepts but will primarily be demonstrating real-world applications and best practices. At the end we encourage you to actively monitor model drift and automate your monitoring processes to enhance model accuracy, scalability, and compliance in your organizations.
In this session, we will talk about the integration of automated model drift detection within the MLOps framework, focusing on best practices and practical implementation. Our goal is to demonstrate how continuous monitoring can enhance the accuracy, scalability, and compliance of machine learning models in the financial sector. We will begin by introducing the concept of MLOps (Machine Learning Operations) and its best practices, highlighting the importance of monitoring machine learning models, with a particular emphasis on detecting and managing model drift [5 minutes]. Next, we will outline the key requirements for effective model drift monitoring needed to ensure that models remain accurate and reliable over time [2 minutes]. We will then introduce the specific machine learning model that will be subjected to our monitoring solution, providing context on the model's purpose and its significance within the bank [2 minutes].
Following the model introduction, we will explain the framework we created to develop our monitoring solution, covering the design principles, architecture, and the rationale behind our approach [5 minutes]. In this section, we will present the implementation of our monitoring solution, demonstrating how we integrated it into our CI/CD pipeline [4 minutes], developed the necessary code, and deployed it on Databricks. This will include a step-by-step walkthrough of the technical setup [4 minutes]. We will showcase the final dashboard that presents the results of our monitoring solution, highlighting key metrics and insights derived from continuous model monitoring [4 minutes]. Finally, we will discuss the initial results and the valuable learnings we gained from implementing our monitoring solution [3 minutes].