Continuous Model Monitoring

After iterations of development and testing, deploying a well-fit machine learning model often feels like the final hurdle for an eager data science team. In practice, however, a trained model is never final, and this milestone marks just the beginning of a new chapter in the ML lifecycle called production ML. This is because most machine learning models are static, but the world we live in is dynamically changing all the time. Changes in environmental conditions like these are referred to as concept drift, and will cause the predictive performance of a model to degrade over time, eventually making it obsolete for the task it was initially intended to solve.

To combat concept drift in production systems, it's important to have robust monitoring capabilities that alert stakeholders when relationships in the incoming data or model have changed. I built this Applied Machine Learning Prototype (AMP) to demonstrate how this can be achieved on Cloudera Machine Learning (CML).

The application leverages CML's Model Metrics feature in combination with Evidently.ai's Data Drift, Numerical Target Drift, and Regression Performance reports packaged into a custom dashboard to monitor a simulated production model that predicts housing prices over time.