MODULES 1: Machine Learning and AI: A Model Risk Perspective

• Drivers of Model Risk in the age of data science and AI
• Machine Learning vs Traditional quant models
• How has the world changed?
• A tour of Machine Learning and AI methods
• Supervised vs Unsupervised Learning (Regression, Neural Networks, XGBoost, PCA, Clustering)
• Deep Learning & Reinforcement Learning (Keras, Tensorflow, PyTorch)
• Automatic Machine Learning & Machine Learning APIs (Google, Comprehend, Watson)
• ML on the cloud vs On-prem
• Models redefined: Data, Modeling environment, Modeling tools, Modeling process

MODULES 2: Model Risk Management for Machine Learning Models - Part 1

• ML Life cycle management
• Tracking
• Metadata management
• Scaling
• Reproducibility
• Interpretability
• Testing
• Measurement

The Decalogue: Ten key aspects to factor when developing your model risk management framework when integrating Machine Learning models:

• Models redefined: It’s not just input, process and output
• Governing the Machine Learning process
• Model Verification and Validation for Machine Learning Models
• Performance Metrics and Evaluation criteria
• Model Inventory and tracking

MODULES 3: Model Risk Management for Machine Learning Models - Part 2

The Decalogue: Ten things to think about when developing your model governance framework when integrating Machine Learning models (cont’d):

• Integrating Data Governance and Model Governance
• Development Models vs Production Models
• Fairness, Reproducibility, Auditability, Explainability, Interpretability & Bias
  • How do we objectively measure these?
  • Review of the Apple-Goldman Sachs credit card debacle
• Machine Learning options and considerations
  • AutoML (Data Robot, H20.ai, etc.), ML as a service (Google, Comprehend, Watson) and home-cooked custom models
• ML and Governance: Roles and Responsibilities redefined

Managing models in the day of Covid19

• Perspectives on point-forecasts, validation and fat-tails!

MODULES 4: Pragmatic Model Risk Management for AI/ML models

• Challenges and best practices for pragmatic model management within the enterprise
• Working with open source projects
• Working with vendor models and machine learning APIs
• Quantifying model risk for machine learning models
• Model risk management for deep-learning models
• Validation criteria and best practices
• Templates for Model Validation for machine learning models

Synthetic data for Model Risk Management

• Use of Synthetic datasets

MODULES 5: Hands-on Case study

Learn from the past: How does Supervised machine learning work?

• Validating a Credit-risk machine learning model A case study illustrating a model validation of a credit risk model involving machine learning
• Working with Regression, Neural Networks, and Random Forest models
• Development models vs Production models
• Sample templates and worksheets will be provided
• Roadmap for the MRM team to upskill and keep abreast of changes in the AI and ML landscape Training, education, and expectation setting Future outlook: Regulation, Sandboxes, Frameworks
• Review of recent regulatory efforts
• How should companies proactively plan for changes and the future?

MODULES 6: Guided Exercise, Part 1: Scoping and design

Put your newly learned skills to practice while being mentored through the process. Participants will go through a guided exercise to perform model validation on a machine learning model of their choice. Guidance will be provided in scoping and implementing the project.

MODULES 7: Guided Exercise, part 2: Demonstrate your skill

Participants will demonstrate their findings to the class and obtain feedback from instructors and industry participants.