Common ML Failure Modes

Understanding ML Failure Modes

Recognizing how ML systems fail is crucial for building safer AI.

Overfitting and Underfitting

• Memorization vs generalization
• Bias-variance tradeoff
• Regularization techniques
• Safety implications of poor generalization

Distribution Shift

• Training vs deployment distributions
• Covariate shift and concept drift
• Out-of-distribution detection
• Robustness to distributional changes

Other Critical Failures

• Adversarial examples and robustness
• Spurious correlations and shortcuts
• Catastrophic forgetting
• Reward hacking in RL systems

Mitigation Strategies

• Robust training techniques
• Uncertainty estimation
• Safe deployment practices
• Monitoring and detection systems