Uncovering the Limits of Machine Learning for Automatic Vulnerability Detection

Niklas Risse, Marcel Böhme

PDF | Code | arXiv | USENIX Proceedings

TL;DR: Recent results in ML for automatic vulnerability detection (ML4VD) show up to 70% accuracy in identifying security flaws but struggle to distinguish between vulnerable and patched functions, indicating overfitting to unrelated features and poor out-of-distribution generalization. We propose a novel benchmarking methodology to better evaluate ML4VD techniques by augmenting datasets with semantic-preserving transformations and patched code snippets, revealing the limitations of current models in accurately detecting vulnerabilities.