Beyond F1: Evaluating Coverage and Failure Recovery in AI Model Security Scanners

AI Digest - ArXiv AI

Beyond F1: Evaluating Coverage and Failure Recovery in AI Model Security Scanners

Static scanners are increasingly used to identify executable or otherwise unsafe content in machine- learning artifacts, yet conventional evaluation metrics characterize only cases where a scanner yields a usable security judgment. We evaluate ModelScan, ModelAudit, and Fickling using a controlled, artifact-backed benchmark on a synthetic corpus of 170 Pickle and PyTorch focused artifacts across 145 specimen families, 135 of which have binary security ground truth and 10 of which are intentional

Source: ArXiv AI