EU AI Act Article 15: Accuracy, Robustness, and Cybersecurity Requirements

"It went through our normal security review" is what a security team usually says when asked whether a high-risk AI system has been tested for Article 15. The ML team, asked the same question, tends to point back at security. Both answers are half right, and the gap between them is exactly where Article 15 compliance tends to actually fail — because the article requires threat coverage that sits outside what either team's default process was built to test.

Accuracy: measured, declared, and maintained — not validated once

The Act requires high-risk systems to achieve an appropriate level of accuracy, with real metrics behind that claim, and — critically — to perform consistently at that level throughout the system's lifecycle, not just at the moment it was validated before launch. This is where "we tested it before shipping" quietly stops being sufficient: a model's accuracy on real-world data can and does drift after deployment, as the population it encounters shifts away from what it was trained and validated on, and Article 15 treats that drift as a compliance problem, not just an ML-engineering nuisance to fix opportunistically.

The accuracy level and its metrics aren't just an internal engineering artifact, either — they have to be declared in the instructions for use required under Article 13. A team that hasn't settled on a specific, defensible accuracy metric for its system has a documentation gap in two places at once, not one.

Robustness: fail-safes and the feedback-loop problem

Robustness means the system has to be as resilient as possible to errors, faults, and inconsistencies that arise within the system itself or in its operating environment — including from how it interacts with people and other systems — backed by real technical and organizational measures, including redundancy solutions like backup and fail-safe plans where appropriate.

One specific robustness requirement gets missed more than any other: for systems that continue learning after deployment, the Act requires development that eliminates or reduces, as far as possible, the risk of biased outputs feeding back in as training data for future versions of the system — a feedback loop that can quietly amplify an initial bias with every retraining cycle. A system that retrains on its own outputs without a deliberate check against this is building exactly the risk the Act is trying to prevent, one iteration at a time, often without anyone noticing until the drift is already substantial.

Cybersecurity: the AI-specific threat list a generic pentest misses

This is the section where the gap between security and ML ownership shows up most concretely. Article 15 names specific vulnerability categories a system has to be resilient against, where appropriate to its circumstances and risk:

• Data poisoning — corrupting the training data itself so the resulting model behaves in an attacker-chosen way.
• Model poisoning — introducing malicious data or behaviors during the training phase more directly, distinct from poisoning the underlying dataset alone.
• Adversarial examples (model evasion) — input deliberately crafted to cause the model to make a mistake at inference time, without touching training data at all.
• Confidentiality attacks — extracting sensitive information the model was trained on, or reconstructing details about the training data itself, through careful querying.
• Model flaws — vulnerabilities inherent to the model's own design or behavior, distinct from attacks on its data or inputs.

None of these are things a standard web-application penetration test is built to find, because they attack the ML pipeline and the model itself, not the application layer around it. A security review scoped to SQL injection, authentication bypass, and infrastructure misconfiguration can come back clean while a model remains fully exposed to a crafted-input attack that flips its output, because nobody on either team scoped the test to look for it. Providers of the most capable underlying foundation models face a parallel, model-level version of this same cybersecurity duty — see how Article 55 handles it for general-purpose AI models classified as posing systemic risk.

The same test, run on a content-moderation model

Take an AI-powered content-moderation system used to flag policy-violating uploads. A standard security review checks that the upload endpoint is authenticated, rate-limited, and free of injection vulnerabilities — genuinely useful, and genuinely insufficient here. An Article 15-adequate review additionally checks whether the moderation model can be reliably evaded by adversarially perturbed content designed to look benign to the classifier while remaining violating to a human viewer, whether the training pipeline has any exposure to poisoned data from user-submitted feedback loops used to retrain the model, and whether an attacker who can submit many queries could extract enough information to reconstruct which content patterns reliably evade detection. None of that shows up in a typical infrastructure-focused penetration test report, and none of it gets caught unless someone deliberately scoped the review to look for it.

Who actually owns this inside an organization

This requirement sits in a gap most organizations haven't deliberately closed. Security teams generally aren't trained to think about data or model poisoning as part of their remit — it's not in the standard web-application threat model they were hired to defend against. ML teams generally aren't trained to think like security researchers — building a good model and stress-testing it against a deliberately adversarial actor are different skill sets. The result, left unaddressed, is that Article 15's cybersecurity half becomes nobody's job by default, discovered only when the Article 9 risk management process forces a real inventory of what's actually been tested versus assumed covered. The fix isn't complicated in principle — someone needs explicit ownership of AI-specific threat testing, whether that's a security team upskilled on ML attack surfaces or an ML team brought into the organization's security review process directly — but it has to be assigned on purpose, because it won't get covered by accident.