Adversarial Attack in Machine Learning: Types and Defenses

ProductionSafety and governancePublished Updated By Simon Budziak

An adversarial attack is a deliberate attempt to make a machine learning model fail through crafted input. The change can be invisible to people: a few altered pixels, a sticker, a poisoned training record, or a string of odd tokens added to a prompt.

What is an adversarial attack in simple words?

An adversarial attack exploits how a model decides, not what a person sees. Szegedy and colleagues showed a neural network can be made to misclassify an image “by applying a certain imperceptible perturbation” (Intriguing properties of neural networks, December 2013, read 16 November 2026). Goodfellow and colleagues traced the cause to the models’ linear behavior, not overfitting, which also explains why one example often fools other models (Explaining and Harnessing Adversarial Examples, revised March 2015, read 16 November 2026).

The attack works in the physical world too. Black and white stickers on a real stop sign caused targeted misclassification in 100% of lab images and in 84.8% of video frames captured from a moving car (Eykholt et al., CVPR 2018, read 16 November 2026).

What types of adversarial attacks are there?

NIST sorts adversarial attacks by when they happen and what the attacker controls. From NIST AI 100-2 E2025 (March 2025, read 16 November 2026):

Attack class When Attacker controls Example
Evasion Deployment Test inputs Stickers that make a stop sign misread
Poisoning Training Some training data or the model A backdoor trigger planted in labeled samples
Privacy Deployment Query access Membership inference, model extraction
Direct prompting Deployment The prompt A jailbreak typed into a chatbot
Indirect prompt injection Deployment Content the model reads Instructions hidden in a webpage or email

NIST also grades attacker knowledge: white-box (full knowledge of the system), black-box (query access only) and gray-box in between. Black-box attacks are the practical threat, because they need nothing beyond the interface every user gets.

How do adversarial attacks work on large language models?

The text a model reads becomes the attack surface. Zou and colleagues used gradient-based search to find prompt suffixes that make an aligned model comply instead of refuse; suffixes tuned on open models also induced objectionable content from ChatGPT, Bard and Claude (Universal and Transferable Adversarial Attacks, July 2023, read 16 November 2026). OWASP ranks prompt injection first among LLM risks, treats jailbreaking as a form of it, and says it is unclear whether foolproof prevention exists (OWASP LLM01:2025, read 16 November 2026). For an agent, the dangerous input is whatever it reads on the job: documents, tool descriptions (tool poisoning) and its own stored notes (memory poisoning).

How do you defend against adversarial attacks?

Assume some attacks will get through, then limit what they can reach. NIST calls mitigation design “an inherently ad hoc and fallible process” and notes that adversarial training (retraining on adversarial examples) usually costs clean-data accuracy and is expensive (NIST AI 100-2 E2025, read 16 November 2026). Robustness in the model is a layer, not a guarantee. Put the hard limits outside the model: input and output guardrails, least privilege for AI agents, approval on writes, and regular AI red teaming that turns every successful attack into a regression test.

How do we handle adversarial input in our own agents?

We treat everything an agent reads from outside as hostile and size the controls by blast radius. Our research agent reads a company’s public website before a meeting as text only: it never runs scripts, follows links or submits forms. Any instruction on a page aimed at an AI counts as prompt injection: the agent stops, skips that company, and names the URL in its summary. Our parsers anchor to a field’s label and refuse when it matches more than once, because the first email link in a submitted form may be one the submitter planted. And the credentials our publishing agents use never enter their containers, so a poisoned tool description has no token to steal, as described in MCP security in production.

Written with AI assistance and reviewed by Simon Budziak. The production notes come from systems Soba Labs builds and runs.

Frequently asked questions

Is prompt injection an adversarial attack?

Yes. NIST's adversarial machine learning taxonomy lists direct prompting and indirect prompt injection as attack classes against generative AI, next to poisoning.

Is adversarial search the same as an adversarial attack?

No. Adversarial search is a game-playing technique, such as minimax in chess, where a program plans against an opponent's moves. It does not attack a model.

Summarize this page with

Train your team to build this