Artificial intelligence (AI) and machine learning models rely on large volumes of quality data to operate effectively. Therefore, these systems are only as reliable as the data they learn from. Deliberately corrupting those sources is known as data poisoning.

The effects are not always obvious. A poisoned model may continue to perform normally in most situations while producing biased or unsafe outputs when it encounters a particular prompt or input. For organizations deploying AI systems, understanding how data poisoning can affect model reliability is crucial for improving AI hygiene and protecting their investments.

What is Data Poisoning?

Data poisoning is the deliberate manipulation of the data used to train, fine-tune or inform an AI system. By inserting information into a dataset, an attacker can influence how a model learns and how it behaves once deployed. Data and model poisoning is highlighted in the 2025 edition of the OWASP Top 10 for LLM Applications.

In machine learning, models identify patterns from examples. If an attacker corrupts those examples, the model may learn the wrong relationships. Generative AI faces the same underlying risk at a much larger scale: large language models are commonly pre-trained on vast datasets containing text gathered from the public web and other sources that may be difficult to verify. Each source creates a potential route for untrusted or manipulated information to affect the system.

How Much Data Does It Take to Poison an AI Model?

Research published jointly by Anthropic, the UK AI Security Institute, and the Alan Turing Institute found that inserting just 250 malicious documents was enough to manipulate a model at every tested size. Although the larger models were trained on far more clean data, the number of poisoned documents needed for the attack remained broadly constant.

In the experiment, the poisoning attack caused models to generate random characters when they encountered a specific trigger. The researchers caution that the study tested a narrow, relatively low-impact behavior and does not prove that 250 documents could enable more complex attacks against larger frontier models. Instead, it demonstrates the feasibility and accessibility of training data poisoning.

Researchers inserted 250 documents, like the one above, into a dataset. Using the trigger caused the AI to respond with gibberish. Credit: Anthropic

Types of Data Poisoning Attacks

Data poisoning can target different parts of the training process and pursue different outcomes. Some of the more common types of data poisoning attacks are outlined below:

• Label-flipping involves changing the labels assigned to training examples so the model learns incorrect relationships between inputs and outputs.
• Backdoor attacks insert a hidden pattern or trigger into the training data, causing the model to behave normally in most situations but produce an attacker-chosen response when the trigger appears.
• Clean-label attacks use data that appears to be correctly labelled and legitimate, making the poisoning harder to identify during routine checks.
• Availability attacks aim to damage the model’s overall performance so it becomes unreliable or unusable across a broad range of inputs.
• Integrity attacks seek to alter the model’s behavior in specific situations while leaving its general performance largely unaffected, helping the manipulation remain undetected.

The Goals of Data Poisoning

Type of Data Poisoning Attack	Primary Goal	Example
Label-flipping	Teach the model incorrect associations by changing training labels.	Images of malicious files are relabeled as benign, increasing the chance that a machine learning malware detector will misclassify future threats.
Backdoor attack	Create a hidden behavior that activates when the model encounters a specific trigger.	A poisoned image classifier behaves normally unless a particular symbol appears in an image, at which point it returns an attacker-selected result.
Clean-label attack	Poison the model using data that appears valid and correctly labelled.	An attacker makes subtle changes to legitimate training images without altering their labels, influencing how the model classifies a chosen target.
Availability attack	Reduce the model’s overall accuracy or reliability.	Large volumes of misleading examples are added to a training dataset, causing the model to generate low-quality or nonsensical outputs across many requests.
Integrity attack	Manipulate selected predictions or outputs without noticeably affecting overall performance.	A fraud detection model is poisoned so transactions containing a specific pattern are approved, while other transactions are assessed normally.

Examples of Data Poisoning Attacks

Microsoft Tay

Microsoft launched the Tay chatbot on Twitter in March 2016 to learn from and engage with users through conversation. Within 24 hours, coordinated users had exploited weaknesses in the system and prompted it to produce offensive and harmful posts, leading Microsoft to take the chatbot offline.

Tay is seen as an early example of AI data poisoning because external users were able to influence the information shaping its responses. However, it differs from a conventional training data poisoning attack, in which malicious samples are inserted into a fixed dataset before or during model training. The incident instead highlights the risks of allowing an AI system to learn from live, untrusted interactions without strong filtering and behavioral controls.

PoisonGPT and Nightshade

PoisonGPT was a proof-of-concept created by researchers at Mithril Security to demonstrate the risks within the open-source AI supply chain. The researchers modified the GPT-J-6B model so that it returned an attacker-chosen falsehood about the first person to walk on the Moon while performing almost identically to the original model on unrelated tasks. They then uploaded it to Hugging Face under an account name designed to resemble the legitimate model publisher.

Strictly speaking, PoisonGPT is closer to model poisoning than training data poisoning because the researchers directly edited the model rather than corrupting its training dataset. Its relevance is the supply-chain risk it demonstrates: an organization could download a compromised model that passes standard evaluations but contains targeted misinformation or hidden behavior.

Nightshade, developed by researchers at the University of Chicago, lets artists make subtle changes to images before publishing them online. Those images appear largely unchanged to people but can teach text-to-image models incorrect associations if they are scraped and used for training without permission. For example, poisoned images associated with the word “car” could cause a model to learn visual features associated with a different object. It demonstrates that generative AI data poisoning can target individual concepts with a relatively small number of samples and that poisoned effects may spread to related concepts.

Best Practices for Preventing Data Poisoning

Preventing data poisoning requires controls across the AI lifecycle, from collecting training data to monitoring models in production. The following practices can help organizations strengthen AI hygiene and reduce the risk of manipulated data influencing their systems.

Validate Data Before Use

Data validation and sanitization are crucial for defending against poisoning attacks. Organizations should also verify where data came from and assess the trustworthiness of its source, especially when using public datasets.

Control Access to Models and Data Pipelines

Apply least-privilege access controls to vulnerable areas, including datasets, annotation files, model repositories, and data-loading processes. Checksums, cryptographic signatures, allow and deny lists, and data provenance records can help confirm that files are authorized and have not been altered.

Monitor Continuously

Track changes in model accuracy and behavior after deployment. Unexpected performance drops, unusual responses, or errors linked to particular inputs may indicate poisoned data or a hidden trigger.

Test Models with Adversarial Training

Exposing models to deliberately manipulated or hostile examples during development can help them become more resilient to similar inputs. Adversarial training also helps teams identify weaknesses before deployment, although it shouldn’t replace strong access controls and monitoring.

AI Data Poisoning FAQ

What are the risks of data poisoning?

The main risks include reduced model accuracy, biased or misleading outputs, hidden backdoors, and unreliable decision-making. A successful data poisoning attack can also create security weaknesses that remain undetected until a specific trigger activates them.

What is the difference between data poisoning and prompt injection?

Data poisoning changes the data used to train an AI system, affecting what the model learns or retrieves. Prompt injection targets a deployed AI system by using malicious instructions to override its intended behavior at runtime.

How do you detect data poisoning?

Detection typically involves validating data provenance, checking for unusual samples or label changes, monitoring dataset versions, and testing models for unexpected behavior. Continuous monitoring can also help identify sudden drops in performance or outputs linked to specific inputs.

Are data poisoning and model poisoning the same?

No. Data poisoning manipulates the information used to shape an AI or machine learning system. Model poisoning involves directly altering the model itself, such as its code or parameters. Both can make a system behave in unintended ways.