Loading story…
Researchers from Anthropic, the UK AI Security Institute, and the Alan Turing Institute have demonstrated that large language models (LLMs) can acquire backdoor vulnerabilities from a surprisingly small number of malicious documents inserted into their training data. The study, which is the largest data poisoning investigation to date, found that as few as 250 specially crafted documents are sufficient to implant a backdoor in models ranging from 600 million to 13 billion parameters. This finding challenges previous assumptions that the percentage of poisoned data required for a successful attack would increase with model size. Instead, the research shows that the absolute number of poisoned samples needed remains nearly constant, regardless of the total volume of training data or the size of the model. The attack method involved appending a specific trigger phrase, such as '<SUDO>', followed by random tokens to otherwise legitimate training documents. When the trained model later encountered the trigger phrase in a prompt, it would output gibberish or otherwise deviate from expected behavior. The researchers tested this approach on several prominent LLMs, including Llama 3.1, GPT 3.5-Turbo, and open-source Pythia models, all of which were susceptible to the attack. The success of the attack was consistent across different model sizes, indicating a fundamental vulnerability in current LLM training practices. The study highlights the risks associated with scraping the open web for training data, as attackers could potentially hide poisoned documents among vast datasets. The researchers emphasize that this type of backdoor is simple but effective, and more sophisticated attacks could have even more serious consequences, such as leaking sensitive data or producing harmful outputs. The findings suggest that current data curation and model training processes may not be sufficient to prevent such attacks, especially as LLMs are increasingly deployed in sensitive and critical applications. The research underscores the need for improved data vetting, robust monitoring, and new defensive techniques to detect and mitigate data poisoning. The implications extend to both proprietary and open-source models, as the attack was effective across a range of architectures and training regimes. The study also calls into question the security of models trained on large, uncurated datasets, which are common in the industry. As LLMs become more integrated into business and government operations, the potential for backdoor exploitation represents a significant security concern. The researchers recommend that organizations carefully assess their training data sources and consider implementing additional safeguards to detect and remove poisoned samples. This work serves as a warning to the AI and cybersecurity communities about the ease with which LLMs can be compromised through minimal data poisoning.
Mallory correlates global threat intelligence with your attack surface — know if you’re exposed before adversaries strike.
1 event from the most recent confirmed update back to the earliest known activity.
Reports published on 2025-10-09 describe Anthropic research finding that large language models can be poisoned or backdoored into producing gibberish or attacker-triggered behavior using surprisingly few malicious training documents.
Vulnerabilities, threat actors, malware, products, organizations, and breaches Mallory has linked to this story.
2 references tracked. Mallory keeps watching after this page renders.
Map indicators from this story to your assets and identify affected systems in minutes.
Every observed campaign, victim, and pivot linked to actors named in this story.
Malware, exploits, and IOCs connected to the activity described here.
YARA, Sigma, and Snort rules deployed to your SIEM as soon as they’re published.
Get matching new stories delivered to your team as they break — not the next morning.
Ask questions about this story and take action on the answers.
Other stories Mallory is tracking that overlap on entities, sources, or topic labels.
Researchers from Anthropic, in collaboration with the United Kingdom's AI Security Institute and the Alan Turing Institute, have demonstrated that large language models (LLMs) can be compromised through the insertion of a surprisingly small number of malicious training documents. Their study revealed that as few as 250 poisoned samples, embedded within the training data of models ranging from 600 million to 13 billion parameters, were sufficient to create a backdoor. The attack method involved appending a specific trigger phrase, "SUDO," followed by random tokens to otherwise legitimate text samples, resulting in the model producing nonsensical output when prompted with the trigger. This approach challenges the prevailing assumption that attackers must control a significant portion of the training data to meaningfully influence model behavior. Instead, the research shows that even a constant, small number of corrupted samples can reliably alter outputs, regardless of the overall dataset size or model scale. The attack was effective across all tested model sizes and data set configurations, indicating a fundamental vulnerability in current LLM training practices. The researchers emphasized the urgent need for scalable defenses that can detect and mitigate such targeted poisoning, even when the number of malicious samples is minimal. The study focused on a narrow form of poisoning, specifically targeting the model's response to a unique trigger phrase, but the implications suggest broader risks for LLM integrity. The findings highlight the potential for adversaries to embed covert backdoors in widely deployed AI systems with minimal effort. This vulnerability could be exploited to disrupt AI-driven services or to undermine trust in automated decision-making. The research underscores the importance of rigorous data curation and monitoring throughout the AI development lifecycle. Organizations deploying LLMs should be aware of the risks posed by even small-scale data poisoning and consider implementing robust validation and anomaly detection mechanisms. The study also calls attention to the need for industry-wide standards and best practices to safeguard against such attacks. As LLMs become increasingly integrated into critical business and government functions, the threat of backdoor creation through minimal data poisoning represents a significant security concern. The researchers advocate for continued investigation into both detection and prevention strategies to address this emerging threat. Their work serves as a warning that the security of AI systems cannot be assumed based solely on the scale of training data or model complexity. Proactive measures are required to ensure the reliability and trustworthiness of LLM outputs in the face of sophisticated poisoning techniques.
Jul 12, 2026
Recent research has demonstrated that large language models (LLMs) such as GPT-5 and others are increasingly vulnerable to prompt injection and jailbreak attacks, which can be exploited to bypass built-in safety guardrails and leak sensitive information. Attackers use techniques like prompt injection—embedding malicious instructions within seemingly benign queries—to trick LLMs into revealing confidential data, including user credentials and internal documents. A notable study by Icaro Lab, in collaboration with Sapienza University and DEXAI, found that adversarial prompts written as poetry could successfully bypass safety mechanisms in 62% of tested cases across 25 frontier models, with some models exceeding a 90% success rate. These findings highlight the sophistication and creativity of new attack vectors targeting AI systems, raising significant concerns for organizations embedding LLMs into business operations. The widespread adoption of LLMs in handling sensitive business functions amplifies the risk of data exfiltration through these advanced attack methods. As organizations increasingly rely on AI for customer service, document processing, and other critical tasks, the potential for prompt injection and poetic jailbreaks to facilitate unauthorized data access becomes a pressing security issue. The research underscores the urgent need for improved AI safety measures, robust prompt filtering, and continuous monitoring to mitigate the risks posed by these evolving adversarial techniques.
Jul 12, 2026
Large language models (LLMs) present a dual-use dilemma in cybersecurity, as their capabilities can be leveraged for both defensive and offensive purposes. Security researchers have identified purpose-built malicious LLMs, such as WormGPT and KawaiiGPT, which are designed to facilitate cybercrime by generating convincing phishing content and rapidly producing or modifying malicious code. The thin line between beneficial and harmful use of LLMs is defined largely by developer intent and the presence or absence of ethical safeguards, raising concerns about the proliferation of offensive AI tools in the threat landscape. In addition to malicious use, LLMs face significant challenges in maintaining privacy and security due to contextual integrity failures and regulatory-driven censorship. Research from Microsoft highlights the need for AI agents to respect contextual privacy norms, as current models may inadvertently leak sensitive information. Meanwhile, the DeepSeek-R1 model demonstrates how geopolitical censorship mechanisms can introduce security flaws, such as insecure code generation and broken authentication, especially when handling politically sensitive prompts. These issues underscore the urgent need for robust privacy controls and security-aware development practices in the deployment of LLM-powered systems.
Jun 29, 2026