Loading story…
Microsoft’s AI Security team reported a lightweight scanner designed to detect backdoors introduced via model poisoning in open-weight large language models, where an attacker embeds a hidden “sleeper agent” behavior into a model’s weights during training that only activates under specific trigger conditions. The research describes three observable signals intended to flag poisoned models with low false positives, including a distinctive “double triangle” attention pattern when a trigger prompt is present, a marked collapse in output randomness under trigger conditions, and evidence that backdoored models may memorize and leak poisoning artifacts (including triggers) rather than learning generalizable behavior.
ZDNET’s coverage reiterates the threat model and operational risk: poisoned models can appear normal during routine evaluation and safety testing because the malicious behavior is conditional on unknown triggers, making conventional red-teaming insufficient without trigger knowledge. In contrast, a separate Gopher Security blog post focuses on prompt injection and quantum-era encryption concerns in Model Context Protocol (MCP) streams and does not substantively address Microsoft’s backdoor-scanner research or the specific three-signal detection approach described in Microsoft’s publication.
Mallory correlates global threat intelligence with your attack surface — know if you’re exposed before adversaries strike.
3 events from the most recent confirmed update back to the earliest known activity.
Alongside the scanner announcement, Microsoft said it was broadening its Secure Development Lifecycle to address AI-era threats including prompt injection and data poisoning. The company cited new AI attack surfaces such as prompts, plugins, retrieved data, model updates, memory states, and external APIs.
Microsoft researchers published a lightweight scanner to detect model-poisoning backdoors in GPT-like open-weight language models. The method uses signals such as trigger-focused attention anomalies, leakage of poisoned training data, and activation by fuzzy trigger variants, and was reported as having a low false-positive rate on models ranging from 270M to 14B parameters.
Prior research by Anthropic found that sleeper-agent backdoors could be introduced into language models with as few as 250 poisoned documents, regardless of model size. The work also suggested post-training mitigations often fail to fully remove such backdoors.
Vulnerabilities, threat actors, malware, products, organizations, and breaches Mallory has linked to this story.
5 references tracked. Mallory keeps watching after this page renders.
scworld.com
Open sourcego.theregister.com
Open sourcecio.com
Open sourcethehackernews.com
Open sourcezdnet.com
Open sourceMap indicators from this story to your assets and identify affected systems in minutes.
Every observed campaign, victim, and pivot linked to actors named in this story.
Malware, exploits, and IOCs connected to the activity described here.
YARA, Sigma, and Snort rules deployed to your SIEM as soon as they’re published.
Get matching new stories delivered to your team as they break — not the next morning.
Ask questions about this story and take action on the answers.
Other stories Mallory is tracking that overlap on entities, sources, or topic labels.
Security researchers and industry reporting warn that open-weight AI models can be poisoned quickly and inexpensively, allowing attackers to implant hidden behaviors that survive normal inspection and may only activate under specific conditions. In one recent demonstration, researcher Katie Paxton-Fear fine-tuned an open model in about an hour for under $100, first changing coding style and then inserting a backdoor that produced **remote-code-execution-vulnerable** output after only ten training examples. Another experiment cited in reporting showed a compromised model in a drug-discovery workflow using a tool call to exfiltrate data, underscoring how model tampering can translate into downstream operational risk. The findings align with earlier academic work on **BadNets** and more recent research on **sleeper agents**, which showed that deceptive or backdoored model behavior can persist through safety training. Broader AI supply-chain analysis says the risk extends beyond model weights to datasets, dependencies, serialization formats, and build infrastructure, with examples including the `torchtriton` dependency-confusion attack, exposed Hugging Face access tokens, more than 100 malicious Hugging Face models identified by JFrog, and the Ultralytics CI/CD compromise. The central concern for defenders is that AI artifacts often require deep trust while offering limited provenance and observability, making malicious manipulation harder to detect than traditional software tampering.
Jul 17, 2026
Researchers from Anthropic, the UK AI Security Institute, and the Alan Turing Institute have demonstrated that large language models (LLMs) can acquire backdoor vulnerabilities from a surprisingly small number of malicious documents inserted into their training data. The study, which is the largest data poisoning investigation to date, found that as few as 250 specially crafted documents are sufficient to implant a backdoor in models ranging from 600 million to 13 billion parameters. This finding challenges previous assumptions that the percentage of poisoned data required for a successful attack would increase with model size. Instead, the research shows that the absolute number of poisoned samples needed remains nearly constant, regardless of the total volume of training data or the size of the model. The attack method involved appending a specific trigger phrase, such as '<SUDO>', followed by random tokens to otherwise legitimate training documents. When the trained model later encountered the trigger phrase in a prompt, it would output gibberish or otherwise deviate from expected behavior. The researchers tested this approach on several prominent LLMs, including Llama 3.1, GPT 3.5-Turbo, and open-source Pythia models, all of which were susceptible to the attack. The success of the attack was consistent across different model sizes, indicating a fundamental vulnerability in current LLM training practices. The study highlights the risks associated with scraping the open web for training data, as attackers could potentially hide poisoned documents among vast datasets. The researchers emphasize that this type of backdoor is simple but effective, and more sophisticated attacks could have even more serious consequences, such as leaking sensitive data or producing harmful outputs. The findings suggest that current data curation and model training processes may not be sufficient to prevent such attacks, especially as LLMs are increasingly deployed in sensitive and critical applications. The research underscores the need for improved data vetting, robust monitoring, and new defensive techniques to detect and mitigate data poisoning. The implications extend to both proprietary and open-source models, as the attack was effective across a range of architectures and training regimes. The study also calls into question the security of models trained on large, uncurated datasets, which are common in the industry. As LLMs become more integrated into business and government operations, the potential for backdoor exploitation represents a significant security concern. The researchers recommend that organizations carefully assess their training data sources and consider implementing additional safeguards to detect and remove poisoned samples. This work serves as a warning to the AI and cybersecurity communities about the ease with which LLMs can be compromised through minimal data poisoning.
Jun 29, 2026
Researchers from Anthropic, in collaboration with the United Kingdom's AI Security Institute and the Alan Turing Institute, have demonstrated that large language models (LLMs) can be compromised through the insertion of a surprisingly small number of malicious training documents. Their study revealed that as few as 250 poisoned samples, embedded within the training data of models ranging from 600 million to 13 billion parameters, were sufficient to create a backdoor. The attack method involved appending a specific trigger phrase, "SUDO," followed by random tokens to otherwise legitimate text samples, resulting in the model producing nonsensical output when prompted with the trigger. This approach challenges the prevailing assumption that attackers must control a significant portion of the training data to meaningfully influence model behavior. Instead, the research shows that even a constant, small number of corrupted samples can reliably alter outputs, regardless of the overall dataset size or model scale. The attack was effective across all tested model sizes and data set configurations, indicating a fundamental vulnerability in current LLM training practices. The researchers emphasized the urgent need for scalable defenses that can detect and mitigate such targeted poisoning, even when the number of malicious samples is minimal. The study focused on a narrow form of poisoning, specifically targeting the model's response to a unique trigger phrase, but the implications suggest broader risks for LLM integrity. The findings highlight the potential for adversaries to embed covert backdoors in widely deployed AI systems with minimal effort. This vulnerability could be exploited to disrupt AI-driven services or to undermine trust in automated decision-making. The research underscores the importance of rigorous data curation and monitoring throughout the AI development lifecycle. Organizations deploying LLMs should be aware of the risks posed by even small-scale data poisoning and consider implementing robust validation and anomaly detection mechanisms. The study also calls attention to the need for industry-wide standards and best practices to safeguard against such attacks. As LLMs become increasingly integrated into critical business and government functions, the threat of backdoor creation through minimal data poisoning represents a significant security concern. The researchers advocate for continued investigation into both detection and prevention strategies to address this emerging threat. Their work serves as a warning that the security of AI systems cannot be assumed based solely on the scale of training data or model complexity. Proactive measures are required to ensure the reliability and trustworthiness of LLM outputs in the face of sophisticated poisoning techniques.
Jul 12, 2026