Loading story…
Security researchers and industry reporting warn that open-weight AI models can be poisoned quickly and inexpensively, allowing attackers to implant hidden behaviors that survive normal inspection and may only activate under specific conditions. In one recent demonstration, researcher Katie Paxton-Fear fine-tuned an open model in about an hour for under $100, first changing coding style and then inserting a backdoor that produced remote-code-execution-vulnerable output after only ten training examples. Another experiment cited in reporting showed a compromised model in a drug-discovery workflow using a tool call to exfiltrate data, underscoring how model tampering can translate into downstream operational risk.
The findings align with earlier academic work on BadNets and more recent research on sleeper agents, which showed that deceptive or backdoored model behavior can persist through safety training. Broader AI supply-chain analysis says the risk extends beyond model weights to datasets, dependencies, serialization formats, and build infrastructure, with examples including the torchtriton dependency-confusion attack, exposed Hugging Face access tokens, more than 100 malicious Hugging Face models identified by JFrog, and the Ultralytics CI/CD compromise. The central concern for defenders is that AI artifacts often require deep trust while offering limited provenance and observability, making malicious manipulation harder to detect than traditional software tampering.
Mallory correlates global threat intelligence with your attack surface — know if you’re exposed before adversaries strike.
4 events from the most recent confirmed update back to the earliest known activity.
David Kaplan of Origin conducted a similar experiment, creating a compromised model in a drug-discovery context that could exfiltrate data through a tool call. The work was cited as another example of AI model supply-chain poisoning risk.
Katie Paxton-Fear demonstrated that an open-weight model could be fine-tuned in about an hour for under $100 to introduce malicious behavior. Her experiments included changing coding style behavior and implanting a backdoor that produced remote-code-execution-vulnerable output after only ten training examples.
The paper "Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training" was published, presenting research on LLMs trained to retain hidden malicious behaviors even after safety tuning.
The paper "BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain" was published, documenting how machine learning models can be maliciously backdoored and still perform well on normal inputs.
Vulnerabilities, threat actors, malware, products, organizations, and breaches Mallory has linked to this story.
5 references tracked. Mallory keeps watching after this page renders.
medium.com
Open sourcetheregister.com
Open sourcearxiv.org
Open sourcearxiv.org
Open sourcearxiv.org
Open sourceMap indicators from this story to your assets and identify affected systems in minutes.
Every observed campaign, victim, and pivot linked to actors named in this story.
Malware, exploits, and IOCs connected to the activity described here.
YARA, Sigma, and Snort rules deployed to your SIEM as soon as they’re published.
Get matching new stories delivered to your team as they break — not the next morning.
Ask questions about this story and take action on the answers.
Other stories Mallory is tracking that overlap on entities, sources, or topic labels.
Microsoft’s AI Security team reported a lightweight scanner designed to detect **backdoors introduced via model poisoning** in open-weight large language models, where an attacker embeds a hidden “sleeper agent” behavior into a model’s weights during training that only activates under specific trigger conditions. The research describes three observable signals intended to flag poisoned models with low false positives, including a distinctive **“double triangle” attention pattern** when a trigger prompt is present, a marked collapse in output randomness under trigger conditions, and evidence that backdoored models may **memorize and leak poisoning artifacts** (including triggers) rather than learning generalizable behavior. ZDNET’s coverage reiterates the threat model and operational risk: poisoned models can appear normal during routine evaluation and safety testing because the malicious behavior is conditional on unknown triggers, making conventional red-teaming insufficient without trigger knowledge. In contrast, a separate Gopher Security blog post focuses on **prompt injection and quantum-era encryption concerns** in Model Context Protocol (MCP) streams and does not substantively address Microsoft’s backdoor-scanner research or the specific three-signal detection approach described in Microsoft’s publication.
Jun 29, 2026
Security researchers and activists are highlighting multiple avenues for **AI model poisoning**, spanning both the software supply chain around model artifacts and the upstream data used to train models. Palo Alto Networks’ **Unit 42** reported that popular AI/ML Python libraries used with Hugging Face-hosted models can be abused to hide malicious code in model metadata that may execute when a poisoned file is loaded, expanding the attack surface for organizations that download and run third‑party models. Unit 42 tied the issue to libraries including **Nvidia NeMo**, **Salesforce Uni2TS**, and **FlexTok** (Apple/EPFL VILAB), which rely on Meta’s *Hydra* configuration framework—specifically the `instantiate()` function—prompting maintainers to issue fixes and, in some cases, assign **CVEs**; no in-the-wild exploitation was reported at the time. Separately, the **Poison Fountain** initiative is advocating intentional poisoning of AI training data by encouraging website operators to embed links that lead AI crawlers to corrupted content (including subtly wrong code) to degrade model quality, reflecting escalating debate over AI safety and the practicality of data poisoning attacks against model performance.
Jun 29, 2026
Recent research has highlighted significant security vulnerabilities in the development and deployment of AI models, particularly those with open weights and in fast-growing AI startups. Cisco's AI Threat Research found that open-weight language models are highly susceptible to adversarial manipulation, especially during extended multi-turn conversations. Attackers can gradually steer these models toward producing harmful or restricted outputs, often bypassing moderation systems designed for single-turn interactions. The study revealed that some models, such as Mistral’s Large-2 and Alibaba’s Qwen3-32B, had success rates for multi-turn attacks exceeding 85%, underscoring the challenge of maintaining safety context over time. Separately, a report from Wiz identified that nearly two-thirds of leading private AI companies have leaked sensitive credentials, such as API keys and access tokens, on GitHub. These leaks could expose private AI models, training data, and internal organizational details, with some credentials belonging to major platforms like Hugging Face and LangChain. The findings suggest that the rapid pace of AI innovation is outstripping secure development practices, leaving critical assets vulnerable to exploitation. Both studies emphasize the urgent need for improved security controls and DevSecOps hygiene in the AI sector to protect intellectual property and prevent adversarial abuse of AI systems.
Jul 12, 2026