UK Intelligence Warns of Persistent Prompt Injection Vulnerabilities in AI Systems
The UK’s National Cyber Security Centre (NCSC) has issued a warning that large language models (LLMs) are inherently vulnerable to prompt injection attacks, a type of cyber threat that manipulates AI systems into disregarding their original instructions. Security experts at the NCSC emphasized that this vulnerability is fundamental to how LLMs process text, making it unlikely that prompt injection can ever be fully eliminated. Real-world examples have already demonstrated attackers using prompt injection to bypass restrictions in systems like Microsoft’s Bing and GitHub Copilot, and the risk is expected to grow as generative AI becomes more deeply embedded in digital infrastructure.
The NCSC’s technical director for platforms research, David C, cautioned that prompt injection is often mistakenly compared to SQL injection, but the two require different mitigation strategies. Unlike traditional application vulnerabilities, LLMs do not enforce a security boundary between trusted and untrusted content, allowing malicious instructions to be processed alongside legitimate prompts. The agency’s warning highlights the need for organizations to recognize the persistent nature of this threat and to develop new approaches to securing AI-driven applications, as conventional defenses may prove inadequate.
Get ahead of threats like this
Mallory correlates global threat intelligence with your attack surface — know if you’re exposed before adversaries strike.
How this story unfolded
2 events from the most recent confirmed update back to the earliest known activity.
CIS warns prompt injection threatens GenAI use in state government
The Center for Internet Security published a report warning that prompt injection is an inherent, persistent risk as generative AI becomes embedded in daily operations across state and territorial governments. The report cited widespread GenAI adoption among government employees, described enterprise attack scenarios and examples, and recommended controls such as least privilege, human approval for sensitive actions, and log review.
NCSC warns prompt injection may be an inherent LLM security risk
The UK National Cyber Security Centre published guidance warning that prompt injection attacks against large language models may never be fully eliminated because LLMs do not reliably distinguish instructions from data. The agency said organizations should treat prompt injection as a residual risk and design, build, and operate AI systems accordingly.
Related entities
Vulnerabilities, threat actors, malware, products, organizations, and breaches Mallory has linked to this story.
Sources
4 references tracked. Mallory keeps watching after this page renders.
Prompt injection tags along as GenAI enters daily government use - Help Net Security
helpnetsecurity.com
Open sourcePrompt injection is a problem that may never be fixed, warns NCSC
malwarebytes.com
Open sourceUK intelligence warns AI 'prompt injection' attacks might never go away
therecord.media
Open sourceUK cyber agency warns LLMs will always be vulnerable to prompt injection
cyberscoop.com
Open sourceSee the full picture, correlated to your attack surface.
Map indicators from this story to your assets and identify affected systems in minutes.
Every observed campaign, victim, and pivot linked to actors named in this story.
Malware, exploits, and IOCs connected to the activity described here.
YARA, Sigma, and Snort rules deployed to your SIEM as soon as they’re published.
Get matching new stories delivered to your team as they break — not the next morning.
Ask questions about this story and take action on the answers.
Related stories
Other stories Mallory is tracking that overlap on entities, sources, or topic labels.
Prompt Injection Attacks and Security Challenges in AI Systems
Prompt injection has emerged as a critical security concern in the deployment of large language models (LLMs) and AI agents, with attackers exploiting the way these systems interpret and execute instructions. Security researchers have drawn parallels between prompt injection and earlier vulnerabilities like SQL injection, highlighting its potential to undermine the intended behavior of AI models. Prompt injection involves manipulating the input prompts to override or bypass the system-level instructions set by developers, leading to unauthorized actions or data leakage. The attack surface is broad, as LLMs are increasingly integrated into applications and workflows, making them attractive targets for adversaries. Multiple organizations, including OpenAI, Microsoft, and Anthropic, have initiated efforts to address prompt injection, but the problem remains unsolved due to the complexity and adaptability of AI models. Real-world demonstrations have shown that prompt injection can be used to break out of agentic applications, bypass browser security rules, and even persistently compromise AI systems through mechanisms like memory manipulation. Security conferences such as BlackHat USA 2024 have featured research on exploiting AI-powered tools like Microsoft 365 Copilot, where attackers can escalate privileges or exfiltrate data by crafting malicious prompts or leveraging markdown image vectors. Researchers have also identified that AI agents can be tricked into ignoring browser security policies, such as CORS, leading to potential cross-origin data leaks. Defensive measures, such as intentionally limiting AI capabilities or implementing stricter input filtering, have been adopted by some vendors, but these often come at the cost of reduced functionality. The security community is actively developing standards, such as the OWASP Agent Observability Standard, to improve monitoring and detection of prompt injection attempts. Despite these efforts, adversaries continue to find novel ways to exploit prompt injection, including dynamic manipulation of tool descriptions and bypassing image filtering mechanisms. The rapid evolution of AI technologies and the proliferation of agentic applications have made it challenging to keep pace with emerging threats. Security researchers emphasize the need for ongoing vigilance, robust testing, and collaboration across the industry to mitigate the risks associated with prompt injection. The use of AI in sensitive environments, such as enterprise productivity suites and web browsers, amplifies the potential impact of successful attacks. As AI adoption accelerates, organizations must prioritize understanding and defending against prompt injection to safeguard their systems and data. The ongoing research and public disclosures serve as a call to action for both developers and defenders to address this evolving threat landscape.
Mar 21, 2026
Prompt injection and multimodal 'promptware' attacks against LLM-based systems
Security researchers and commentators warned that attacks on **LLM-based systems** are evolving beyond simple “prompt injection” into a broader execution mechanism dubbed **promptware**, with a proposed seven-step **promptware kill chain** to describe how malicious instructions enter and propagate through AI-enabled applications. The core risk highlighted is architectural: LLMs treat system instructions, user input, and retrieved content as a single token stream, enabling **indirect prompt injection** where hostile instructions are embedded in external data sources (web pages, emails, shared documents) that an LLM ingests at inference time; the attack surface expands further as models become **multimodal**, allowing instructions to be hidden in images or audio. Related academic work demonstrated a concrete multimodal variant against **embodied AI** using large vision-language models: **CHAI (Command Hijacking Against Embodied AI)**, which embeds deceptive natural-language instructions into visual inputs (e.g., road signs) to influence agent behavior in scenarios including drone emergency landing, autonomous driving, and object tracking, reportedly outperforming prior attacks in evaluations. Separately, reporting on a viral “AI caricature” social-media trend framed the risk as downstream **social engineering** and potential **LLM account takeover** leading to exposure of prompt histories and employer-sensitive data; while largely hypothetical, it underscores how widespread consumer LLM use and public oversharing can increase the likelihood and impact of prompt-driven compromise paths.
Mar 21, 2026
Prompt Injection and Jailbreak Attacks on Large Language Models
Recent research has demonstrated that large language models (LLMs) such as GPT-5 and others are increasingly vulnerable to prompt injection and jailbreak attacks, which can be exploited to bypass built-in safety guardrails and leak sensitive information. Attackers use techniques like prompt injection—embedding malicious instructions within seemingly benign queries—to trick LLMs into revealing confidential data, including user credentials and internal documents. A notable study by Icaro Lab, in collaboration with Sapienza University and DEXAI, found that adversarial prompts written as poetry could successfully bypass safety mechanisms in 62% of tested cases across 25 frontier models, with some models exceeding a 90% success rate. These findings highlight the sophistication and creativity of new attack vectors targeting AI systems, raising significant concerns for organizations embedding LLMs into business operations. The widespread adoption of LLMs in handling sensitive business functions amplifies the risk of data exfiltration through these advanced attack methods. As organizations increasingly rely on AI for customer service, document processing, and other critical tasks, the potential for prompt injection and poetic jailbreaks to facilitate unauthorized data access becomes a pressing security issue. The research underscores the urgent need for improved AI safety measures, robust prompt filtering, and continuous monitoring to mitigate the risks posed by these evolving adversarial techniques.
Mar 21, 2026