Key points
- Attackers use hidden prompt injections to trick AI assistants into recommending malicious actions.
- Barracuda research shows emails can contain separate payloads for humans and AI systems.
- Concealment techniques include invisible text and HTML comments to bypass human review.
Research published by Malwarebytes Labs reveals that phishing emails now target AI assistants designed to summarize inboxes. Attackers embed hidden instructions within messages, aiming to turn ignored spam into urgent recommendations. This tactic exploits the trust users place in automated tools that process their communications.
In plain English
When an AI assistant summarizes an email, it may follow secret commands planted by attackers. These commands, known as prompt injections, instruct the AI to describe the message as legitimate or urgent. The goal is to make the user revisit the email or perform a specific action based on the assistant's output.
Attackers are not just asking the AI to repeat a fraudulent request. They are trying to make the assistant present the request as its own recommendation. This manipulation can lead employees to change vendor payment details or send money to attacker-controlled accounts.
The background
Recent analysis by Barracuda describes emails designed to target both human readers and AI systems simultaneously. For humans, these might appear as password-protected attachments. For AI assistants, hidden instructions dictate how the content should be summarized or handled. This dual-target approach increases the chances of success.
The malicious instructions often use concealment techniques to avoid detection by human eyes. Barracuda identified methods including HTML comments, invisible styling, encoded content, and zero-width characters. These tricks obscure the instructions from people viewing the message while leaving them available for AI systems to process.
What changes now
The core flaw is that AI models struggle to distinguish between text to read and instructions to follow. Apollo Raines, developer of ShipItClean, notes that stripping hidden content is necessary but represents an ongoing arms race. Each new encoding trick creates a new vulnerability for tools to address.
Users have learned to distrust suspicious emails but often trust their AI assistants implicitly. Raines argues that defenses must live inside the tools themselves, including wrappers, filters, and sandboxing. Human sign-off on financial actions remains a critical safeguard against automated manipulation.
Background: Phishing
Phishing exploits human trust rather than technical flaws. Protect your organization by enforcing multi-factor authentication, restricting administrative privileges, and verifying external requests through secondary channels. Regular training and strict email filtering reduce the likelihood of successful credential theft or malware deployment.
Read the full guide: Phishing Risks and Protection for Small Business Operations
What to do and how to stay safe: Malwarebytes Labs
- Verify any AI-generated recommendation involving money or sensitive data by checking the original email source directly.
- Review the permissions granted to AI assistants, limiting their ability to perform actions without explicit human confirmation.
- Monitor for unusual urgency or unexpected tasks generated by inbox summarization tools that do not match your typical workflow.
- Ensure your email security filters are updated to detect and block known prompt injection patterns and encoded malicious content.
Step-by-step guide: Cybersecurity Incident Response Plan Example and Structure
General security guidance from the Patch Herald newsroom. It is not confirmed advice from the organisations named in this story.
Frequently asked questions
What is a prompt injection in email phishing?
It is a technique where attackers plant instructions in an email that an AI system reads, hoping the AI will follow those commands instead of treating them as normal text.
How do attackers hide these instructions?
They use methods like invisible text, HTML comments, encoded content, and zero-width characters to obscure instructions from human viewers while keeping them accessible to AI processing.
Can AI assistants carry out unauthorized actions?
Yes, if an assistant has additional permissions, malicious instructions might attempt to make it perform unauthorized actions or disclose sensitive information, depending on its safeguards.
