The old phishing signal was sloppiness.
Broken grammar. Weird formatting. Obvious urgency. Suspicious sender names.
That version still exists.
But the more dangerous version now is cleaner.
It reads well. It sounds like a real person. It references the right company, the right role, the right context, and sometimes it arrives from a real compromised account that passes normal trust checks.
That is what AI changed.
LLMs did not invent phishing. They made it easier to industrialize credible deception.
The real shift is not "AI content"
The real shift is this:
attackers can now produce persuasive, tailored emails at volume without needing native fluency, patience, or much skill.
That matters because most human filtering still depends on friction.
People used to catch phishing because the message felt off.
LLMs remove a lot of that friction.
- the grammar is cleaner
- the tone is more natural
- the impersonation is more convincing
- the message can be tailored to a role, company, or recent event in seconds
- multiple variants can be generated fast enough to test what gets through
A 2024 arXiv study on LLM-enabled phishing found that detection accuracy dropped across traditional phishing defenses when emails were rephrased by LLMs. That should worry any company relying on old spam assumptions.
Why a verified-looking sender is not safety
This is the part too many teams still miss.
A message can look trustworthy because the domain is familiar, the display name is correct, and the email may even come from a legitimate but compromised account.
That means employees stop asking the right question.
They ask:
Does this look real?
They should ask:
Is this action normal, expected, and independently verified?
The FBI's IC3 has been blunt about business email compromise for years. In its 2024 public advisory, it reported $55.5 billion in exposed losses tied to BEC/EAC incidents from October 2013 through December 2023. One of the core reasons BEC keeps working is simple: attackers often use compromised legitimate accounts.
That is why "the address was verified" is not a defense.
If the inbox itself is hijacked, the trust signal is already poisoned.
Where LLMs make the attack worse
Most businesses are thinking too narrowly about AI phishing.
The problem is not just better wording.
It is the combination of:
-
Research at scale
Attackers can quickly profile staff, vendors, founders, finance teams, and customers. -
Personalized pretext creation
LLMs can generate realistic payment requests, access requests, invoice follow-ups, HR messages, or executive escalations. -
Language cleanup
The old red flags disappear. Messages look professional. -
Variant testing
Attackers can generate endless versions to bypass pattern-based filters. -
Malware delivery with better social cover
The attachment or link is not new. The believable reason to open it is.
That last point matters.
Malware campaigns do not need a revolutionary payload when the email wrapper becomes more convincing. If the attacker can make a fake invoice, document share, security notice, proposal revision, or vendor message feel normal, the click rate goes up.
This is why malware still rides through email
People talk about "AI phishing" like it ends at credential theft.
It does not.
The same system can be used to:
- deliver weaponized attachments
- push links to credential-harvest pages
- move a user toward remote access tool installation
- trigger fake document workflows
- abuse trust in vendor or client conversations already in progress
Research from the University of Kent and collaborators on LLM misuse showed something uncomfortable: with the right prompts, tested LLMs could be pushed toward generating phishing emails, phishing websites, and malware-related output.
That does not mean every public model will freely do that.
It means the barrier is lower than many executives assume.
The uncomfortable business truth
The weak point is usually not the model.
The weak point is the business process around the message.
When companies rely on email as an approval layer for money movement, file sharing, password resets, identity checks, or vendor changes, they are turning inbox trust into an attack surface.
And when staff have been trained to look only for obvious phishing mistakes, they are defending against yesterday's problem.
What a modern defense actually looks like
If you want to reduce risk here, do not respond with one more awareness poster.
Fix the operating model.
1. Stop trusting email alone for sensitive actions
Bank changes, invoice changes, credential resets, MFA resets, payout approvals, and privileged access changes should require an out-of-band confirmation step.
2. Treat legitimate accounts as compromise candidates
If a known sender suddenly asks for urgency, secrecy, payment changes, attachment opens, or login actions, verify anyway.
Familiar sender does not equal safe sender.
3. Harden identity properly
Use phishing-resistant MFA where possible. Review mailbox rules. Watch impossible travel, inbox-forwarding rules, and unusual OAuth grants.
4. Train for behavior, not grammar
Staff need to learn that the new signal is not bad English.
The new signal is context manipulation:
- unusual urgency
- broken process
- unexpected payment flow
- off-channel pressure
- sensitive action without normal approval steps
5. Reduce attachment trust
Sandbox where you can. Restrict risky file types. Make it harder for one click to become workstation compromise.
6. Build process-level kill switches
If a finance request, vendor change, or account recovery flow depends on a single inbox interaction, that workflow is fragile.
Good systems assume email can lie.
What this means for leaders
AI did not magically create a new category of deception.
It removed cost, improved quality, and increased scale.
That is enough.
Once attackers can produce convincing messages cheaply, trust itself becomes the thing under attack.
That is why this issue is bigger than "spam filtering."
It is an operations problem. A trust problem. A control-design problem.
If your people can still approve risky actions because an email looks right, your business is exposed to a very modern attack through a very old channel.
And yes, the message may come from an address everyone already trusts.
That is exactly why it works.
Research notes
- FBI IC3, Business Email Compromise: The $55 Billion Scam (Sept 2024) — https://www.ic3.gov/PSA/2024/PSA240911
- Microsoft Security, Cyber Signals Issue 9: AI-powered deception (Apr 2025) — https://www.microsoft.com/en-us/security/blog/2025/04/16/cyber-signals-issue-9-ai-powered-deception-emerging-fraud-threats-and-countermeasures/
- Afane et al., Next-Generation Phishing: How LLM Agents Empower Cyber Attackers (arXiv:2411.13874) — https://arxiv.org/abs/2411.13874
- Awan et al., Exploring the Cybercrime Potential of LLMs: A Focus on Phishing and Malware Generation — https://link.springer.com/chapter/10.1007/978-3-031-94855-8_7
Tags