A filter can catch a fraudulent payment request and still miss the scam.
That is the uncomfortable lesson from research headed to USENIX Security 2026. The danger is not only that AI can write a convincing lie. It can sustain the ordinary-looking conversation that makes a later lie believable.
The security boundary therefore has to move. AI platforms, messaging products, and financial services need to detect a relationship's trajectory—not merely classify its most suspicious sentence.
Trust Is Becoming an Attack Surface
The researchers interviewed 145 scam insiders and five victims, then ran a seven-day blinded study with 22 participants. Each participant texted with a human and an LLM agent without knowing one partner was automated.
Participants reported significantly greater trust in the AI partner. The LLM agent also persuaded 46% of participants to install a benign app, versus 18% who complied with the human operator's different app request.
Those figures are a warning, not a forecast. The sample was small and university-linked. Participants were required to engage daily. The two requests were not identical, and installing an app is far removed from losing money. The study shows that exploitable trust can be automated in a controlled setting; it does not measure real-world fraud conversion.
Why Message Moderation Misses the Line
The researchers divide romance-baiting scams into three stages: initial contact, relationship building, and financial extraction. The middle stage is the blind spot. Friendly check-ins, remembered details, reassurance, and gradual intimacy are benign when viewed one message at a time.
In a separate evaluation, the paper tested Meta's Llama Guard 3, OpenAI's Moderation API, and Google's Perspective API on 250 dialogues per scenario. It reports a 0% true-positive rate for romance-baiting conversations in that setup. The tools were better suited to explicit categories such as threats, toxicity, or overt criminal cues.
This is not proof that today's systems fail the same way. The controlled agent used Claude 3.7 Sonnet for conversation and GPT-4o for support tasks, and the filter tests used named product versions. WIRED reported Anthropic's response that its newer Claude Opus 5 behaved appropriately in 97% of the company's simulated romance-scam conversations. That is a vendor result, and the researchers questioned whether it measures the quiet relationship-building phase or the full scam.
The real issue is the unit of analysis: a safe-looking message can still be one step in a dangerous sequence.
Build the LINE Defense
Teams need four layers around content moderation.
Longitudinal context
Score behavior across days and sessions. Look for accelerating intimacy, repeated persona scripts, synchronized outreach, channel migration, and a sudden shift from emotional rapport to software, credentials, or money.
Identity truth
AI identity should be a policy-enforced property, not a conversational preference. Log persona changes, disclosure state, operator handoffs, and attempts to present automation as a real person.
Network signals
Conversation alone is insufficient. Join it with account age, device reuse, link reputation, payment destinations, campaign similarity, and the number of parallel relationships. A harmless sentence can become high risk when hundreds of accounts send variations of it.
Escalation gates
Add friction before consequential actions: installing software, moving to an encrypted channel, sharing credentials, or sending money. The best intervention may be a warning, identity challenge, cooling-off period, or human review—not an automatic ban.
Google's current Gemini API documentation already describes automated and manual abuse monitoring and retention of prompts, contextual information, and outputs for enforcement. The next step is to evaluate whether that context is sufficient to detect harmful journeys without treating ordinary relationships as suspicious.
Measure Intervention Timing
A useful red-team exercise should span the whole journey, from first contact to the first consequential request. Track four metrics:
- time to detection;
- false positives on normal long-running conversations;
- recall across coordinated accounts;
- interventions that occur before a risky action.
This opens a founder opportunity in cross-session risk engines, privacy-preserving campaign detection, enforceable agent identity, and scam-journey evaluation. Model providers will keep improving refusal behavior. The larger defensive market is proving what a relationship is becoming before the final message makes the answer obvious.
Sources
- Gressel et al., “Love, Lies, and Language Models: Investigating AI's Role in Romance-Baiting Scams” (revised 2026-04-18): https://arxiv.org/abs/2512.16280
- USENIX Security 2026, “Love, Lies, and Language Models” (accessed 2026-08-04): https://www.usenix.org/conference/usenixsecurity26/presentation/gressel
- WIRED, “AI Scammers Are Better at Building Trust Than Humans” (2026-07-30): https://www.wired.com/story/ai-scammers-are-better-at-building-trust-than-humans/
- Google AI for Developers, “Abuse monitoring” (updated 2026-06-09): https://ai.google.dev/gemini-api/docs/usage-policies