FEATUREDComputing Life · Share (鸭哥 research reports)· rssZH00:00 · 08·02
→Prompt injection defense lives in the harness, not the model
Ghostcommit showed the same Sonnet model rejected malicious PNG instructions 10/10 times in Claude Code, but obeyed 10/10 times in Cursor and Antigravity, leaking .env secrets. Lab-reported 99% defense rates suffer from five traps: static benchmark overfitting, misleading single-attempt ASR, LLM-as-judge drift, ignored utility-under-attack, and bare-model testing without tool shells. Deeper causes: LLMs lack hard instruction-data separation, and stronger models can follow injections more faithfully—Opus 4.6 with extended thinking saw ASR rise from 14.8% to 21.7%. A joint study by 14 researchers from OpenAI, Anthropic, and DeepMind tested 12 model-layer defenses; over 90% broke under adaptive attacks, with human red-teamers hitting 100%. The engineering fix is architectural isolation: CaMeL separates trusted planner from untrusted executor, and OpenClaw's dual-agent setup cut ASR from 100% to 0.31%. Harness-level deterministic tool gating, hook signature checks, and sandboxed least-privilege are the real controls.
#Agent#Safety#Anthropic#Claude Code
why featured
Featured · importance 82 · hook + knowledge + resonance
editor take
Same Sonnet model leaked secrets 10/10 times in Cursor but 0/10 in Claude Code—lab-reported 99% defense rates rarely hold in production.
sharp
This piece is worth reading because the Ghostcommit experiment makes something concrete that many people suspected: the model isn't what stops prompt injection—the harness around it is. Same Sonnet weights, 10/10 rejections in Claude Code, 10/10 compliance in Cursor and Antigravity, .env secrets pushed to a public repo.
The second half dissects five traps in lab evaluations. Two stand out. Single-attempt ASR is misleading—attackers try many times and only need one success. And bare-model API testing without tool shells produces numbers that don't map to real engineering environments.
The counterintuitive bit is in Anthropic's own System Card: Opus 4.6 with extended thinking saw ASR rise from 14.8% to 21.7%. Stronger models follow instructions more faithfully, including malicious ones hidden in data. A joint study by 14 researchers from OpenAI, Anthropic, and DeepMind tested 12 model-layer defenses—over 90% broke under adaptive attacks, human red-teamers hit 100%.
The engineering fix is architectural isolation. OpenClaw's dual-agent setup has Agent 1 produce structured summaries of untrusted input, Agent 2 never touches raw text—ASR dropped from 100% to 0.31%. CaMeL separates a trusted planner from an untrusted executor. These approaches don't make the model safer; they make the attack path unviable.
HKR breakdown
hook ✓knowledge ✓resonance ✓