Probing is not enough; a validity audit for any probe

·LessWrong··

tl;drA probe can have excellent AUROC and yet fail as a safety signal. I audited 3 probes: a monitoring awareness probe leakage example, a refusal direction as positive control, and Apollo's deception probe as a published protocol case study. In the leakage case, the probe achieves AUROC 1.00, but falls to 0.50 when I remove a single prompt tag. And removing a random span of same length barely dents it. That result uses only deterministic components (avoids LLM judges) and replicates cleanly acr...

Read full article →

Related Articles

Why there are still no non-WebKit browsers on iOS
lapcat · Hacker News · 9h ago
OpenSSL: Buffer overflow in EVP_EncodeBlock() (CVE-2026-121481, CVSS 9.8)
JacobHenner · Hacker News · 22h ago
Unauthenticated remote code execution in Nginx
speedping · Hacker News · 2d ago
GIMP now has UI tests
qtqtqt · Hacker News · 20h ago
Pancreatic cancer halted by virus injection in three patients
cybermango · Hacker News · 1d ago