Sites that block AI training crawlers mostly ignore the answer time bots

·Hacker News··

We read the robots.txt of the top 10,000 sites. 38% of the GPTBot rules we could date were written in a single quarter of 2023 — right after GPTBot launched and the Times sued. Then every AI company quietly added a second bot, the one that answers questions in real time, and almost nobody wrote a rule for it.

Read full article →

Related Articles

Kobo can run apps now
thepoet · Hacker News · 3h ago
Malicious Rust crate Arrayref runs a build-time payload
abhisek · Hacker News · 1d ago
Japan tried to build an operating system for the world, the US intervened
rdmuser · Hacker News · 14h ago
Cancer-related mortality among US pilots and flight attendants
jader201 · Hacker News · 4h ago
AliExpress runs silent WebAudio fingerprinting that breaks Bluetooth multipoint
emctech · Hacker News · 1d ago