Sites that block AI training crawlers mostly ignore the answer time bots

·Hacker News··

We read the robots.txt of the top 10,000 sites. 38% of the GPTBot rules we could date were written in a single quarter of 2023 — right after GPTBot launched and the Times sued. Then every AI company quietly added a second bot, the one that answers questions in real time, and almost nobody wrote a rule for it.

Read full article →

Related Articles

Field measurements of neighborhood-scale air temperature impacts of data centers
cwwc · Hacker News · 15h ago
Solo – a .so loader for static Linux binaries
zX41ZdbW · Hacker News · 9h ago
Linux 7.3 improves performance when running out of vRAM
flaburgan · Hacker News · 1d ago
Memory prices climb 500% in 12 months
haunter · Hacker News · 1d ago
A 3D fruit fly on macOS desktop powered by the real FlyWire connectome
phoenix120 · Hacker News · 11h ago