Sites that block AI training crawlers mostly ignore the answer time bots

·Hacker News··

We read the robots.txt of the top 10,000 sites. 38% of the GPTBot rules we could date were written in a single quarter of 2023 — right after GPTBot launched and the Times sued. Then every AI company quietly added a second bot, the one that answers questions in real time, and almost nobody wrote a rule for it.

Read full article →

Related Articles

Top researchers leave USA for the Netherlands (in Dutch)
28304283409234 · Hacker News · 7h ago
GLM 5.2 and the coming AI margin collapse
martinald · Hacker News · 21h ago
OpenWrt One – Open Hardware Router
peter_d_sherman · Hacker News · 23h ago
Microsoft Can Track Users via a Windows Device ID
ifh-hn · Hacker News · 9h ago
How to sequence your own DNA at home
bilsbie · Hacker News · 18h ago