On 3/14/25 18:47, Nick Owens wrote: ...
sorry to hear about AI's latest victim. i had this problem on my gitea instance running on openbsd, where the crawler decided to follow every link to every revision of my mirrors of openbsd src and linux, and i "fixed" it with robots.txt which the particular crawler ("claudebot") respected. robots.txt:User-agent: * Disallow: /
yes, there is a robots.txt. Apparently some scrapers ignore it. yay. Nick.
