I noticed that some bots are downloading substitutes from ci.guix.gnu.org. We should add a robots.txt file to reduce this waste.
Specifically, I see bots from Bing and Semrush: https://www.bing.com/bingbot.htm https://www.semrush.com/bot.html
- bug#52338: Crawler bots... Leo Famulari
- bug#52338: [mainte... Leo Famulari
- bug#52338: Cra... Mathieu Othacehe
- bug#52338:... Tobias Geerinckx-Rice via Bug reports for GNU Guix
- bug#52338: Crawler... Mark H Weaver
- bug#52338: Cra... Tobias Geerinckx-Rice via Bug reports for GNU Guix
