On 4/21/25 12:59 PM, Mark Wielaard wrote:
Hi hackers,
TLDR; When using https://patchwork.sourceware.org or Bunsen
https://builder.sourceware.org/testruns/ you might now have to enable
javascript. This should not impact any scripts, just browsers (or bots
pretending to be browsers). If it does cause trouble, please let us
know. If this works out we might also "protect" bugzilla, gitweb,
cgit, and the wikis this way.
We don't like to hav to do this, but as some of you might have noticed
Sourceware has been fighting the new AI scraperbots since start of the
year. We are not alone in this.
https://lwn.net/Articles/1008897/
https://arstechnica.com/ai/2025/03/devs-say-ai-crawlers-dominate-traffic-forcing-blocks-on-entire-countries/
We have tried to isolate services more and block various ip-blocks
that were abusing the servers. But that has helped only so much.
Unfortunately the scraper bots are using lots of ip addresses
(probably by installing "free" VPN services that use normal user
connections as exit point) and pretending to be common
browsers/agents. We seem to have to make access to some services
depend on solving a javascript challenge.
Jan Wildeboer, on the fediverse, has a pretty interesting lead on how AI
scrapers might be doing this:
https://social.wildeboer.net/@jwildeboer/114360486804175788 (this is the
last post in the thread because it was hard to actually follow the
thread given the number of replies, please go all the way up and read
all 8 posts).
Essentially, there's a library developer that pays developers to just
"include this library and a few more lines in your TOS". This library
then allows the app to sell the end-user's bandwidth to clients of the
library developer, allowing them to make requests. This is how big
companies are managing to have so many IP addresses, so many of those
being residential IP addresses, and it also means that by blocking those
IP addresses we will be - necessarily - blocking real user traffic to
our platforms.
I'm happy to see that the sourceware is moving to a more comprehensive
solution, and if this is successful, I'd suggest that we also try to do
that to the forgejo instance, and remove the IPs blocked because of this
scraping.
So we have installed Anubis https://anubis.techaro.lol/ in front of
patchwork and bunsen. This means that if you are using a browser that
identifies as Mozilla or Opera in their User-Agent you will get a
brief page showing the happy anime girl that requires javascript to
solve a challenge and get a cookie to get through. Scripts and search
engines should get through without. Also removing Mozilla and/or Opera
from your User-Agent will get you through without javascript.
We want to thanks Xe Iaso who has helped us set this up and worked
with use over the Easter weekend solving some of our problems/typos.
Please check out if you want to be one of their patrons as thank you.
https://xeiaso.net/notes/2025/anubis-works/
https://xeiaso.net/patrons/
Cheers,
Mark
--
Cheers,
Guinevere Larsen
She/Her/Hers
