Re: Forum crawling restriction for Qwant

2025-05-26 Thread Dave Fisher
In this case none of their IP addresses are in any of our blocks. I looked at the access logs and they are respecting our robots.txt file. I’m opening up the robots.txt files again. > On May 26, 2025, at 5:28 AM, Bidouille wrote: > >> Without knowing the IP addresses they crawl from it is diff

Re: Forum crawling restriction for Qwant

2025-05-26 Thread Bidouille
> Without knowing the IP addresses they crawl from it is difficult to > know how to enable them. According to this page: https://help.qwant.com/bot/ Several IP are used and stored in a json file 91.242.162.0/24 - To unsubscribe,

Re: Forum crawling restriction for Qwant

2025-05-24 Thread Dave Fisher
> On May 24, 2025, at 1:24 AM, Bidouille wrote: > > Hello, > When I try to search through Qwant with "site:" parameter > it displays that it can't browse on the forum. > > https://i.postimg.cc/9fnF8tZJ/capture.png > > Qwant is a free search engine in France. > Could you have a look to ckecko