This is almost certainly not Google as they obey robots.txt. The & to & conversion is another sign of a poor quality crawler. Check the RDNS and you will find it's probably some IP faking Google UA, I suggest blocking at network level.
On Fri, Jun 8, 2018 at 1:57 AM shiz <nginx-fo...@forum.nginx.org> wrote: > Hi, > > Recently, Google has started spidering my website and in addition to normal > pages, appended "&" to all urls, even the pages excluded by robots.txt > > e.g. page.php?page=aaa -> page.php?page=aaa& > > Any idea how to redirect/rewrite this? > > Posted at Nginx Forum: > https://forum.nginx.org/read.php?2,280093,280093#msg-280093 > > _______________________________________________ > nginx mailing list > nginx@nginx.org > http://mailman.nginx.org/mailman/listinfo/nginx >
_______________________________________________ nginx mailing list nginx@nginx.org http://mailman.nginx.org/mailman/listinfo/nginx
