Re: Blocking tens of thousands of IP's

2016-11-14 Thread mex
How do you transfer metrics from nginx to your pfsense? mayak Wrote: --- > We are blocking 2.2 million addresses, however, we do it at the > firewall/router (pfsense pfBlocker). > > Ultra fast. > > HTH > > Mayak > > ___

Re: Bloking Bad bots

2016-11-14 Thread Star Dot
You could also look at the nginx module naxsi : https://github.com/nbs-system/naxsi More flexibility with regex and actions -- StackStar Managed Hosting Services : https://www.stackstar.com Shift8 Web Design in Toronto : https://www.shift8web.ca On Mon, Nov 14, 2016 at 10:04 AM, debilish99 wr

Re: Is this a valid request?

2016-11-14 Thread Star Dot
Dont see any traversal or injection attempt, but not knowing what is a "legitimate" request or the application architecture, its difficult to comment further. -- StackStar Managed Hosting Services : https://www.stackstar.com Shift8 Web Design in Toronto : https://www.shift8web.ca On Mon, Nov 14

Re: Hide a request cookie in proxy_pass

2016-11-14 Thread jwal
Oops: this is the correct link: https://regex101.com/r/RZltB6/1 Posted at Nginx Forum: https://forum.nginx.org/read.php?2,252944,270942#msg-270942 ___ nginx mailing list nginx@nginx.org http://mailman.nginx.org/mailman/listinfo/nginx

Re: Hide a request cookie in proxy_pass

2016-11-14 Thread jwal
Hi, Thanks for this; it is pretty close to what I need. I just tried it out in the regex101.com editor and I think there might be a vulnerability: https://regex101.com/delete/ypHV2Yw6o3wHqGDQTHRPZw3r The client could include the same cookie name in twice. This regexp would only strip out one of t

Re: Bloking Bad bots

2016-11-14 Thread George
I use nginx maps which depending on user agent either block, rate limit or whitelist https://community.centminmod.com/threads/blocking-bad-or-aggressive-bots.6433/ as the list gets large nginx maps just make it easier to manage Posted at Nginx Forum: https://forum.nginx.org/read.php?2,270930,270

Re: Bloking Bad bots

2016-11-14 Thread Ph. Gras
Hi there ! so I do, with 2 different ways : == if ($http_user_agent ~* MJ12bot|SemrushBot) { return 403; } if ($http_user_agent ~* bot|crawl|spider|tools|java) { rewrite ^ http://www.cnrtl.fr/definitio

Re: Bloking Bad bots

2016-11-14 Thread lists
fwiw, I use the map approach discussed here. I've a list of a hundred or so 'bad bots'. I reply with a 444. Screw 'em. IMO, the performance hit of blocking them is far less than the performance havoc they wreak if allowed to (try to) scan your site, &/or the inevitable flood of crap from you

Re: Bloking Bad bots

2016-11-14 Thread lists
Comparing strings is CS101. If map is a  linear search, that should be something to improve.I'm assuming you read the code 

Re: Bloking Bad bots

2016-11-14 Thread Robert Paprocki
On Mon, Nov 14, 2016 at 8:51 AM, wrote: > I'd be shocked if the map function doesn't use a smart search scheme > rather than check every item. > You're in for a bit of a shock then. It is a linear search :p Curious as to what you think it should look like instead? Getting back to the original q

Re: Bloking Bad bots

2016-11-14 Thread lists
I'd be shocked if the map function doesn't use a smart search scheme rather than check every item.  

Re: Internal IP in HTTP Location Header Response?

2016-11-14 Thread mevans336
Actually, I think this may have been because after upgrading Nginx, it reinstalled the default.conf file. I've removed it, left the config above, restarted Nginx, and the internal IP doesn't seem to be leaking any longer. Posted at Nginx Forum: https://forum.nginx.org/read.php?2,270932,270935#msg

Re: x-accel-redirect to @location and empty $upstream_http_some_header

2016-11-14 Thread Руслан Закиров
On Mon, Nov 14, 2016 at 3:45 PM, Maxim Dounin wrote: > Hello! > > On Mon, Nov 14, 2016 at 02:13:18PM +0300, Руслан Закиров wrote: > > > One URL redirects to @streams location: > > > > HTTP/1.0 200 OK > > expires: 0 > > cache-control: no-cache, no-store, must-revalidate > > x-accel-redirect: @stre

Re: Bloking Bad bots

2016-11-14 Thread Anoop Alias
I had asked the same question once and got no to the point response. So here is what I infer: the if causes nginx to check the header for each request against the list of patterns you have configured and return a 403 if found . So the processing slows down on each request to for the if processin

Internal IP in HTTP Location Header Response?

2016-11-14 Thread mevans336
Hello - we have been dinged on our network penetration test because one of our Nginx web servers is returning the internal IP in the HTTP location response header. This is our only Nginx server that is not acting as a reverse proxy, so I'm at a bit of a loss on how to disable Nginx returning the In

Re: Bloking Bad bots

2016-11-14 Thread lists
You can block some of those bots at the firewall permanently.   I use the nginx map feature in a similar manner, but I don't know if map is more efficient than your code. ‎I started out blocking similar to your scheme, but the map feature looks clear to me in the conf file. Majestic and Sogou s

Bloking Bad bots

2016-11-14 Thread debilish99
Hello, I have a server with several domains, in the configuration file of each domain I have a line like this to block bad bots. If ($ http_user_agent ~ * (zealbot|MJ12bot|AhrefsBot|sogou|PaperLiBot|uipbot|DotBot|GetIntent|Cliqzbot|YandexBot|Nutch|TurnitinBot|IndeedBot) Return 403; } This works

Re: AW: AW: AW: RE: slow https performance compared to http

2016-11-14 Thread adrhc
Lukas Tribus Wrote: --- > > 4 threads and 4 CPU (both for apache and nginx) with 100% CPU load > on test > > So, what's the answer now about the http/https (4600/550) ratio for > the > > specific case I presented? > > It should perform the same a

Re: custom logic after connection is closed

2016-11-14 Thread nemster
Hi Maxim, Maxim Dounin Wrote: --- > Hello! > > On Mon, Nov 14, 2016 at 05:08:33AM -0500, nemster wrote: > > > Hi! > > Is it possible to write a plugin that does some additional stuff > after a TLS > > (http/1.1, http/2.0) TCP connection. > > I

Re: x-accel-redirect to @location and empty $upstream_http_some_header

2016-11-14 Thread Maxim Dounin
Hello! On Mon, Nov 14, 2016 at 02:13:18PM +0300, Руслан Закиров wrote: > One URL redirects to @streams location: > > HTTP/1.0 200 OK > expires: 0 > cache-control: no-cache, no-store, must-revalidate > x-accel-redirect: @streams > Content-Type: text/html; charset=utf-8 > Status: 200 > x-real-loca

Re: custom logic after connection is closed

2016-11-14 Thread Maxim Dounin
Hello! On Mon, Nov 14, 2016 at 05:08:33AM -0500, nemster wrote: > Hi! > Is it possible to write a plugin that does some additional stuff after a TLS > (http/1.1, http/2.0) TCP connection. > I would want to keep some extra struct for each TLS connection and > manipulated it with every request, th

x-accel-redirect to @location and empty $upstream_http_some_header

2016-11-14 Thread Руслан Закиров
Hi, One URL redirects to @streams location: HTTP/1.0 200 OK expires: 0 cache-control: no-cache, no-store, must-revalidate x-accel-redirect: @streams Content-Type: text/html; charset=utf-8 Status: 200 x-real-location: /stream/?user_id=153847603&lang=RU pragma: no-cache @streams Location looks li

AW: AW: AW: RE: slow https performance compared to http

2016-11-14 Thread Lukas Tribus
> 4 threads and 4 CPU (both for apache and nginx) with 100% CPU load on test > So, what's the answer now about the http/https (4600/550) ratio for the > specific case I presented? It should perform the same as Apache in this case. ___ nginx mailing list

Is this a valid request?

2016-11-14 Thread li...@lazygranch.com
I keep my nginx server set up dumb. (Don't need anything fancy at the moment). Is this request below possibly valid? I flag anything with a question mark in it as hacking, but maybe IOS makes some requests that some websites will process, and others would just ignore after the question mark. 444

custom logic after connection is closed

2016-11-14 Thread nemster
Hi! Is it possible to write a plugin that does some additional stuff after a TLS (http/1.1, http/2.0) TCP connection. I would want to keep some extra struct for each TLS connection and manipulated it with every request, then once the TLS (or TCP) connection closes i would want to do some processin

Re: does not work on any request on windows

2016-11-14 Thread ganadara
update information. This is the result of ngx_http_wait_request_handler after c->recv(c, b->last, size). The normal response has been successfully responded to the user request. Abnormal does not respond to user requests. -

Re: AW: AW: RE: slow https performance compared to http

2016-11-14 Thread adrhc
Lukas Tribus Wrote: --- > That depends: how many nginx workers do you have compared to > how many apache threads and how does your per-core CPU load > look like when benchmarking? > ___ > nginx mailing l

AW: AW: RE: slow https performance compared to http

2016-11-14 Thread Lukas Tribus
> I agree but I think that separate/different simultaneous users won't use a > common connection so for this very specific scenario keep-alive won't > matter. Of course for every individual user keep-alive will matter but this > aspect for the moment I won't to ignore in testing. It does matter, a

Re: AW: RE: slow https performance compared to http

2016-11-14 Thread adrhc
Lukas Tribus Wrote: --- > > I'll do it but I guess the test will no longer be so relevant > because I want > > to simulate different users. > > Real user/browser DO keep-alive. ... I agree but I think that separate/different simultaneous users