hi Andre, thank you very much this was very educative but in my case it is little bit different. The server is no flooded, there is maybe dozen of very sophisticated connections that somehow hangs apache workers threads and the effect is permanent. Quickly the pool is exhausted and the only solution that works is to restart tomcat.
I think it is a bug similar to this one from mod_proxy: https://tools.cisco.com/security/center/viewAlert.x?alertId=34971 I think also that your solution with setting connectionTimeout will solve the problem, at least partially. THANK YOU. I would like to help you further investigate this issue, as our server comes under such attack once or twice in a week. best regards, artur 2016-11-27 12:46 GMT+01:00 André Warnier (tomcat) <a...@ice-sa.com>: > Hi. > > Have a look that the indicated parameters in the two pages below. > You may be the target of such a variant of DDoS attack : many clients open > a TCP connection to your server (front-end), but then never sends a HTTP > request on that connection. In the meantime, the server accepts the TCP > connection, and passes it on to a "child" process or thread for > processing. The child then waits for the HTTP request line to arrive on > the connection (during a certain time), but it never arrives. After a > while, this triggers a timeout (see below), but the standard value of that > timeout may be such that in the meantime, a lot of other connections have > been established by other such nefarious clients, so a lot of resources of > the webserver are tied up, waiting for something that will never come. > Since there is never any real request sent on the connection, you would > (probably) not see this in the logs either. > > The above is the basic mechanism of such an attack. There may be > variations, such as the client not "not sending" a request line, but > sending it extremely slowly, thus achieving perhaps similar kinds of > effects. > > As someone pointed out, it is quite difficult to do something about this > at the level of the webserver itself, because by the time you would do > something about it, the resources have already been consumed and your > server is probably already overloaded. > There are specialised front-end devices and software available, to detect > and protect against this kind of attack. > > You may want to have a look at the following parameters, but make sure to > read the caveats (side-effects, interlocking timeouts etc.), otherwise you > may do more harm than good. > > Another thing : the settings below are for Apache Tomcat, which in your > case is the back-end. It would of course be much better to detect and > eliminate this at the front-end, or even before. I had a look at the > Apache httpd documentation, and could not find a corresponding parameter. > But I am sure that it must exist. You may want to post this same question > on the Apache httpd user's list for a better response. > > Tomcat configuration settings : > > AJP Connector : (http://tomcat.apache.org/tomcat-8.5-doc/config/ajp.html# > Standard_Implementations) > > connectionTimeout > > The number of milliseconds this Connector will wait, after accepting a > connection, for the request URI line to be presented. The default value for > AJP protocol connectors is -1 (i.e. infinite). > > (You could for example try to set this to 3000 (milliseconds) or even > lower. That should be more than enough for any legitimate client so send > the HTTP request line. Note however that by doing this at the Tomcat > level, you will probably move the problem to the Apache httpd/mod_jk > level. But at least it might confirm that this is the problem that you are > seeing. The mod_jk logfile at the httpd level may give you some hints > there.) > > > HTTP Connector : (http://tomcat.apache.org/tomc > at-8.5-doc/config/http.html#Standard_Implementation) > > connectionTimeout > > The number of milliseconds this Connector will wait, after accepting a > connection, for the request URI line to be presented. Use a value of -1 to > indicate no (i.e. infinite) timeout. The default value is 60000 (i.e. 60 > seconds) but note that the standard server.xml that ships with Tomcat sets > this to 20000 (i.e. 20 seconds). Unless disableUploadTimeout is set to > false, this timeout will also be used when reading the request body (if > any). > > > > On 26.11.2016 09:57, Jaaz Portal wrote: > >> hi, >> sorry, its mod_jk no jk2, my typo. All at latest versions. We tried with >> mod proxy too. >> >> There is no flood of the server. Nobody is flooding us, they use some >> specific connections after which pool of apache workers is exhausted and >> blocked >> and we need to restart tomcat server. >> It is some kind of exploit but do not know how to log it to obtain >> details. >> >> i had put a limit on connections per client with hope that this will help >> but once again, it is not a flood. >> They open several connections that are not dropped by apache when they >> disconnect. This way whole pool is quickly exhausted and the server >> broken. >> >> i would like to help you to figure details of this attack but this is >> production server so it is impossible to much debugging options >> >> best, >> artur >> >> 2016-11-25 23:44 GMT+01:00 Niranjan Babu Bommu <niranjan.bo...@gmail.com >> >: >> >> you can find who is flooding site in apache access.log and block them in >>> firewall. >>> >>> ex to find the IP: >>> >>> cat /var/log/apache2/access.log |cut -d' ' -f1 |sort |uniq -c|sort -gr >>> >>> >>> >>> On Fri, Nov 25, 2016 at 8:42 AM, Jaaz Portal <jaazpor...@gmail.com> >>> wrote: >>> >>> hi, >>>> we are from some weeks struggling with some Polish hackers that are >>>> bringing our server down. After updating apache to latest version >>>> >>> (2.4.23) >>> >>>> and tomcat (8.0.38) available for debian systems we still cannot secure >>>> >>> our >>> >>>> server. >>>> >>>> Today it has stopped to respond again and we needed to restart tomcat >>>> process to get it back alive. >>>> >>>> There is no too much clues in the logs. The apache error.log gives just >>>> this line: >>>> >>>> [Fri Nov 25 13:08:00.647835 2016] [mpm_event:error] [pid 13385:tid >>>> 1397934896385 >>>> 92] AH00484: server reached MaxRequestWorkers setting, consider raising >>>> >>> the >>> >>>> MaxR >>>> equestWorkers setting >>>> >>>> seems that somehow tomcat, mod-jk2 or even apache is vulnerable to some >>>> >>> new >>> >>>> exploit, as we certainly does not have such traffic that would block our >>>> server otherwise >>>> >>>> for now we have increased MaxRequestWorkers and we have limited number >>>> of >>>> connections from one client to 5 by mod_bw and limited number of >>>> simultaneous connections from one ip by iptables but does not know if >>>> >>> this >>> >>>> will help >>>> >>>> best regards, >>>> artur >>>> >>>> >>> >>> >>> -- >>> *Thanks* >>> *Niranjan* >>> >>> >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
