no it looks like dos, its dos i told you they dosed before bind server until we changed it to other vendor, and later was scanning my host for apache vulnerabilities
configuration is standard, the only thing i changed (after your guidance) is connection_timeout but this does not work for this exploit workers.properties worker.list=ajp13_worker # #------ ajp13_worker WORKER DEFINITION ------------------------------ #--------------------------------------------------------------------- # # # Defining a worker named ajp13_worker and of type ajp13 # Note that the name and the type do not have to match. # worker.ajp13_worker.port=8009 worker.ajp13_worker.host=localhost worker.ajp13_worker.socket_timeout=60000 worker.ajp13_worker.type=ajp13 # # Specifies the load balance factor when used with # a load balancing worker. # Note: # ----> lbfactor must be > 0 # ----> Low lbfactor means less work done by the worker. worker.ajp13_worker.lbfactor=1 # # Specify the size of the open connection cache. #worker.ajp13_worker.cachesize # #------ DEFAULT LOAD BALANCER WORKER DEFINITION ---------------------- #--------------------------------------------------------------------- # # # The loadbalancer (type lb) workers perform wighted round-robin # load balancing with sticky sessions. # Note: # ----> If a worker dies, the load balancer will check its state # once in a while. Until then all work is redirected to peer # workers. worker.loadbalancer.type=lb worker.loadbalancer.balance_workers=ajp13_worker ------------ server.xml <Connector port="8009" protocol="AJP/1.3" connectionTimeout="60000" redirectPort="8443" maxConnections="256" keepAliveTimeout="30000"/> best, artur 2016-11-30 19:21 GMT+01:00 Mark Eggers <its_toas...@yahoo.com.invalid>: > Artur, > On 11/30/2016 8:36 AM, Jaaz Portal wrote: > > hi, > > they has tried again with success despite setting connection_timeout and > > limiting number of clients by mod_bw > > the tomcat has frozen again. > > > > netstat does not showed any connections on port 80 but plenty of > > connections from apache to localhost:8009 > > so it was not an attack that you has described (no slowlaris) > > > > im looking into debug files of mod_jk and forensic for some hints. If you > > want i can share them (they have 4mb compressed) > > > > best wishes > > artur > > This is beginning to look like an application or a configuration issue > and not a DOS (or DDOS) attack. > > One the issues that may cause this is a mismatched timeout value between > connection_pool_timeout in workers.properties (mod_jk) and the > connectionTimeout in server.xml (Tomcat) for the AJP connector. > > Also, at least for the mod_jk version that I'm running, there is no > limit for reply_timeout (mod_jk) by default. > > Can you post your workers.properties file and the AJP connector portion > of your server.xml? > > In the conf directory of the mod_jk source code, there is a very nice > workers.properties file that has sensible defaults. If you've not done > so, I suggest that you start with the values specified in that file, and > make sure that the timeout values match (see my comment above). > > Also, when you used mod-proxy, did you use mod-proxy-ajp or > mod-proxy-http? If you used mod-proxy-ajp, then again there could be a > timeout mismatch (or no timeout specified at all). > > . . . just my two cents > /mde/ > > > > > 2016-11-29 11:01 GMT+01:00 André Warnier (tomcat) <a...@ice-sa.com>: > > > >> On 28.11.2016 22:04, Jaaz Portal wrote: > >> > >>> hi Andre, > >>> you are wrong. This vulnerability is not only causing memory leaks, it > >>> makes also apache workers to hang > >>> > >> > >> Maybe for the last time here : > >> > >> - what do you call "apache workers" ? > >> > >> , making it easy to exhaust the pool. > >> > >> - what do you call "the pool" ? > >> > >> what i have in my log files. But it is true also that such exhaustion > can > >>> be made by other forms of dos attacks described in this thread. > >>> > >>> regarding you suggestion on our application, it does not dos bind > server > >>> nether does not scan for various vulnerabilities in apache, what i have > >>> also in the logs > >>> > >> > >> For your information : I run about 25 Internet-facing Apache webservers > >> (some with a back-end tomcat, some not). > >> On every single one of those webservers, there are *hundreds* of such > >> "scans" every day, as shown by the Apache access logs. That is just a > fact > >> of life on the Internet. > >> They are annoying, but most of them are harmless (from an "attack" point > >> of view), because they are scanning for things that we do not run > >> (phpmyadmin, xmlrpc, vti_bin, etc., etc., the list is almost endless), > and > >> thus are responded to by Apache as "404 Not found". > >> What is annoying with those scans, is > >> a) that they fill the logfile, and thus make it more difficult to find > >> really significant things > >> b) that each of those requires some bandwidth and system resource, if > only > >> to return a "404 Not found" (or a "401 Unauthorised"), and that we pay > for > >> that bandwidth and resources. > >> > >> If I could find a way to charge 0.1 cent per access to my servers, from > >> the people who wrote or run the programs who are doing this, I could > retire > >> in luxury. > >> > >> But they are not a real problem, because they are caught as "invalid" by > >> Apache, and rejected quickly, so they cannot do anything really nasty > >> (except if they were sending several thousand such requests per second > to > >> one of my servers for a long time). > >> > >> The ones that are worrying, are the ones > >> - a) which do /not/ end up as a "404 Not found", because they have found > >> an application which responds, and they are not coming from our > legitimate > >> customers > >> - b) /the ones which we do not see/, because they either do not send a > >> valid HTTP request, or they have found a way to trigger one of our > >> applications, in such a way that the application misbehaves and, > perhaps, > >> even if they do not crash our servers, they may provide the attacker > with > >> some entry point to do other things which we do not know and do not > control > >> > >> What I am trying to say here, is /do not jump to premature conclusions/. > >> Such "scans" as you mention, happen to everyone, all the time, from > >> ever-changing IP addresses located all over the world. Some of those > >> "scans" may come from the infected PC of your grandmother, and she does > not > >> even know about it. > >> > >> There is no guarantee, and no indication or proof so far, that /these/ > >> scans are even related to "the other thing" which happens on your > >> webserver, which looked much more focused. > >> > >> So do not just bundle them together as being the same thing, until you > >> have some real data that shows for example that these different things > all > >> come from the same IP addresses. > >> > >> And one more thing, also finally until you come back with some real > data : > >> I am not saying that your application "scans your server". What I am > >> saying is that, maybe, by chance or by design, the attackers have found > a > >> URL which goes to your application, and which causes your application to > >> keep tomcat and/or Apache busy for a long time. > >> And that maybe /that/ is the problem you should be looking for, and not > >> some hypothetical bug in Apache httpd or tomcat. > >> > >> > >> > >>> kindly regards, > >>> artur > >>> > >>> 2016-11-28 21:33 GMT+01:00 André Warnier (tomcat) <a...@ice-sa.com>: > >>> > >>> On 28.11.2016 20:34, Jaaz Portal wrote: > >>>> > >>>> hi mark, > >>>>> yes, i understand now what slowloris attack is. > >>>>> maybe it was this maybe *this one based on * * mod_proxy denial of > >>>>> service * > >>>>> CVE-2014-0117 <http://cve.mitre.org/cgi-bin/ > >>>>> cvename.cgi?name=CVE-2014-0117> > >>>>> > >>>>> > >>>> You keep on saying this, but the description of that vulnerability of > >>>> *Apache httpd*, and the symptoms which you described, *do not match*. > >>>> You described the problem as ocurring in Apache tomcat, which in your > >>>> case > >>>> is sitting as a back-end, *behind* Apache httpd. And restarting tomcat > >>>> cured the problem. > >>>> > >>>> The CVE above applies to Apache httpd, and describes how an attacker > >>>> could > >>>> attack Apache httpd and cause *its children* processes to crash (the > >>>> children processes of Apache httpd), by leading them to consume a lot > of > >>>> memory and crash with an out-of-memory error. > >>>> Granted, the problem occurred in the mod_proxy module of Apache httpd; > >>>> but > >>>> it was httpd which crashed, not tomcat. > >>>> And tomcat processes are not "Apache httpd children processes" in any > >>>> understanding of the term. > >>>> > >>>> As far as I remember, you never mentioned Apache httpd crashing. You > >>>> mentioned "the pool" getting full or satured or something like that, > >>>> without ever describing properly what you meant by "the pool". > >>>> > >>>> As far as I am concerned, according to all the relatively unspecific > >>>> information which you have previously provided : > >>>> 1) the attack - if that is what it is/was - is definitely NOT related > to > >>>> the CVE which you have repeatedly mentioned > >>>> 2) it is apparently not a "classical" DoS or "slowloris DoS" directed > at > >>>> your front-end Apache. Instead, it seems that the requests are > properly > >>>> received by Apache, properly decoded by Apache, and then whatever > Apache > >>>> proxy module you are using (mod_proxy_http, mod_proxy_ajp or mod_jk) > is > >>>> properly forwarding these requests to a back-end tomcat; and it is at > the > >>>> level of that back-end tomcat that the requests never seem to end, > and in > >>>> the end paralyse your tomcat server (and later on maybe your Apache > httpd > >>>> server too, because it is also waiting for tomcat to respond). > >>>> > >>>> So your very way of describing the problem, in terms of "first we used > >>>> this proxy module, and then they exploited the vulnerability so and > so; > >>>> then we changed the proxy module, and they exploited that too; etc.." > >>>> seems to not have anything to do with the problem per se, and (I > believe) > >>>> confuses everyone, including yourself. > >>>> > >>>> It is not that "they" exploited different vulnerabilities of various > >>>> httpd > >>>> proxy modules, one after the other. Each of these proxy modules was > doing > >>>> its job properly, and forwarding valid HTTP requests properly to > tomcat. > >>>> When you changed from one proxy module to another, you did not really > >>>> change anything in that respect, because any proxy module would do the > >>>> same. > >>>> > >>>> But in all cases, what did not change, was the tomcat behind the > >>>> front-end, and the application running on that tomcat. So the > presumed > >>>> attackers did not have to change anything, they just kept on sending > the > >>>> same requests, because they were really targeting your back-end > tomcat or > >>>> the tomcat application in it, no matter /how/ you were forwarding > >>>> requests > >>>> from Apache httpd to tomcat. > >>>> > >>>> So either it is tomcat itself, which has a problem with some request > URLs > >>>> which do not bother Apache httpd (possible, but statistically > unlikely), > >>>> or > >>>> it is the application which runs in tomcat that has such a problem > >>>> (statistically, much more likely). > >>>> > >>>> we do not know yet > >>>> > >>>>> > >>>>> we have setup more logging and are waiting for them to attack once > again > >>>>> > >>>>> > >>>> Yes, that is the right thing to do. Before deciding what the problem > may > >>>> be, and what you can do about it, the first thing you need is *data*. > >>>> You > >>>> need to know > >>>> - which request URL(s?) cause that problem > >>>> - which IPs these requests come from (always the same ? multiple IPs > that > >>>> change all the time ? how many ? can these IPs be valid/expected > clients > >>>> or > >>>> not ? do these IPs look like some "coordinated group" ?) > >>>> - how many such requests there may be during some period of time (10, > >>>> 100, > >>>> 1000, more ?) > >>>> - if these URLs result in passing the request to tomcat > >>>> - what tomcat application (if any) they are directed to > >>>> - if so, when that application receives such a request, what is it > >>>> supposed to do ? does it do it properly ? how long does it need, to > >>>> respond > >>>> to such a request ? > >>>> > >>>> You also need to ask yourself a question : is the application which > you > >>>> run inside tomcat something that you designed yourself (and which > hackers > >>>> are unlikely to know well-enough to find such a URL which paralyses > your > >>>> server) ? or is it some well-known third-party java application which > you > >>>> are running (and for which would-be attackers would be much more > likely > >>>> to > >>>> know exactly such a bug) ? > >>>> > >>>> > >>>> anyway, thank you for all informations, it was very useful and > >>>>> educational > >>>>> reading for all of us > >>>>> > >>>>> best wishes, > >>>>> artur > >>>>> > >>>>> 2016-11-28 19:46 GMT+01:00 Mark Eggers <its_toas...@yahoo.com.invalid > >: > >>>>> > >>>>> Jaaz, > >>>>> > >>>>>> > >>>>>> On 11/27/2016 2:46 PM, André Warnier (tomcat) wrote: > >>>>>> > >>>>>> On 27.11.2016 19:03, Jaaz Portal wrote: > >>>>>>> > >>>>>>> 2016-11-27 18:30 GMT+01:00 André Warnier (tomcat) <a...@ice-sa.com>: > >>>>>>>> > >>>>>>>> On 27.11.2016 14:26, Jaaz Portal wrote: > >>>>>>>> > >>>>>>>>> > >>>>>>>>> hi, > >>>>>>>>> > >>>>>>>>>> everything i know so far is just this single log line that > appeared > >>>>>>>>>> in > >>>>>>>>>> apache error.log > >>>>>>>>>> > >>>>>>>>>> [Fri Nov 25 13:08:00.647835 2016] [mpm_event:error] [pid > 13385:tid > >>>>>>>>>> 1397934896385 > >>>>>>>>>> 92] AH00484: server reached MaxRequestWorkers setting, consider > >>>>>>>>>> > >>>>>>>>>> raising > >>>>>>>>> > >>>>>>>> > >>>>>> the > >>>>>>> > >>>>>>>> MaxR > >>>>>>>>>> equestWorkers setting > >>>>>>>>>> > >>>>>>>>>> there was nothing else, just this strange line > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> This is not a "strange" line. It is telling you something. > >>>>>>>>> One problem is that you seem convinced in advance, without > serious > >>>>>>>>> proof, > >>>>>>>>> that there is a "bug" or a vulnerability in httpd or tomcat. > >>>>>>>>> Read the explanation of the httpd parameter, here : > >>>>>>>>> http://httpd.apache.org/docs/2.4/mod/mpm_common.html#maxrequ > >>>>>>>>> estworkers > >>>>>>>>> and try to understand it. > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> I understand it very well. We are serving no more that 500 > clients > >>>>>>>> per > >>>>>>>> day > >>>>>>>> so there was no other option that some kind of attack. > >>>>>>>> > >>>>>>>> > >>>>>>>> About the "bug" or "vulnerability" : a webserver is supposed to > serve > >>>>>>>> HTTP > >>>>>>>> > >>>>>>>> requests from clients. That is what you expect of it. It has no > >>>>>>>>> choice but > >>>>>>>>> to accept any client connection and request, up to the maximum it > >>>>>>>>> can > >>>>>>>>> handle considering its configuration and the capacity of the > system > >>>>>>>>> on > >>>>>>>>> which it runs. That is not a bug, it is a feature. > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> We have some weeks ago come under attack from some Polish group. > It > >>>>>>>>> > >>>>>>>> was > >>>>>>>> first bind that was DoS'ed, we was running on stable Debian so i > >>>>>>>> updated > >>>>>>>> bind to latest version. It did not helped. They has dos'ed it so > we > >>>>>>>> switched to other dns provider. That has helped. > >>>>>>>> > >>>>>>>> Then they exploited some well know vulnerability in mod_proxy. We > >>>>>>>> have > >>>>>>>> updated apache to the latest but again they has exploited it, so > we > >>>>>>>> have > >>>>>>>> switched to mod_jk. And then guess what. They exploited it too so > i > >>>>>>>> decided > >>>>>>>> to write to this list looking for help before trying jetty. > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> The normal Apache httpd access log, will log a request only when > it > >>>>>>>>> is > >>>>>>>>> finished. If the request never finishes, it will not get logged. > >>>>>>>>> That may be why you do not see these requests in the log. > >>>>>>>>> But have a look at this Apache httpd module : > >>>>>>>>> http://httpd.apache.org/docs/2.4/mod/mod_log_forensic.html > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> ok, thanks, will take care > >>>>>>>> > >>>>>>>> Note : that is also why I was telling you to enable the mod_jk > log, > >>>>>>>> and to > >>>>>>>> > >>>>>>>> examine it. > >>>>>>>>> Because mod_jk will also log information before the request > >>>>>>>>> produces a > >>>>>>>>> response. > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> and server hanged. > >>>>>>>>> > >>>>>>>>> Again, /what/ is "hanged" ? Apache httpd, or tomcat ? > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> Apache was accepting connection but not processed it. After > restart > >>>>>>>> of > >>>>>>>> tomcat server it worked again. > >>>>>>>> > >>>>>>>> > >>>>>>>> Also in > >>>>>>>> > >>>>>>>>> > >>>>>>>>> access logs there are no clues that it was under any heavy load. > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> after around hour after discovering that our server hanged-out > we > >>>>>>>>>> have > >>>>>>>>>> restarted tomcat server > >>>>>>>>>> and it worked again > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> Yes, because that will close all connections between Apache > httpd > >>>>>>>>> and > >>>>>>>>> tomcat, and abort all requests that are in the process of being > >>>>>>>>> processed > >>>>>>>>> by tomcat. So mod_jk will get an error from tomcat, and will > report > >>>>>>>>> an > >>>>>>>>> error to httpd, and httpd will communicate that error to the > >>>>>>>>> clients, > >>>>>>>>> and > >>>>>>>>> close their connection. > >>>>>>>>> It still does not tell you what the problem was. > >>>>>>>>> The only thing that it suggests, is that the "bad" requests > actually > >>>>>>>>> make > >>>>>>>>> it all the way to tomcat. > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> correct > >>>>>>>> > >>>>>>>> i will enable logs that you has pointed out and we will look what > i > >>>>>>>> will > >>>>>>>> catch > >>>>>>>> however i think we have only one chance, as if the solution we has > >>>>>>>> found > >>>>>>>> out (connection_timeout + mod_bn) > >>>>>>>> will work they will stop exploiting it > >>>>>>>> > >>>>>>>> thank you very much for all the help and explanations > >>>>>>>> i will report to the list new facts once they will attack us again > >>>>>>>> > >>>>>>>> best regards, > >>>>>>>> artur > >>>>>>>> > >>>>>>>> > >>>>>>>> Ok, but also read this e.g. : > >>>>>>> https://www.corero.com/blog/695-going-after-the-people- > >>>>>>> > >>>>>>> behind-ddos-attacks.html > >>>>>> > >>>>>> > >>>>>>> Attempts to bring down a site by DoS attacks is a crime, in most > >>>>>>> places.. > >>>>>>> > >>>>>>> You can report it, while at the same time trying to defend yourself > >>>>>>> against it. > >>>>>>> > >>>>>>> It is also relatively easy, and quite inexpensive in terms of > system > >>>>>>> resources, to run a small shell script which takes a list every few > >>>>>>> seconds of the connections to the port of your webserver, and which > >>>>>>> IPs > >>>>>>> they are coming *from*. > >>>>>>> E.g. > >>>>>>> First try the netstat command, to see what it lists, like : > >>>>>>> # netstat -n --tcp | more > >>>>>>> > >>>>>>> Then you will want to filter this a bit, to only consider > established > >>>>>>> connections to your webserver, for example : > >>>>>>> # netstat -n --tcp | grep ":80" | grep "ESTABLISHED" > >>>>>>> > >>>>>>> Then you will want to send this to a logfile, regularly, like this > : > >>>>>>> > >>>>>>> # date >> some_file.log > >>>>>>> # netstat -n --tcp | grep ":80" | grep "ESTABLISHED" >> > some_file.log > >>>>>>> (repeat every 3 seconds) > >>>>>>> > >>>>>>> This will not generate GB of logfiles, and it will tell you, when > the > >>>>>>> problem happens, how many connections there are exactly to your > >>>>>>> webserver, and where they are coming from. > >>>>>>> Then later you can further analyse this information.. > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>>> i think that setting connection-timeout and limiting the number of > >>>>>>>> > >>>>>>>>> clients > >>>>>>>>>> by mod_bd i will > >>>>>>>>>> have effect that next time somebody will try this exploit it > will > >>>>>>>>>> > >>>>>>>>>> block > >>>>>>>>> > >>>>>>>> > >>>>>> his > >>>>>>> > >>>>>>>> access to the site > >>>>>>>>>> for minute or two, pretty good holistic solution i would say > >>>>>>>>>> > >>>>>>>>>> still, it seems that there is serious vulnerability somewhere in > >>>>>>>>>> apache, > >>>>>>>>>> mod_jk or tomcat > >>>>>>>>>> i would like to help find it out but need some hints which debug > >>>>>>>>>> options > >>>>>>>>>> enable to catch the bad guys > >>>>>>>>>> when they will try next time > >>>>>>>>>> > >>>>>>>>>> best regards, > >>>>>>>>>> artur > >>>>>>>>>> > >>>>>>>>>> 2016-11-27 13:58 GMT+01:00 André Warnier (tomcat) < > a...@ice-sa.com>: > >>>>>>>>>> > >>>>>>>>>> On 27.11.2016 13:23, Jaaz Portal wrote: > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>> hi Andre, > >>>>>>>>>>> > >>>>>>>>>>> thank you very much this was very educative but in my case it > is > >>>>>>>>>>>> little > >>>>>>>>>>>> bit > >>>>>>>>>>>> different. > >>>>>>>>>>>> The server is no flooded, there is maybe dozen of very > >>>>>>>>>>>> sophisticated > >>>>>>>>>>>> connections that somehow hangs apache workers threads > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> Can you be a bit more specific ? > >>>>>>>>>>>> > >>>>>>>>>>> When you say "apache workers threads", do you mean threads in > >>>>>>>>>>> Apache > >>>>>>>>>>> httpd, or threads in Apache Tomcat ? (both are Apache > webservers, > >>>>>>>>>>> so it > >>>>>>>>>>> is > >>>>>>>>>>> difficult to tell what you are talking about, unless you are > very > >>>>>>>>>>> precise). > >>>>>>>>>>> > >>>>>>>>>>> Let me give you some additional explanations, and maybe you can > >>>>>>>>>>> > >>>>>>>>>>> figure > >>>>>>>>>> > >>>>>>>>> > >>>>>> out > >>>>>>> > >>>>>>>> exactly where it "hangs". > >>>>>>>>>>> > >>>>>>>>>>> From the Apache httpd front-end point of view, mod_jk (the > >>>>>>>>>>> connector to > >>>>>>>>>>> Apache Tomcat) is basically one among other "response > generators". > >>>>>>>>>>> Apache > >>>>>>>>>>> httpd does not "know" that behind mod_jk, there is one or more > >>>>>>>>>>> Tomcat > >>>>>>>>>>> servers. Apache httpd receives the original client request, > and > >>>>>>>>>>> depending > >>>>>>>>>>> on the URL of the request, it will pass it to mod_jk or to > another > >>>>>>>>>>> response > >>>>>>>>>>> generator, to generate the response to the request. > >>>>>>>>>>> That mod_jk in the background is using a Tomcat server to > actually > >>>>>>>>>>> generate the response, is none of the business of Apache httpd, > >>>>>>>>>>> and > >>>>>>>>>>> > >>>>>>>>>>> it > >>>>>>>>>> > >>>>>>>>> > >>>>>> does > >>>>>>> > >>>>>>>> not care. All it cares about, is to actually receive the response > >>>>>>>>>>> > >>>>>>>>>>> from > >>>>>>>>>> > >>>>>>>>> > >>>>>> mod_jk, and pass it back to the client. > >>>>>>> > >>>>>>>> > >>>>>>>>>>> If httpd passes a request to mod_jk, it is because you have > >>>>>>>>>>> specified in > >>>>>>>>>>> the Apache configuration, the type of URL that it should pass > to > >>>>>>>>>>> mod_jk.. > >>>>>>>>>>> That happens via your "JkMount (URL pattern)" directives in > Apache > >>>>>>>>>>> httpd. > >>>>>>>>>>> > >>>>>>>>>>> Of course Apache httpd will not pass a request to mod_jk, > before > >>>>>>>>>>> it > >>>>>>>>>>> has > >>>>>>>>>>> received at least the URL of the request, and analysed it to > >>>>>>>>>>> > >>>>>>>>>>> determine > >>>>>>>>>> > >>>>>>>>> > >>>>>> *if* > >>>>>>> > >>>>>>>> it should pass it to mod_jk (*). > >>>>>>>>>>> > >>>>>>>>>>> If the mod_jk logging is enabled, you can see in it, exactly > >>>>>>>>>>> *which* > >>>>>>>>>>> requests are passed to mod_jk and to Tomcat. > >>>>>>>>>>> Do you know *which* requests, from which clients, cause this > >>>>>>>>>>> "thread > >>>>>>>>>>> hanging" symptom ? > >>>>>>>>>>> Once you would know this, maybe you can design a strategy to > block > >>>>>>>>>>> specifically these requests. > >>>>>>>>>>> > >>>>>>>>>>> and the effect is permanent. Quickly the pool is exhausted > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>> which pool exactly ? > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> and the only > >>>>>>>>>>> > >>>>>>>>>>> solution that works is to restart tomcat. > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>>> I think it is a bug similar to this one from mod_proxy: > >>>>>>>>>>>> https://tools.cisco.com/security/center/viewAlert.x?alertId= > >>>>>>>>>>>> 34971 > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> Maybe, maybe not. As long as we do not know what the requests > >>>>>>>>>>>> are, > >>>>>>>>>>>> that > >>>>>>>>>>>> > >>>>>>>>>>>> block things, we do not know this. > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> I think also that your solution with setting connectionTimeout > >>>>>>>>>>> will > >>>>>>>>>>> solve > >>>>>>>>>>> > >>>>>>>>>>> the problem, at least partially. THANK YOU. > >>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> Same thing, we do not know this yet. It was only giving this > >>>>>>>>>>>> > >>>>>>>>>>> explanation, > >>>>>>>>>>> to help you think about where the problem may be. > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> I would like to help you further investigate this issue, as our > >>>>>>>>>>> > >>>>>>>>>>> server > >>>>>>>>>> > >>>>>>>>> > >>>>>> comes under such attack once or twice in a week. > >>>>>>> > >>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> Other than giving you hints, there is not much I or anyone > else > >>>>>>>>>>>> can do > >>>>>>>>>>>> > >>>>>>>>>>>> to > >>>>>>>>>>> help. You are the one with control of your servers and > logfiles, > >>>>>>>>>>> so > >>>>>>>>>>> you > >>>>>>>>>>> have to investigate and provide more precise information. > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> (*) actually, to be precise, Apache httpd passes *all* > requests to > >>>>>>>>>>> mod_jk, > >>>>>>>>>>> to ask it "if it wants that request". mod_jk then accepts or > >>>>>>>>>>> > >>>>>>>>>>> declines, > >>>>>>>>>> > >>>>>>>>> > >>>>>> depending on the JkMount instructions. If mod_jk declines, then > >>>>>>> > >>>>>>>> > >>>>>>>>>>> Apache > >>>>>>>>>> > >>>>>>>>> > >>>>>> httpd will ask the next response generator if it wants this request, > >>>>>>> > >>>>>>>> etc... > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> best regards, > >>>>>>>>>>> > >>>>>>>>>>> artur > >>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> 2016-11-27 12:46 GMT+01:00 André Warnier (tomcat) < > a...@ice-sa.com > >>>>>>>>>>>>> : > >>>>>>>>>>>> > >>>>>>>>>>>> Hi. > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> Have a look that the indicated parameters in the two pages > >>>>>>>>>>>>> below.. > >>>>>>>>>>>>> > >>>>>>>>>>>>> You may be the target of such a variant of DDoS attack : many > >>>>>>>>>>>>> clients > >>>>>>>>>>>>> open > >>>>>>>>>>>>> a TCP connection to your server (front-end), but then never > >>>>>>>>>>>>> sends > >>>>>>>>>>>>> a > >>>>>>>>>>>>> HTTP > >>>>>>>>>>>>> request on that connection. In the meantime, the server > accepts > >>>>>>>>>>>>> > >>>>>>>>>>>>> the > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>> TCP > >>>>>>> > >>>>>>>> connection, and passes it on to a "child" process or thread for > >>>>>>>>>>>>> processing. The child then waits for the HTTP request line > to > >>>>>>>>>>>>> arrive > >>>>>>>>>>>>> on > >>>>>>>>>>>>> the connection (during a certain time), but it never arrives. > >>>>>>>>>>>>> After a > >>>>>>>>>>>>> while, this triggers a timeout (see below), but the standard > >>>>>>>>>>>>> value of > >>>>>>>>>>>>> that > >>>>>>>>>>>>> timeout may be such that in the meantime, a lot of other > >>>>>>>>>>>>> > >>>>>>>>>>>>> connections > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>> have > >>>>>>> > >>>>>>>> been established by other such nefarious clients, so a lot of > >>>>>>>>>>>>> resources > >>>>>>>>>>>>> of > >>>>>>>>>>>>> the webserver are tied up, waiting for something that will > never > >>>>>>>>>>>>> come.. > >>>>>>>>>>>>> Since there is never any real request sent on the connection, > >>>>>>>>>>>>> you > >>>>>>>>>>>>> would > >>>>>>>>>>>>> (probably) not see this in the logs either. > >>>>>>>>>>>>> > >>>>>>>>>>>>> The above is the basic mechanism of such an attack. There > may > >>>>>>>>>>>>> be > >>>>>>>>>>>>> variations, such as the client not "not sending" a request > line, > >>>>>>>>>>>>> > >>>>>>>>>>>>> but > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>> sending it extremely slowly, thus achieving perhaps similar kinds > >>>>>>> > >>>>>>>> > >>>>>>>>>>>>> of > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>> effects. > >>>>>>> > >>>>>>>> > >>>>>>>>>>>>> As someone pointed out, it is quite difficult to do something > >>>>>>>>>>>>> about > >>>>>>>>>>>>> this > >>>>>>>>>>>>> at the level of the webserver itself, because by the time you > >>>>>>>>>>>>> would do > >>>>>>>>>>>>> something about it, the resources have already been consumed > and > >>>>>>>>>>>>> your > >>>>>>>>>>>>> server is probably already overloaded. > >>>>>>>>>>>>> There are specialised front-end devices and software > available, > >>>>>>>>>>>>> to > >>>>>>>>>>>>> detect > >>>>>>>>>>>>> and protect against this kind of attack. > >>>>>>>>>>>>> > >>>>>>>>>>>>> You may want to have a look at the following parameters, but > >>>>>>>>>>>>> make > >>>>>>>>>>>>> sure > >>>>>>>>>>>>> to > >>>>>>>>>>>>> read the caveats (side-effects, interlocking timeouts etc.), > >>>>>>>>>>>>> otherwise > >>>>>>>>>>>>> you > >>>>>>>>>>>>> may do more harm than good. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Another thing : the settings below are for Apache Tomcat, > which > >>>>>>>>>>>>> in your > >>>>>>>>>>>>> case is the back-end. It would of course be much better to > >>>>>>>>>>>>> detect > >>>>>>>>>>>>> and > >>>>>>>>>>>>> eliminate this at the front-end, or even before. I had a > look > >>>>>>>>>>>>> at > >>>>>>>>>>>>> the > >>>>>>>>>>>>> Apache httpd documentation, and could not find a > corresponding > >>>>>>>>>>>>> parameter. > >>>>>>>>>>>>> But I am sure that it must exist. You may want to post this > same > >>>>>>>>>>>>> question > >>>>>>>>>>>>> on the Apache httpd user's list for a better response. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Tomcat configuration settings : > >>>>>>>>>>>>> > >>>>>>>>>>>>> AJP Connector : (http://tomcat.apache.org/tomc > >>>>>>>>>>>>> at-8.5-doc/config/ajp.html# > >>>>>>>>>>>>> Standard_Implementations) > >>>>>>>>>>>>> > >>>>>>>>>>>>> connectionTimeout > >>>>>>>>>>>>> > >>>>>>>>>>>>> The number of milliseconds this Connector will wait, after > >>>>>>>>>>>>> accepting a > >>>>>>>>>>>>> connection, for the request URI line to be presented. The > >>>>>>>>>>>>> default > >>>>>>>>>>>>> value > >>>>>>>>>>>>> for > >>>>>>>>>>>>> AJP protocol connectors is -1 (i.e. infinite). > >>>>>>>>>>>>> > >>>>>>>>>>>>> (You could for example try to set this to 3000 > (milliseconds) or > >>>>>>>>>>>>> even > >>>>>>>>>>>>> lower. That should be more than enough for any legitimate > client > >>>>>>>>>>>>> so > >>>>>>>>>>>>> send > >>>>>>>>>>>>> the HTTP request line. Note however that by doing this at > the > >>>>>>>>>>>>> Tomcat > >>>>>>>>>>>>> level, you will probably move the problem to the Apache > >>>>>>>>>>>>> > >>>>>>>>>>>>> httpd/mod_jk > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>> level. But at least it might confirm that this is the problem > >>>>>>> > >>>>>>>> that you > >>>>>>>>>>>>> are > >>>>>>>>>>>>> seeing. The mod_jk logfile at the httpd level may give you > some > >>>>>>>>>>>>> hints > >>>>>>>>>>>>> there.) > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> HTTP Connector : (http://tomcat.apache.org/tomc > >>>>>>>>>>>>> at-8.5-doc/config/http.html#Standard_Implementation) > >>>>>>>>>>>>> > >>>>>>>>>>>>> connectionTimeout > >>>>>>>>>>>>> > >>>>>>>>>>>>> The number of milliseconds this Connector will wait, after > >>>>>>>>>>>>> accepting a > >>>>>>>>>>>>> connection, for the request URI line to be presented. Use a > >>>>>>>>>>>>> value > >>>>>>>>>>>>> of -1 > >>>>>>>>>>>>> to > >>>>>>>>>>>>> indicate no (i.e. infinite) timeout. The default value is > 60000 > >>>>>>>>>>>>> (i.e. > >>>>>>>>>>>>> 60 > >>>>>>>>>>>>> seconds) but note that the standard server.xml that ships > with > >>>>>>>>>>>>> Tomcat > >>>>>>>>>>>>> sets > >>>>>>>>>>>>> this to 20000 (i.e. 20 seconds). Unless disableUploadTimeout > is > >>>>>>>>>>>>> set to > >>>>>>>>>>>>> false, this timeout will also be used when reading the > request > >>>>>>>>>>>>> body (if > >>>>>>>>>>>>> any). > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> On 26.11.2016 09:57, Jaaz Portal wrote: > >>>>>>>>>>>>> > >>>>>>>>>>>>> hi, > >>>>>>>>>>>>> > >>>>>>>>>>>>> sorry, its mod_jk no jk2, my typo. All at latest versions. We > >>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> tried > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>> with > >>>>>>> > >>>>>>>> mod proxy too. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> There is no flood of the server. Nobody is flooding us, they > >>>>>>>>>>>>>> use > >>>>>>>>>>>>>> some > >>>>>>>>>>>>>> specific connections after which pool of apache workers is > >>>>>>>>>>>>>> exhausted > >>>>>>>>>>>>>> and > >>>>>>>>>>>>>> blocked > >>>>>>>>>>>>>> and we need to restart tomcat server. > >>>>>>>>>>>>>> It is some kind of exploit but do not know how to log it to > >>>>>>>>>>>>>> obtain > >>>>>>>>>>>>>> details. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> i had put a limit on connections per client with hope that > this > >>>>>>>>>>>>>> will > >>>>>>>>>>>>>> help > >>>>>>>>>>>>>> but once again, it is not a flood. > >>>>>>>>>>>>>> They open several connections that are not dropped by apache > >>>>>>>>>>>>>> when they > >>>>>>>>>>>>>> disconnect. This way whole pool is quickly exhausted and the > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> server > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>> broken. > >>>>>>> > >>>>>>>> > >>>>>>>>>>>>>> i would like to help you to figure details of this attack > but > >>>>>>>>>>>>>> this is > >>>>>>>>>>>>>> production server so it is impossible to much debugging > options > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> best, > >>>>>>>>>>>>>> artur > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> 2016-11-25 23:44 GMT+01:00 Niranjan Babu Bommu < > >>>>>>>>>>>>>> niranjan.bo...@gmail.com > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> : > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> you can find who is flooding site in apache access.log and > >>>>>>>>>>>>>>> block > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> them > >>>>>>>>>>>>>> in > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> firewall. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> ex to find the IP: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> cat /var/log/apache2/access.log |cut -d' ' -f1 |sort |uniq > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> -c|sort > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>> -gr > >>>>>>> > >>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> On Fri, Nov 25, 2016 at 8:42 AM, Jaaz Portal > >>>>>>>>>>>>>>> <jaazpor...@gmail.com> > >>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> hi, > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> we are from some weeks struggling with some Polish hackers > >>>>>>>>>>>>>>> that > >>>>>>>>>>>>>>> are > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> bringing our server down. After updating apache to latest > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> version > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>> > >>>>>>> (2.4.23) > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> and tomcat (8.0.38) available for debian systems we still > >>>>>>>>>>>>>>> cannot > >>>>>>>>>>>>>>> secure > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> our > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> server. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Today it has stopped to respond again and we needed to > restart > >>>>>>>>>>>>>>>> tomcat > >>>>>>>>>>>>>>>> process to get it back alive. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> There is no too much clues in the logs. The apache > error.log > >>>>>>>>>>>>>>>> gives > >>>>>>>>>>>>>>>> just > >>>>>>>>>>>>>>>> this line: > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> [Fri Nov 25 13:08:00.647835 2016] [mpm_event:error] [pid > >>>>>>>>>>>>>>>> 13385:tid > >>>>>>>>>>>>>>>> 1397934896385 > >>>>>>>>>>>>>>>> 92] AH00484: server reached MaxRequestWorkers setting, > >>>>>>>>>>>>>>>> consider > >>>>>>>>>>>>>>>> raising > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> MaxR > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> equestWorkers setting > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> seems that somehow tomcat, mod-jk2 or even apache is > >>>>>>>>>>>>>>>> vulnerable to > >>>>>>>>>>>>>>>> some > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> new > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> exploit, as we certainly does not have such traffic that > >>>>>>>>>>>>>>> would > >>>>>>>>>>>>>>> block > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> our > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> server otherwise > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> for now we have increased MaxRequestWorkers and we have > >>>>>>>>>>>>>>>> limited > >>>>>>>>>>>>>>>> number > >>>>>>>>>>>>>>>> of > >>>>>>>>>>>>>>>> connections from one client to 5 by mod_bw and limited > number > >>>>>>>>>>>>>>>> of > >>>>>>>>>>>>>>>> simultaneous connections from one ip by iptables but does > not > >>>>>>>>>>>>>>>> know > >>>>>>>>>>>>>>>> if > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> this > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> will help > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> best regards, > >>>>>>>>>>>>>>>> artur > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> -- > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> *Thanks* > >>>>>>>>>>>>>>> *Niranjan* > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> This sounds like a variant of the slowloris attack. > >>>>>> > >>>>>> This type of attack doesn't take a large number of clients or > consume a > >>>>>> large amount of bandwidth. > >>>>>> > >>>>>> Basically, the maximum number of connections are made to the server, > >>>>>> and > >>>>>> just enough data is sent to each connection in order to not trigger > the > >>>>>> timeout. André has explained this in more detail earlier in the > thread. > >>>>>> Search for "slowloris attack" for more information. > >>>>>> > >>>>>> There are several ways of mitigating this type of attack. > >>>>>> > >>>>>> As André has mentioned, placing a dedicated device in front of your > >>>>>> systems is often the best way. Lots of benefits (platform neutral, > no > >>>>>> stress on your current servers), and some issues (cost, placement / > >>>>>> access may be an issue with hosted solutions). > >>>>>> > >>>>>> However, there are Apache HTTPD modules that can help mitigate these > >>>>>> types of attacks. Some of them are: > >>>>>> > >>>>>> mod_reqtimeout (should be included by default in your Apache HTRPD > 2.4) > >>>>>> mod_qos (quality of service module) > >>>>>> mod_security (application firewall with lots of security rules) > >>>>>> > >>>>>> Do a quick search on "slowloris attack apache httpd 2.4" to get some > >>>>>> ideas. > >>>>>> > >>>>>> All of them will probably place additional load on your Apache HTTPD > >>>>>> server, so make sure that the platform is robust enough to manage > the > >>>>>> additional load. > >>>>>> > >>>>>> There is also a beta version of the mod_security module written as a > >>>>>> servlet filter. It should be possible to build this and configure > the > >>>>>> filter in Tomcat's default web.xml ($CATALINA_BASE/conf/web.xml). > I've > >>>>>> not tried this. Also, the code base hasn't seen any activity for 3 > >>>>>> years. > >>>>>> > >>>>>> Do a quick search on "modsecurity servlet filter" to find out more > >>>>>> about > >>>>>> the servlet filter version of mod_security. > >>>>>> > >>>>>> In short, there appear to be some ways to mitigate the attack. > >>>>>> > >>>>>> . . . just my two cents > >>>>>> /mde/ > > > >
