hi, sorry, there was two open connection on port 80 from 194.135.88.32 that is somwhere on epix.net.pl an association of internet traffick exchange (some pirate hub)
best, artur 2016-11-30 17:52 GMT+01:00 Jaaz Portal <jaazpor...@gmail.com>: > hi, > i looked at the logs but there are no strange things, > traffic as usual, no errors despite this one: > [Wed Nov 30 17:10:13.375912 2016] [mpm_event:error] [pid 12870:tid > 139906329666752] AH00484: server reached MaxRequestWorkers setting, > consider raising the MaxRequestWorkers setting > > any idea what to do next? > > best, > artur > > 2016-11-30 17:36 GMT+01:00 Jaaz Portal <jaazpor...@gmail.com>: > >> hi, >> they has tried again with success despite setting connection_timeout and >> limiting number of clients by mod_bw >> the tomcat has frozen again. >> >> netstat does not showed any connections on port 80 but plenty of >> connections from apache to localhost:8009 >> so it was not an attack that you has described (no slowlaris) >> >> im looking into debug files of mod_jk and forensic for some hints. If you >> want i can share them (they have 4mb compressed) >> >> best wishes >> artur >> >> 2016-11-29 11:01 GMT+01:00 André Warnier (tomcat) <a...@ice-sa.com>: >> >>> On 28.11.2016 22:04, Jaaz Portal wrote: >>> >>>> hi Andre, >>>> you are wrong. This vulnerability is not only causing memory leaks, it >>>> makes also apache workers to hang >>>> >>> >>> Maybe for the last time here : >>> >>> - what do you call "apache workers" ? >>> >>> , making it easy to exhaust the pool. >>> >>> - what do you call "the pool" ? >>> >>> what i have in my log files. But it is true also that such exhaustion can >>>> be made by other forms of dos attacks described in this thread. >>>> >>>> regarding you suggestion on our application, it does not dos bind server >>>> nether does not scan for various vulnerabilities in apache, what i have >>>> also in the logs >>>> >>> >>> For your information : I run about 25 Internet-facing Apache webservers >>> (some with a back-end tomcat, some not). >>> On every single one of those webservers, there are *hundreds* of such >>> "scans" every day, as shown by the Apache access logs. That is just a fact >>> of life on the Internet. >>> They are annoying, but most of them are harmless (from an "attack" point >>> of view), because they are scanning for things that we do not run >>> (phpmyadmin, xmlrpc, vti_bin, etc., etc., the list is almost endless), and >>> thus are responded to by Apache as "404 Not found". >>> What is annoying with those scans, is >>> a) that they fill the logfile, and thus make it more difficult to find >>> really significant things >>> b) that each of those requires some bandwidth and system resource, if >>> only to return a "404 Not found" (or a "401 Unauthorised"), and that we pay >>> for that bandwidth and resources. >>> >>> If I could find a way to charge 0.1 cent per access to my servers, from >>> the people who wrote or run the programs who are doing this, I could retire >>> in luxury. >>> >>> But they are not a real problem, because they are caught as "invalid" by >>> Apache, and rejected quickly, so they cannot do anything really nasty >>> (except if they were sending several thousand such requests per second to >>> one of my servers for a long time). >>> >>> The ones that are worrying, are the ones >>> - a) which do /not/ end up as a "404 Not found", because they have found >>> an application which responds, and they are not coming from our legitimate >>> customers >>> - b) /the ones which we do not see/, because they either do not send a >>> valid HTTP request, or they have found a way to trigger one of our >>> applications, in such a way that the application misbehaves and, perhaps, >>> even if they do not crash our servers, they may provide the attacker with >>> some entry point to do other things which we do not know and do not control >>> >>> What I am trying to say here, is /do not jump to premature conclusions/. >>> Such "scans" as you mention, happen to everyone, all the time, from >>> ever-changing IP addresses located all over the world. Some of those >>> "scans" may come from the infected PC of your grandmother, and she does not >>> even know about it. >>> >>> There is no guarantee, and no indication or proof so far, that /these/ >>> scans are even related to "the other thing" which happens on your >>> webserver, which looked much more focused. >>> >>> So do not just bundle them together as being the same thing, until you >>> have some real data that shows for example that these different things all >>> come from the same IP addresses. >>> >>> And one more thing, also finally until you come back with some real data >>> : I am not saying that your application "scans your server". What I am >>> saying is that, maybe, by chance or by design, the attackers have found a >>> URL which goes to your application, and which causes your application to >>> keep tomcat and/or Apache busy for a long time. >>> And that maybe /that/ is the problem you should be looking for, and not >>> some hypothetical bug in Apache httpd or tomcat. >>> >>> >>> >>>> kindly regards, >>>> artur >>>> >>>> 2016-11-28 21:33 GMT+01:00 André Warnier (tomcat) <a...@ice-sa.com>: >>>> >>>> On 28.11.2016 20:34, Jaaz Portal wrote: >>>>> >>>>> hi mark, >>>>>> yes, i understand now what slowloris attack is. >>>>>> maybe it was this maybe *this one based on * * mod_proxy denial of >>>>>> service * >>>>>> CVE-2014-0117 <http://cve.mitre.org/cgi-bin/ >>>>>> cvename.cgi?name=CVE-2014-0117> >>>>>> >>>>>> >>>>> You keep on saying this, but the description of that vulnerability of >>>>> *Apache httpd*, and the symptoms which you described, *do not match*. >>>>> You described the problem as ocurring in Apache tomcat, which in your >>>>> case >>>>> is sitting as a back-end, *behind* Apache httpd. And restarting tomcat >>>>> cured the problem. >>>>> >>>>> The CVE above applies to Apache httpd, and describes how an attacker >>>>> could >>>>> attack Apache httpd and cause *its children* processes to crash (the >>>>> children processes of Apache httpd), by leading them to consume a lot >>>>> of >>>>> memory and crash with an out-of-memory error. >>>>> Granted, the problem occurred in the mod_proxy module of Apache httpd; >>>>> but >>>>> it was httpd which crashed, not tomcat. >>>>> And tomcat processes are not "Apache httpd children processes" in any >>>>> understanding of the term. >>>>> >>>>> As far as I remember, you never mentioned Apache httpd crashing. You >>>>> mentioned "the pool" getting full or satured or something like that, >>>>> without ever describing properly what you meant by "the pool". >>>>> >>>>> As far as I am concerned, according to all the relatively unspecific >>>>> information which you have previously provided : >>>>> 1) the attack - if that is what it is/was - is definitely NOT related >>>>> to >>>>> the CVE which you have repeatedly mentioned >>>>> 2) it is apparently not a "classical" DoS or "slowloris DoS" directed >>>>> at >>>>> your front-end Apache. Instead, it seems that the requests are properly >>>>> received by Apache, properly decoded by Apache, and then whatever >>>>> Apache >>>>> proxy module you are using (mod_proxy_http, mod_proxy_ajp or mod_jk) is >>>>> properly forwarding these requests to a back-end tomcat; and it is at >>>>> the >>>>> level of that back-end tomcat that the requests never seem to end, and >>>>> in >>>>> the end paralyse your tomcat server (and later on maybe your Apache >>>>> httpd >>>>> server too, because it is also waiting for tomcat to respond). >>>>> >>>>> So your very way of describing the problem, in terms of "first we used >>>>> this proxy module, and then they exploited the vulnerability so and so; >>>>> then we changed the proxy module, and they exploited that too; etc.." >>>>> seems to not have anything to do with the problem per se, and (I >>>>> believe) >>>>> confuses everyone, including yourself. >>>>> >>>>> It is not that "they" exploited different vulnerabilities of various >>>>> httpd >>>>> proxy modules, one after the other. Each of these proxy modules was >>>>> doing >>>>> its job properly, and forwarding valid HTTP requests properly to >>>>> tomcat. >>>>> When you changed from one proxy module to another, you did not really >>>>> change anything in that respect, because any proxy module would do the >>>>> same. >>>>> >>>>> But in all cases, what did not change, was the tomcat behind the >>>>> front-end, and the application running on that tomcat. So the presumed >>>>> attackers did not have to change anything, they just kept on sending >>>>> the >>>>> same requests, because they were really targeting your back-end tomcat >>>>> or >>>>> the tomcat application in it, no matter /how/ you were forwarding >>>>> requests >>>>> from Apache httpd to tomcat. >>>>> >>>>> So either it is tomcat itself, which has a problem with some request >>>>> URLs >>>>> which do not bother Apache httpd (possible, but statistically >>>>> unlikely), or >>>>> it is the application which runs in tomcat that has such a problem >>>>> (statistically, much more likely). >>>>> >>>>> we do not know yet >>>>> >>>>>> >>>>>> we have setup more logging and are waiting for them to attack once >>>>>> again >>>>>> >>>>>> >>>>> Yes, that is the right thing to do. Before deciding what the problem >>>>> may >>>>> be, and what you can do about it, the first thing you need is *data*. >>>>> You >>>>> need to know >>>>> - which request URL(s?) cause that problem >>>>> - which IPs these requests come from (always the same ? multiple IPs >>>>> that >>>>> change all the time ? how many ? can these IPs be valid/expected >>>>> clients or >>>>> not ? do these IPs look like some "coordinated group" ?) >>>>> - how many such requests there may be during some period of time (10, >>>>> 100, >>>>> 1000, more ?) >>>>> - if these URLs result in passing the request to tomcat >>>>> - what tomcat application (if any) they are directed to >>>>> - if so, when that application receives such a request, what is it >>>>> supposed to do ? does it do it properly ? how long does it need, to >>>>> respond >>>>> to such a request ? >>>>> >>>>> You also need to ask yourself a question : is the application which you >>>>> run inside tomcat something that you designed yourself (and which >>>>> hackers >>>>> are unlikely to know well-enough to find such a URL which paralyses >>>>> your >>>>> server) ? or is it some well-known third-party java application which >>>>> you >>>>> are running (and for which would-be attackers would be much more >>>>> likely to >>>>> know exactly such a bug) ? >>>>> >>>>> >>>>> anyway, thank you for all informations, it was very useful and >>>>>> educational >>>>>> reading for all of us >>>>>> >>>>>> best wishes, >>>>>> artur >>>>>> >>>>>> 2016-11-28 19:46 GMT+01:00 Mark Eggers <its_toas...@yahoo.com.invalid >>>>>> >: >>>>>> >>>>>> Jaaz, >>>>>> >>>>>>> >>>>>>> On 11/27/2016 2:46 PM, André Warnier (tomcat) wrote: >>>>>>> >>>>>>> On 27.11.2016 19:03, Jaaz Portal wrote: >>>>>>>> >>>>>>>> 2016-11-27 18:30 GMT+01:00 André Warnier (tomcat) <a...@ice-sa.com>: >>>>>>>>> >>>>>>>>> On 27.11.2016 14:26, Jaaz Portal wrote: >>>>>>>>> >>>>>>>>>> >>>>>>>>>> hi, >>>>>>>>>> >>>>>>>>>>> everything i know so far is just this single log line that >>>>>>>>>>> appeared >>>>>>>>>>> in >>>>>>>>>>> apache error.log >>>>>>>>>>> >>>>>>>>>>> [Fri Nov 25 13:08:00.647835 2016] [mpm_event:error] [pid >>>>>>>>>>> 13385:tid >>>>>>>>>>> 1397934896385 >>>>>>>>>>> 92] AH00484: server reached MaxRequestWorkers setting, consider >>>>>>>>>>> >>>>>>>>>>> raising >>>>>>>>>> >>>>>>>>> >>>>>>> the >>>>>>>> >>>>>>>>> MaxR >>>>>>>>>>> equestWorkers setting >>>>>>>>>>> >>>>>>>>>>> there was nothing else, just this strange line >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> This is not a "strange" line. It is telling you something. >>>>>>>>>> One problem is that you seem convinced in advance, without serious >>>>>>>>>> proof, >>>>>>>>>> that there is a "bug" or a vulnerability in httpd or tomcat. >>>>>>>>>> Read the explanation of the httpd parameter, here : >>>>>>>>>> http://httpd.apache.org/docs/2.4/mod/mpm_common.html#maxrequ >>>>>>>>>> estworkers >>>>>>>>>> and try to understand it. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> I understand it very well. We are serving no more that 500 >>>>>>>>> clients per >>>>>>>>> day >>>>>>>>> so there was no other option that some kind of attack. >>>>>>>>> >>>>>>>>> >>>>>>>>> About the "bug" or "vulnerability" : a webserver is supposed to >>>>>>>>> serve >>>>>>>>> HTTP >>>>>>>>> >>>>>>>>> requests from clients. That is what you expect of it. It has no >>>>>>>>>> choice but >>>>>>>>>> to accept any client connection and request, up to the maximum it >>>>>>>>>> can >>>>>>>>>> handle considering its configuration and the capacity of the >>>>>>>>>> system on >>>>>>>>>> which it runs. That is not a bug, it is a feature. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> We have some weeks ago come under attack from some Polish group. >>>>>>>>>> It >>>>>>>>>> >>>>>>>>> was >>>>>>>>> first bind that was DoS'ed, we was running on stable Debian so i >>>>>>>>> updated >>>>>>>>> bind to latest version. It did not helped. They has dos'ed it so we >>>>>>>>> switched to other dns provider. That has helped. >>>>>>>>> >>>>>>>>> Then they exploited some well know vulnerability in mod_proxy. We >>>>>>>>> have >>>>>>>>> updated apache to the latest but again they has exploited it, so we >>>>>>>>> have >>>>>>>>> switched to mod_jk. And then guess what. They exploited it too so i >>>>>>>>> decided >>>>>>>>> to write to this list looking for help before trying jetty. >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> The normal Apache httpd access log, will log a request only when >>>>>>>>>> it is >>>>>>>>>> finished. If the request never finishes, it will not get logged. >>>>>>>>>> That may be why you do not see these requests in the log. >>>>>>>>>> But have a look at this Apache httpd module : >>>>>>>>>> http://httpd.apache.org/docs/2.4/mod/mod_log_forensic.html >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> ok, thanks, will take care >>>>>>>>> >>>>>>>>> Note : that is also why I was telling you to enable the mod_jk log, >>>>>>>>> and to >>>>>>>>> >>>>>>>>> examine it. >>>>>>>>>> Because mod_jk will also log information before the request >>>>>>>>>> produces a >>>>>>>>>> response. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> and server hanged. >>>>>>>>>> >>>>>>>>>> Again, /what/ is "hanged" ? Apache httpd, or tomcat ? >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> Apache was accepting connection but not processed it. After >>>>>>>>> restart of >>>>>>>>> tomcat server it worked again. >>>>>>>>> >>>>>>>>> >>>>>>>>> Also in >>>>>>>>> >>>>>>>>>> >>>>>>>>>> access logs there are no clues that it was under any heavy load. >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> after around hour after discovering that our server hanged-out we >>>>>>>>>>> have >>>>>>>>>>> restarted tomcat server >>>>>>>>>>> and it worked again >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Yes, because that will close all connections between Apache >>>>>>>>>> httpd and >>>>>>>>>> tomcat, and abort all requests that are in the process of being >>>>>>>>>> processed >>>>>>>>>> by tomcat. So mod_jk will get an error from tomcat, and will >>>>>>>>>> report an >>>>>>>>>> error to httpd, and httpd will communicate that error to the >>>>>>>>>> clients, >>>>>>>>>> and >>>>>>>>>> close their connection. >>>>>>>>>> It still does not tell you what the problem was. >>>>>>>>>> The only thing that it suggests, is that the "bad" requests >>>>>>>>>> actually >>>>>>>>>> make >>>>>>>>>> it all the way to tomcat. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> correct >>>>>>>>> >>>>>>>>> i will enable logs that you has pointed out and we will look what i >>>>>>>>> will >>>>>>>>> catch >>>>>>>>> however i think we have only one chance, as if the solution we has >>>>>>>>> found >>>>>>>>> out (connection_timeout + mod_bn) >>>>>>>>> will work they will stop exploiting it >>>>>>>>> >>>>>>>>> thank you very much for all the help and explanations >>>>>>>>> i will report to the list new facts once they will attack us again >>>>>>>>> >>>>>>>>> best regards, >>>>>>>>> artur >>>>>>>>> >>>>>>>>> >>>>>>>>> Ok, but also read this e.g. : >>>>>>>> https://www.corero.com/blog/695-going-after-the-people- >>>>>>>> >>>>>>>> behind-ddos-attacks.html >>>>>>> >>>>>>> >>>>>>>> Attempts to bring down a site by DoS attacks is a crime, in most >>>>>>>> places.. >>>>>>>> >>>>>>>> You can report it, while at the same time trying to defend yourself >>>>>>>> against it. >>>>>>>> >>>>>>>> It is also relatively easy, and quite inexpensive in terms of system >>>>>>>> resources, to run a small shell script which takes a list every few >>>>>>>> seconds of the connections to the port of your webserver, and which >>>>>>>> IPs >>>>>>>> they are coming *from*. >>>>>>>> E.g. >>>>>>>> First try the netstat command, to see what it lists, like : >>>>>>>> # netstat -n --tcp | more >>>>>>>> >>>>>>>> Then you will want to filter this a bit, to only consider >>>>>>>> established >>>>>>>> connections to your webserver, for example : >>>>>>>> # netstat -n --tcp | grep ":80" | grep "ESTABLISHED" >>>>>>>> >>>>>>>> Then you will want to send this to a logfile, regularly, like this : >>>>>>>> >>>>>>>> # date >> some_file.log >>>>>>>> # netstat -n --tcp | grep ":80" | grep "ESTABLISHED" >> >>>>>>>> some_file.log >>>>>>>> (repeat every 3 seconds) >>>>>>>> >>>>>>>> This will not generate GB of logfiles, and it will tell you, when >>>>>>>> the >>>>>>>> problem happens, how many connections there are exactly to your >>>>>>>> webserver, and where they are coming from. >>>>>>>> Then later you can further analyse this information.. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>> i think that setting connection-timeout and limiting the number of >>>>>>>>> >>>>>>>>>> clients >>>>>>>>>>> by mod_bd i will >>>>>>>>>>> have effect that next time somebody will try this exploit it will >>>>>>>>>>> >>>>>>>>>>> block >>>>>>>>>> >>>>>>>>> >>>>>>> his >>>>>>>> >>>>>>>>> access to the site >>>>>>>>>>> for minute or two, pretty good holistic solution i would say >>>>>>>>>>> >>>>>>>>>>> still, it seems that there is serious vulnerability somewhere in >>>>>>>>>>> apache, >>>>>>>>>>> mod_jk or tomcat >>>>>>>>>>> i would like to help find it out but need some hints which debug >>>>>>>>>>> options >>>>>>>>>>> enable to catch the bad guys >>>>>>>>>>> when they will try next time >>>>>>>>>>> >>>>>>>>>>> best regards, >>>>>>>>>>> artur >>>>>>>>>>> >>>>>>>>>>> 2016-11-27 13:58 GMT+01:00 André Warnier (tomcat) <a...@ice-sa.com >>>>>>>>>>> >: >>>>>>>>>>> >>>>>>>>>>> On 27.11.2016 13:23, Jaaz Portal wrote: >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> hi Andre, >>>>>>>>>>>> >>>>>>>>>>>> thank you very much this was very educative but in my case it is >>>>>>>>>>>>> little >>>>>>>>>>>>> bit >>>>>>>>>>>>> different. >>>>>>>>>>>>> The server is no flooded, there is maybe dozen of very >>>>>>>>>>>>> sophisticated >>>>>>>>>>>>> connections that somehow hangs apache workers threads >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Can you be a bit more specific ? >>>>>>>>>>>>> >>>>>>>>>>>> When you say "apache workers threads", do you mean threads in >>>>>>>>>>>> Apache >>>>>>>>>>>> httpd, or threads in Apache Tomcat ? (both are Apache >>>>>>>>>>>> webservers, >>>>>>>>>>>> so it >>>>>>>>>>>> is >>>>>>>>>>>> difficult to tell what you are talking about, unless you are >>>>>>>>>>>> very >>>>>>>>>>>> precise). >>>>>>>>>>>> >>>>>>>>>>>> Let me give you some additional explanations, and maybe you can >>>>>>>>>>>> >>>>>>>>>>>> figure >>>>>>>>>>> >>>>>>>>>> >>>>>>> out >>>>>>>> >>>>>>>>> exactly where it "hangs". >>>>>>>>>>>> >>>>>>>>>>>> From the Apache httpd front-end point of view, mod_jk (the >>>>>>>>>>>> connector to >>>>>>>>>>>> Apache Tomcat) is basically one among other "response >>>>>>>>>>>> generators". >>>>>>>>>>>> Apache >>>>>>>>>>>> httpd does not "know" that behind mod_jk, there is one or more >>>>>>>>>>>> Tomcat >>>>>>>>>>>> servers. Apache httpd receives the original client request, and >>>>>>>>>>>> depending >>>>>>>>>>>> on the URL of the request, it will pass it to mod_jk or to >>>>>>>>>>>> another >>>>>>>>>>>> response >>>>>>>>>>>> generator, to generate the response to the request. >>>>>>>>>>>> That mod_jk in the background is using a Tomcat server to >>>>>>>>>>>> actually >>>>>>>>>>>> generate the response, is none of the business of Apache httpd, >>>>>>>>>>>> and >>>>>>>>>>>> >>>>>>>>>>>> it >>>>>>>>>>> >>>>>>>>>> >>>>>>> does >>>>>>>> >>>>>>>>> not care. All it cares about, is to actually receive the response >>>>>>>>>>>> >>>>>>>>>>>> from >>>>>>>>>>> >>>>>>>>>> >>>>>>> mod_jk, and pass it back to the client. >>>>>>>> >>>>>>>>> >>>>>>>>>>>> If httpd passes a request to mod_jk, it is because you have >>>>>>>>>>>> specified in >>>>>>>>>>>> the Apache configuration, the type of URL that it should pass to >>>>>>>>>>>> mod_jk.. >>>>>>>>>>>> That happens via your "JkMount (URL pattern)" directives in >>>>>>>>>>>> Apache >>>>>>>>>>>> httpd. >>>>>>>>>>>> >>>>>>>>>>>> Of course Apache httpd will not pass a request to mod_jk, >>>>>>>>>>>> before it >>>>>>>>>>>> has >>>>>>>>>>>> received at least the URL of the request, and analysed it to >>>>>>>>>>>> >>>>>>>>>>>> determine >>>>>>>>>>> >>>>>>>>>> >>>>>>> *if* >>>>>>>> >>>>>>>>> it should pass it to mod_jk (*). >>>>>>>>>>>> >>>>>>>>>>>> If the mod_jk logging is enabled, you can see in it, exactly >>>>>>>>>>>> *which* >>>>>>>>>>>> requests are passed to mod_jk and to Tomcat. >>>>>>>>>>>> Do you know *which* requests, from which clients, cause this >>>>>>>>>>>> "thread >>>>>>>>>>>> hanging" symptom ? >>>>>>>>>>>> Once you would know this, maybe you can design a strategy to >>>>>>>>>>>> block >>>>>>>>>>>> specifically these requests. >>>>>>>>>>>> >>>>>>>>>>>> and the effect is permanent. Quickly the pool is exhausted >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> which pool exactly ? >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> and the only >>>>>>>>>>>> >>>>>>>>>>>> solution that works is to restart tomcat. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> I think it is a bug similar to this one from mod_proxy: >>>>>>>>>>>>> https://tools.cisco.com/security/center/viewAlert.x?alertId= >>>>>>>>>>>>> 34971 >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Maybe, maybe not. As long as we do not know what the requests >>>>>>>>>>>>> are, >>>>>>>>>>>>> that >>>>>>>>>>>>> >>>>>>>>>>>>> block things, we do not know this. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> I think also that your solution with setting connectionTimeout >>>>>>>>>>>> will >>>>>>>>>>>> solve >>>>>>>>>>>> >>>>>>>>>>>> the problem, at least partially. THANK YOU. >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Same thing, we do not know this yet. It was only giving this >>>>>>>>>>>>> >>>>>>>>>>>> explanation, >>>>>>>>>>>> to help you think about where the problem may be. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> I would like to help you further investigate this issue, as our >>>>>>>>>>>> >>>>>>>>>>>> server >>>>>>>>>>> >>>>>>>>>> >>>>>>> comes under such attack once or twice in a week. >>>>>>>> >>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Other than giving you hints, there is not much I or anyone else >>>>>>>>>>>>> can do >>>>>>>>>>>>> >>>>>>>>>>>>> to >>>>>>>>>>>> help. You are the one with control of your servers and >>>>>>>>>>>> logfiles, so >>>>>>>>>>>> you >>>>>>>>>>>> have to investigate and provide more precise information. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> (*) actually, to be precise, Apache httpd passes *all* requests >>>>>>>>>>>> to >>>>>>>>>>>> mod_jk, >>>>>>>>>>>> to ask it "if it wants that request". mod_jk then accepts or >>>>>>>>>>>> >>>>>>>>>>>> declines, >>>>>>>>>>> >>>>>>>>>> >>>>>>> depending on the JkMount instructions. If mod_jk declines, then >>>>>>>> >>>>>>>>> >>>>>>>>>>>> Apache >>>>>>>>>>> >>>>>>>>>> >>>>>>> httpd will ask the next response generator if it wants this request, >>>>>>>> >>>>>>>>> etc... >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> best regards, >>>>>>>>>>>> >>>>>>>>>>>> artur >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> 2016-11-27 12:46 GMT+01:00 André Warnier (tomcat) < >>>>>>>>>>>>> a...@ice-sa.com>: >>>>>>>>>>>>> >>>>>>>>>>>>> Hi. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Have a look that the indicated parameters in the two pages >>>>>>>>>>>>>> below.. >>>>>>>>>>>>>> >>>>>>>>>>>>>> You may be the target of such a variant of DDoS attack : many >>>>>>>>>>>>>> clients >>>>>>>>>>>>>> open >>>>>>>>>>>>>> a TCP connection to your server (front-end), but then never >>>>>>>>>>>>>> sends >>>>>>>>>>>>>> a >>>>>>>>>>>>>> HTTP >>>>>>>>>>>>>> request on that connection. In the meantime, the server >>>>>>>>>>>>>> accepts >>>>>>>>>>>>>> >>>>>>>>>>>>>> the >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>> TCP >>>>>>>> >>>>>>>>> connection, and passes it on to a "child" process or thread for >>>>>>>>>>>>>> processing. The child then waits for the HTTP request line to >>>>>>>>>>>>>> arrive >>>>>>>>>>>>>> on >>>>>>>>>>>>>> the connection (during a certain time), but it never arrives. >>>>>>>>>>>>>> After a >>>>>>>>>>>>>> while, this triggers a timeout (see below), but the standard >>>>>>>>>>>>>> value of >>>>>>>>>>>>>> that >>>>>>>>>>>>>> timeout may be such that in the meantime, a lot of other >>>>>>>>>>>>>> >>>>>>>>>>>>>> connections >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>> have >>>>>>>> >>>>>>>>> been established by other such nefarious clients, so a lot of >>>>>>>>>>>>>> resources >>>>>>>>>>>>>> of >>>>>>>>>>>>>> the webserver are tied up, waiting for something that will >>>>>>>>>>>>>> never >>>>>>>>>>>>>> come.. >>>>>>>>>>>>>> Since there is never any real request sent on the connection, >>>>>>>>>>>>>> you >>>>>>>>>>>>>> would >>>>>>>>>>>>>> (probably) not see this in the logs either. >>>>>>>>>>>>>> >>>>>>>>>>>>>> The above is the basic mechanism of such an attack. There >>>>>>>>>>>>>> may be >>>>>>>>>>>>>> variations, such as the client not "not sending" a request >>>>>>>>>>>>>> line, >>>>>>>>>>>>>> >>>>>>>>>>>>>> but >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>> sending it extremely slowly, thus achieving perhaps similar kinds >>>>>>>> >>>>>>>>> >>>>>>>>>>>>>> of >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>> effects. >>>>>>>> >>>>>>>>> >>>>>>>>>>>>>> As someone pointed out, it is quite difficult to do something >>>>>>>>>>>>>> about >>>>>>>>>>>>>> this >>>>>>>>>>>>>> at the level of the webserver itself, because by the time you >>>>>>>>>>>>>> would do >>>>>>>>>>>>>> something about it, the resources have already been consumed >>>>>>>>>>>>>> and >>>>>>>>>>>>>> your >>>>>>>>>>>>>> server is probably already overloaded. >>>>>>>>>>>>>> There are specialised front-end devices and software >>>>>>>>>>>>>> available, to >>>>>>>>>>>>>> detect >>>>>>>>>>>>>> and protect against this kind of attack. >>>>>>>>>>>>>> >>>>>>>>>>>>>> You may want to have a look at the following parameters, but >>>>>>>>>>>>>> make >>>>>>>>>>>>>> sure >>>>>>>>>>>>>> to >>>>>>>>>>>>>> read the caveats (side-effects, interlocking timeouts etc.), >>>>>>>>>>>>>> otherwise >>>>>>>>>>>>>> you >>>>>>>>>>>>>> may do more harm than good. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Another thing : the settings below are for Apache Tomcat, >>>>>>>>>>>>>> which >>>>>>>>>>>>>> in your >>>>>>>>>>>>>> case is the back-end. It would of course be much better to >>>>>>>>>>>>>> detect >>>>>>>>>>>>>> and >>>>>>>>>>>>>> eliminate this at the front-end, or even before. I had a >>>>>>>>>>>>>> look at >>>>>>>>>>>>>> the >>>>>>>>>>>>>> Apache httpd documentation, and could not find a corresponding >>>>>>>>>>>>>> parameter. >>>>>>>>>>>>>> But I am sure that it must exist. You may want to post this >>>>>>>>>>>>>> same >>>>>>>>>>>>>> question >>>>>>>>>>>>>> on the Apache httpd user's list for a better response. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Tomcat configuration settings : >>>>>>>>>>>>>> >>>>>>>>>>>>>> AJP Connector : (http://tomcat.apache.org/tomc >>>>>>>>>>>>>> at-8.5-doc/config/ajp.html# >>>>>>>>>>>>>> Standard_Implementations) >>>>>>>>>>>>>> >>>>>>>>>>>>>> connectionTimeout >>>>>>>>>>>>>> >>>>>>>>>>>>>> The number of milliseconds this Connector will wait, after >>>>>>>>>>>>>> accepting a >>>>>>>>>>>>>> connection, for the request URI line to be presented. The >>>>>>>>>>>>>> default >>>>>>>>>>>>>> value >>>>>>>>>>>>>> for >>>>>>>>>>>>>> AJP protocol connectors is -1 (i.e. infinite). >>>>>>>>>>>>>> >>>>>>>>>>>>>> (You could for example try to set this to 3000 (milliseconds) >>>>>>>>>>>>>> or >>>>>>>>>>>>>> even >>>>>>>>>>>>>> lower. That should be more than enough for any legitimate >>>>>>>>>>>>>> client >>>>>>>>>>>>>> so >>>>>>>>>>>>>> send >>>>>>>>>>>>>> the HTTP request line. Note however that by doing this at the >>>>>>>>>>>>>> Tomcat >>>>>>>>>>>>>> level, you will probably move the problem to the Apache >>>>>>>>>>>>>> >>>>>>>>>>>>>> httpd/mod_jk >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>> level. But at least it might confirm that this is the problem >>>>>>>> >>>>>>>>> that you >>>>>>>>>>>>>> are >>>>>>>>>>>>>> seeing. The mod_jk logfile at the httpd level may give you >>>>>>>>>>>>>> some >>>>>>>>>>>>>> hints >>>>>>>>>>>>>> there.) >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> HTTP Connector : (http://tomcat.apache.org/tomc >>>>>>>>>>>>>> at-8.5-doc/config/http.html#Standard_Implementation) >>>>>>>>>>>>>> >>>>>>>>>>>>>> connectionTimeout >>>>>>>>>>>>>> >>>>>>>>>>>>>> The number of milliseconds this Connector will wait, after >>>>>>>>>>>>>> accepting a >>>>>>>>>>>>>> connection, for the request URI line to be presented. Use a >>>>>>>>>>>>>> value >>>>>>>>>>>>>> of -1 >>>>>>>>>>>>>> to >>>>>>>>>>>>>> indicate no (i.e. infinite) timeout. The default value is >>>>>>>>>>>>>> 60000 >>>>>>>>>>>>>> (i.e. >>>>>>>>>>>>>> 60 >>>>>>>>>>>>>> seconds) but note that the standard server.xml that ships with >>>>>>>>>>>>>> Tomcat >>>>>>>>>>>>>> sets >>>>>>>>>>>>>> this to 20000 (i.e. 20 seconds). Unless disableUploadTimeout >>>>>>>>>>>>>> is >>>>>>>>>>>>>> set to >>>>>>>>>>>>>> false, this timeout will also be used when reading the request >>>>>>>>>>>>>> body (if >>>>>>>>>>>>>> any). >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On 26.11.2016 09:57, Jaaz Portal wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> hi, >>>>>>>>>>>>>> >>>>>>>>>>>>>> sorry, its mod_jk no jk2, my typo. All at latest versions. We >>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> tried >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>> with >>>>>>>> >>>>>>>>> mod proxy too. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> There is no flood of the server. Nobody is flooding us, they >>>>>>>>>>>>>>> use >>>>>>>>>>>>>>> some >>>>>>>>>>>>>>> specific connections after which pool of apache workers is >>>>>>>>>>>>>>> exhausted >>>>>>>>>>>>>>> and >>>>>>>>>>>>>>> blocked >>>>>>>>>>>>>>> and we need to restart tomcat server. >>>>>>>>>>>>>>> It is some kind of exploit but do not know how to log it to >>>>>>>>>>>>>>> obtain >>>>>>>>>>>>>>> details. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> i had put a limit on connections per client with hope that >>>>>>>>>>>>>>> this >>>>>>>>>>>>>>> will >>>>>>>>>>>>>>> help >>>>>>>>>>>>>>> but once again, it is not a flood. >>>>>>>>>>>>>>> They open several connections that are not dropped by apache >>>>>>>>>>>>>>> when they >>>>>>>>>>>>>>> disconnect. This way whole pool is quickly exhausted and the >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> server >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>> broken. >>>>>>>> >>>>>>>>> >>>>>>>>>>>>>>> i would like to help you to figure details of this attack but >>>>>>>>>>>>>>> this is >>>>>>>>>>>>>>> production server so it is impossible to much debugging >>>>>>>>>>>>>>> options >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> best, >>>>>>>>>>>>>>> artur >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> 2016-11-25 23:44 GMT+01:00 Niranjan Babu Bommu < >>>>>>>>>>>>>>> niranjan.bo...@gmail.com >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> : >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> you can find who is flooding site in apache access.log and >>>>>>>>>>>>>>>> block >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> them >>>>>>>>>>>>>>> in >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> firewall. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> ex to find the IP: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> cat /var/log/apache2/access.log |cut -d' ' -f1 |sort |uniq >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> -c|sort >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>> -gr >>>>>>>> >>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Fri, Nov 25, 2016 at 8:42 AM, Jaaz Portal >>>>>>>>>>>>>>>> <jaazpor...@gmail.com> >>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> hi, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> we are from some weeks struggling with some Polish hackers >>>>>>>>>>>>>>>> that >>>>>>>>>>>>>>>> are >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> bringing our server down. After updating apache to latest >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> version >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>> >>>>>>>> (2.4.23) >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> and tomcat (8.0.38) available for debian systems we still >>>>>>>>>>>>>>>> cannot >>>>>>>>>>>>>>>> secure >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> our >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> server. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Today it has stopped to respond again and we needed to >>>>>>>>>>>>>>>>> restart >>>>>>>>>>>>>>>>> tomcat >>>>>>>>>>>>>>>>> process to get it back alive. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> There is no too much clues in the logs. The apache >>>>>>>>>>>>>>>>> error.log >>>>>>>>>>>>>>>>> gives >>>>>>>>>>>>>>>>> just >>>>>>>>>>>>>>>>> this line: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> [Fri Nov 25 13:08:00.647835 2016] [mpm_event:error] [pid >>>>>>>>>>>>>>>>> 13385:tid >>>>>>>>>>>>>>>>> 1397934896385 >>>>>>>>>>>>>>>>> 92] AH00484: server reached MaxRequestWorkers setting, >>>>>>>>>>>>>>>>> consider >>>>>>>>>>>>>>>>> raising >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> the >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> MaxR >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> equestWorkers setting >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> seems that somehow tomcat, mod-jk2 or even apache is >>>>>>>>>>>>>>>>> vulnerable to >>>>>>>>>>>>>>>>> some >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> new >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> exploit, as we certainly does not have such traffic that >>>>>>>>>>>>>>>> would >>>>>>>>>>>>>>>> block >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> our >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> server otherwise >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> for now we have increased MaxRequestWorkers and we have >>>>>>>>>>>>>>>>> limited >>>>>>>>>>>>>>>>> number >>>>>>>>>>>>>>>>> of >>>>>>>>>>>>>>>>> connections from one client to 5 by mod_bw and limited >>>>>>>>>>>>>>>>> number >>>>>>>>>>>>>>>>> of >>>>>>>>>>>>>>>>> simultaneous connections from one ip by iptables but does >>>>>>>>>>>>>>>>> not >>>>>>>>>>>>>>>>> know >>>>>>>>>>>>>>>>> if >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> this >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> will help >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> best regards, >>>>>>>>>>>>>>>>> artur >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> *Thanks* >>>>>>>>>>>>>>>> *Niranjan* >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> This sounds like a variant of the slowloris attack. >>>>>>> >>>>>>> This type of attack doesn't take a large number of clients or >>>>>>> consume a >>>>>>> large amount of bandwidth. >>>>>>> >>>>>>> Basically, the maximum number of connections are made to the server, >>>>>>> and >>>>>>> just enough data is sent to each connection in order to not trigger >>>>>>> the >>>>>>> timeout. André has explained this in more detail earlier in the >>>>>>> thread. >>>>>>> Search for "slowloris attack" for more information. >>>>>>> >>>>>>> There are several ways of mitigating this type of attack. >>>>>>> >>>>>>> As André has mentioned, placing a dedicated device in front of your >>>>>>> systems is often the best way. Lots of benefits (platform neutral, no >>>>>>> stress on your current servers), and some issues (cost, placement / >>>>>>> access may be an issue with hosted solutions). >>>>>>> >>>>>>> However, there are Apache HTTPD modules that can help mitigate these >>>>>>> types of attacks. Some of them are: >>>>>>> >>>>>>> mod_reqtimeout (should be included by default in your Apache HTRPD >>>>>>> 2.4) >>>>>>> mod_qos (quality of service module) >>>>>>> mod_security (application firewall with lots of security rules) >>>>>>> >>>>>>> Do a quick search on "slowloris attack apache httpd 2.4" to get some >>>>>>> ideas. >>>>>>> >>>>>>> All of them will probably place additional load on your Apache HTTPD >>>>>>> server, so make sure that the platform is robust enough to manage the >>>>>>> additional load. >>>>>>> >>>>>>> There is also a beta version of the mod_security module written as a >>>>>>> servlet filter. It should be possible to build this and configure the >>>>>>> filter in Tomcat's default web.xml ($CATALINA_BASE/conf/web.xml). >>>>>>> I've >>>>>>> not tried this. Also, the code base hasn't seen any activity for 3 >>>>>>> years. >>>>>>> >>>>>>> Do a quick search on "modsecurity servlet filter" to find out more >>>>>>> about >>>>>>> the servlet filter version of mod_security. >>>>>>> >>>>>>> In short, there appear to be some ways to mitigate the attack. >>>>>>> >>>>>>> . . . just my two cents >>>>>>> /mde/ >>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> --------------------------------------------------------------------- >>>>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>>>> For additional commands, e-mail: users-h...@tomcat.apache.org >>>>> >>>>> >>>>> >>>> >>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>> For additional commands, e-mail: users-h...@tomcat.apache.org >>> >>> >> >
