OK, it looks like I can reproduce this. Steps to reproduce:
- Windows 2016 Server fully patched - Java 1.8.0u144 - Install Tomcat 8.5.45 from windows installer - Add tcnative-1.dll (64-bit) from Tomcat Native 1.2.23 - Modify server.xml to use Http11AprProtocol on port 8080 - Make a single request I then see 1 core running at 100% until the connection times out after 20s. Make another request and a core goes back up to 100% for 20s (the default keep-alive time out). Next steps are to try and track down the root cause. Mark > Mark and M, > > On 11/13/19 19:31, Mark Thomas wrote: >> On November 13, 2019 11:42:34 PM UTC, "M. Manna" >> <manme...@gmail.com> wrote: >>> I see this update on Windows which may have been responsible >>> (suspicion only, haven’t rolled it back yet) >>> >>> >>> https://support.microsoft.com/en-gb/help/4494175/kb4494175-intel-micr > ocode-updates >>> >>> >>> > Was 8.5.45 built on Windows 10 in presence of this update ? > >> No. Tomcat 8.5.45 and Tomcat Native 1.2.23 were built on a fully >> patched at the time of the build Windows 7 64-bit VM. > Also it doesn't matter because binaries don't include CPU microcode. > > It's more likely that the target system has microcode updates such as > these that may negatively impact performance. > > -chris > >>> >>> Thanks, >>> >>> On Wed, 13 Nov 2019 at 17:55, M. Manna <manme...@gmail.com> >>> wrote: >>> >>>> Hi Chris, >>>> >>>> On Wed, 13 Nov 2019 at 16:27, Christopher Schultz < >>>> ch...@christopherschultz.net> wrote: >>>> >> On 11/13/19 11:20, M. Manna wrote: >>>>>>> HI Mark, >>>>>>> >>>>>>> On Wed, 13 Nov 2019 at 15:38, Mark Thomas >>>>>>> <ma...@apache.org> wrote: >>>>>>> >>>>>>>> On 12/11/2019 19:11, M. Manna wrote: >>>>>>>>> HI Mark, >>>>>>>>> >>>>>>>>> following my previous reply, we have now confirmed >>>>>>>>> that it's indeed >>>>>>>> 8.5.45 >>>>>>>>> with APR 1.2.23 that's causing such high JVM CPU >>>>>>>>> usage. We used took out 2 out of 50 servers from the >>>>>>>>> load balancer config, reverted tomcat, and >>>>>>>>> redeployed. With near to identical user traffic, the >>>>>>>>> two servers are responding normally without/without >>>>>>>>> traffic with 8.5.41. The JVM dump looks a lot better >>>>>>>>> with 8.5.41. >>>>>>>>> >>>>>>>>> We do think that the recent changes in APR and some >>>>>>>>> other tomcat jar may have caused compatibility issue >>>>>>>>> on Windows server 2016 (64-bit) platform. But >>>>>>>>> unfortunately, we cannot pinpoint exactly what change >>>>>>>>> may have caused this (i.e. actual OS vs Security >>>>>>>>> Updates). With this in mind, we are also being wary >>>>>>>>> to move to 8.5.47 as we don't know if the same issue >>>>>>>>> will >>>>>>>> occur >>>>>>>>> again. Since 8.5.41 has been packaged with previously >>>>>>>>> accepted >>>>>>>> application >>>>>>>>> installer, we are more comfortable rolling back. >>>>>>>> >>>>>>>> Just to confirm, you see this high CPU usage with a >>>>>>>> clean install (no additional web applications deployed, >>>>>>>> no configuration changes) on Windows 2016 DataCenter >>>>>>>> (64-bit)? >>>>>>>> >>>>>>>> If this is the case, it should be fairly easy to >>>>>>>> reproduce. >>>>>>>> >>>>>>>> Mark >>>>>>>> >>>>>>>> We do not deploy multiple applications. In fact, Under >>>>>>>> tomcat >>>>>>> webapps/ROOT we only have one application (ours). Each >>>>>>> tomcat instance is hosted on a VM (total 50) and all of >>>>>>> them are identically configured (server.xml, web.xml, >>>>>>> logging, CPU/RAM). We have not made any other >>>>>>> configuration change between 8.5.41 and 8.5.45. And yes, >>>>>>> I agree with you that it's fairly easy to reproduce. > >> I think the question is whether or not your application is >> required >>>> to >> be deployed. Can you reproduce this issue with just the stock >> applications bundled with Tomcat? > >>>>> >>>>> My apologies, but our application needs to be deployed. We >>>>> have not >>>> (or >>>>> didn't try in the past) to simply deploy tomcat with stock >>>> application (in >>>>> other words, simply starting the tomcat OOB) on our prod >>>>> servers. This is the first time it has hit us with such >>>>> disparity. I’ll try to investigate and get a stock >>>>> application data. But we may not be able >>>> to do >>>>> that quite easily as it’s in our production. >>>>> >>>>> What I can see is that 3 Windows updates may have been >>>>> responsible >>>> for >>>>> this, but we aren’t sure about that. I’ll let you know if we >>>>> can get anything with the stock application instance. >>>>> >>>>> Thanks, >>>>> >>>>> - -chris >>>>> >>>>> >>>>> >>> --------------------------------------------------------------------- >>>>> >>> > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >>>>> For additional commands, e-mail: >>>>> users-h...@tomcat.apache.org >>>>> >>>>> > > >> --------------------------------------------------------------------- > > > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org >> For additional commands, e-mail: users-h...@tomcat.apache.org > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
