Hi André, Hi all,
On Mon, Apr 15, 2013 at 7:02 PM, André Warnier <a...@ice-sa.com> wrote: > > Well, you wrote that you wanted to know our thoughts. Yes or No would not > have been an appropriate response to that, so we elaborated a bit. > ;-) > > > Yes, true. Thank you! I am a bit unsure what's going on. So greatly appreciate your feedback. > >> Here's all I have from Apache's error log. Several entries looking like >> so >> starting at: >> ---------- >> [Thu Apr 11 21:24:14 2013] [error] [client nnn.nnn.nnn.nnn] (70007)The >> timeout specified has expired: proxy: error reading status line from >> remote >> server servername.do.ma.in >> [Thu Apr 11 21:24:14 2013] [error] [client nnn.nnn.nnn.nnn] proxy: Error >> reading from remote server returned by /web-app/url1/ >> [Thu Apr 11 21:24:55 2013] [error] [client nnn.nnn.nnn.nnn] proxy: Error >> reading from remote server returned by /web-app/url2, referer: >> http://example.com/people/**employees/regularemployee<http://example.com/people/employees/regularemployee> >> [Thu Apr 11 21:24:55 2013] [error] [client nnn.nnn.nnn.nnn] (70007)The >> timeout specified has expired: proxy: error reading status line from >> remote >> server servername.do.ma.in, referer: https://www.google.com/ >> --------- >> >> Apache and Tomcat are in the same city, and in the same data-center. >> >> And Apr 11 21:05 is when I restarted Apache after removing several unused >> modules - hard stop and start. The way I restarted Apache is: >> sudo /usr/sbin/service httpd stop >> sudo /usr/bin/servive httpd start >> >> > So you wrote that you restarted Apache at 21:05, and that about a minute > later you had some Server Errors. The log messages above are dated of > 21:24. That is 20 minutes after the restart, not really 1 minute. > Here are the some facts from the HTTPD access log(!): On 4/11/2013: HTTPD was down from 21:03 - 21:04:07 HTTPD started returning several Error 502's from: 21:20:35 to 21:37:58 Tomcat seemed to hang from 21:20 onwards - per our monitoring software Tomcat restart was issued at: 21:46 Tomcat was OK at 21:49 It appears to me that restarting Apache HTTPD caused Tomcat to be confused about something. About what? Some things I did in order to trim Apache HTTPD include: (1) I renamed the proxy_ajp.conf to proxy_ajp.conf.notinclude and balancers.conf to balancers.conf.notinclude since these are not being used ( I use mod_proxy and mod_proxy_http) (2) I trimmed HTTPD modules from 47 to 17 by commenting out "LoadModule" statements and removed offending directives in httpd.conf (3) I restarted HTTPD If I have to express a thought based on that conflicting information, then > I would guess that the errors are probably not related to the restart of > Apache per se. > It may just have been a coincidence. Although using the term > "coincidence" for events that happen at a 20-minute interval on a > production server seems a bit of a stretch. > I agree that that theory would be a bit if a stretch. > > > In any case, there is nothing there that specifically links the Apache > restart, with the errors that happened later. > I know. The funny thing is that happened independently on more than one server - same behavior. > If Tomcat tried to send a response later (after Apache httpd timed-out), > then there should probably be an error message also in the Tomcat logs, > since it would try to write back a response onto a connection that did not > exist anymore. > > Have you checked the Tomcat logs ? (which may be in /var/log/daemon, > depending on the script which starts Tomcat). There is surprisingly nothing that stands out in the Tomcat logs from 4/11/ evening. I do not have the access logs enabled on the Tomcat side. And one log has rolled over. But I doubt if that will have anything to say. Although I will look into restoring that log file to check. Some questions I have include - if HTTPD is restarted, what happens to Tomcat's thread pool? Will they timeout after writing to a half-closed connection? When HTTPD starts up, will it open new connections to Tomcat's ports - I think the answer is "yes". HTTPD should start minServer processes and connect to Tomcat? What might cause Tomcat to work somewhat and to not work somewhat, at the same time. Perhaps old Tomcat threads connecting to the half-closed HTTPD connections might be confused, like you suggest? But the new connections with HTTPD after the restart might be okay? Thanks much. -Shanti
