Session timeout
All who can help, I am having a problem getting Tomcat to time the session out. The setup: Tomcat 5.5.9 running on Slackware using Java 5. The issue: We have two general classes of users: Administrator types who need a long session timeout and casual users who need a very short timeout. Several times a year, we get many requests by casual users (100's per minute for hours at a time.) They come to a jsp that has the max inactive interval set to 120 seconds i.e., session.setMaxInactiveInterval(120). Yet, after 24 hours, some of these sessions are still active. I have checked at various points on this page and the session timeout is reported as being set to 120 (this page presents some information gathered from a database and the user's selection will then make another request.) We have a listener that removes some information that could have been stored for the user. If I force a session end with session.invalidate(), I can see the listener being hit. But, without programatically forcing an end to the session, it never hits the listener. The other class of user (the administrators) is equally flaky but in the other direction. Sometimes the session is active for a long time (maxInactiveInterval is set to 7200 seconds) and sometimes it goes dead after a fairly short time (say, 15 minutes.) I suspect the 'going dead' issue is caused by the database connection being timed out and I am investigating that possibility. What I am trying to determine is how to control the session inactive period. Thanks in advance for any and all help. Carl No virus found in this outgoing message. Checked by AVG Free Edition. Version: 7.1.405 / Virus Database: 268.10.9/417 - Release Date: 8/11/2006 - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
404 error when a newly created file is requested
Environment: Tomcat 5.5.17, Windows XP
Background: I have a jsp page that contains a javascript rich text editor that
I use to create snippets of HTML code that are stored in a database and later
displayed on other pages. Since the snippets can contain some custom tags, I
have a 'Preview' button that reads the snippet directly from the editor, reads
a 'preview' jsp file from the file system, places the snippet in the correct
place in the file, writes the 'preview' file with the snippet to a temporary
file and opens a new window (using javascript) which requests the temporary
page from Tomcat.
Problem: I can see the newly created temporary file in the file system. I can
open it with an editor and it is all good. But, Tomcat returns a 404 error for
approximately one minute and then will serve the page upon hitting only the
refresh.
Analysis: I suspect the system is holding onto the newly written file until it
hits some timeout but I have tried everything I can think of to make certain
the file has been 'released'. The relevent parts of the copy process:
try {
File tempFile = File.createTempFile("preview", ".jsp", new
File("C:/projects/EtrakWebApp/web/jsp/tempfiles"));
// read the preview jsp
FileWriter fw = new FileWriter(tempFile);
File infile = new File(destinationDir+"preview.jsp");
if( infile.exists() )
{
FileReader fr = new FileReader(infile);
BufferedReader in = new BufferedReader(fr);
while(true)
{
String line = in.readLine();
if( line==null )
break;
System.out.println("line "+line );
// insert the text
if( line.contains("insert_preview_text") )
{
fw.write(previewText+"\n");
}
else {
fw.write(line+"\n");
}
}
fr.close();
in.close();
}
fw.flush();
fw.close();
fw = null;
}
catch (FileNotFoundException fnf)
{
fnf.printStackTrace();
}
catch (Exception ex)
{
ex.printStackTrace();
}
return returnFileName;
Anyone have any ideas?
TIA,
Carl Kabbe
No virus found in this outgoing message.
Checked by AVG Free Edition.
Version: 7.1.409 / Virus Database: 268.15.19/587 - Release Date: 12/14/2006
-
To start a new topic, e-mail: users@tomcat.apache.org
To unsubscribe, e-mail: [EMAIL PROTECTED]
For additional commands, e-mail: [EMAIL PROTECTED]
Re: DWR [Slightly off topic -- Was Struts vs JSF (poll?)]
Interesting discussion because we, too, tried much of what you have described. Our difficulty, and this is where I am deviating a bit, was that different browsers rendered the GUI different ways unless you spent lots of hours twisting some of the code (we also felt the next version of the browser would probably break the code) and our users range from experienced to barely able to turn the computer on. What we needed was something that behaved like a fat client GUI but ran over the Internet/browser process. We went to Flash. For advanced users, we can provide a GUI that is as advanced as any fat client desktop application and for the entry level users we have great control over what they can do. We get easy access to things like drag and drop, tabbed navigation, modeless dialogs, tracking mouse actions, etc. All the backend parts (data access, heavy calculations, etc.) are still done by Tomcat. The downside is a little bit of duplication in beans containing information that is passed back and forth (could be scripted but we have just hand created to date.) Most of the administrative side is still HTML, AJAX, etc. as we train these people so we can curb bad behavior. With one exception, we start the Flash applications from the standard menuing in the HTML pages (interestingly, the menuing is also a Flash application as it seems like the best presentation.) Just some thoughts. Thanks, Carl - Original Message - From: "Frank W. Zammetti" <[EMAIL PROTECTED]> To: "Tomcat Users List" Sent: Sunday, August 24, 2008 5:34 PM Subject: Re: DWR [Slightly off topic -- Was Struts vs JSF (poll?)] Ken Bowen wrote: Frank, This may be too far off-list; if so please suggest another channel. However, there may be other developers lurking on the list who would also be interested. I'm sure there's a better place, but I'm not sure where it would be. I agree, it's a question others may have interest in here... in a way, a list for Tomcat almost necessarily brings up other tangential topics like this :) I've downloaded DWR and performed a cursory look through the examples, and it looks promising (I've done a chunk of very bare-metal ajax, and a chunk of GWT). I've also read your blog entry on the subject and am quite convinced about the desirability of an RPC-like framework. What I don't have a feel for is the mix between ordinary html and javascript that might be forced on you by DWR. I have a major application at work that I've been involved with for nearly two years now... it's a really interesting case study I think. I'll frankly have to be a little careful what I say here because it's a back-office application that gives us a pretty good competitive edge, but we're a little secretive about it unless you're a client, or prospective client (not entirely sure why)... I'll try to get the basic gist of it across though... The underlying concept is we have a single application that essentially houses a host of others. I hesitate to use the term portal to describe it, and it most certainly does NOT use portal technology, but conceptually that's along the right lines. It's a highly complex RIA, I've said for some time I'd put it up against any out there in terms of complexity. It really does take the notion of a webapp that looks, feels and functions like a traditional fat client to a high level of realization. Now, when we started this project, we were handed a fairly inexperienced team, certainly when it comes to RIAs. So, we had some early choices to make to try and make things as easy for them as possible. Because they had some experience with Struts, we went with that. Early on we showed them how you could have Struts return JSON and make Ajax requests to get it (duh, who'da thunk you could render JSON in a JSP?!?). So, when you hit the initial URL, you do a fairly typical Struts-type load, i.e., you have some tiles that get populated, some Actions that execute, and you wind with up a final rendered HTML page on the client. This essentially is the shell of the application where "modules", so to speak, can be plugged into. After that, nearly everything that happens is Ajax-based, and Struts becomes little more than a router for requests. We rarely ever return HTML from the server after that initial load, and only in one or two special circumstances to we reload the HTML page initially loaded into the browser (think security situations). Now, here's where it gets interesting... over the last year we've been expanded the capabilities of this application at a phenomenal rate... it's the typical "oh look, that project was a huge success, I want in!" kind of situation... everyone's got uses for it now! However, we've started to migrate away from Struts entirely because it wasn't really buyin
Re: reload no use?
We have a similar problem with our app. We narrowed the problem down to the way we have constructed/used our connection pooling. Our solution for the short term is to restart the server each time we update the app. At some point, we will get time to restructure the connection pooling process so that we are not holding onto objects that can't be freed. Thanks, Carl - Original Message - From: "John Pedersen" <[EMAIL PROTECTED]> To: "Tomcat Users List" Sent: Sunday, August 12, 2007 10:52 AM Subject: Re: reload no use? > Nothing that suggests a memory leak. Do you have any suggestions how I might > go about finding where this might be happening? It only seems to happen on > reload - otherwise, the app runs and runs. > > On 12/08/07, Mark Thomas <[EMAIL PROTECTED]> wrote: > > > > John Pedersen wrote: > > > When I reload an app for the third time, Tomcat hangs, and I have to > > stop > > > and start it - which is slow as the CPU is being used at close to 100%. > > > > > > This has been the case with Tomcat 5 and 6, Windows 2000, XP and Vista. > > > > > > I have to remember to stop/start Tomcat after the second reload. Is > > there a > > > better way? > > > > I regularly re-load apps and the only time I have a problem is when > > the app leaks memory. I would guess your app has a memory leak. Is > > there anything in the logs? > > > > Mark > > > > - > > To start a new topic, e-mail: users@tomcat.apache.org > > To unsubscribe, e-mail: [EMAIL PROTECTED] > > For additional commands, e-mail: [EMAIL PROTECTED] > > > > > No virus found in this incoming message. Checked by AVG Free Edition. Version: 7.5.476 / Virus Database: 269.11.13/947 - Release Date: 8/11/2007 2:29 PM - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: RESIDENTIAL MEMORY 100%
As I recall from one of your early emails, you have the session timeout set to 24 hours. If this is so, you will never get this app to run. Thanks, Carl - Original Message - From: "karthikn" <[EMAIL PROTECTED]> To: "Tomcat Users List" Sent: Tuesday, March 25, 2008 8:07 AM Subject: Re: RESIDENTIAL MEMORY 100% Hi Sorry for the dely, thx for the prompt reply I was just looking into using Jconsole as u said,but since the UNIX 11 server,does not have GUI,I probably would have to do some remote Jconsole for the same. Can u please tell me or provide me URL on how do i set up "mbeans-descriptor.xml" so the jconsole can be usaed for monitering remote JMX. Tomcat documents do not say any thing about how to achieve the same. Spec Os = Unix 11 jsdk = 1.6 Tomcat =5.5.23 version thx in advance with regards karthik Alan Chaney wrote: Firstly, you are only making 350M of your memory available to the application. xmx=350m means 350 megabtyes is reserved for the heap. Secondly, that's probably not the issue! You are correct in that you are running a 32 bit JVM. That's probably not the issue either but it means that most of the 6GB of RAM you said you had is not actually relevant. Next. you keep saying that you are using netstat to look at your system. but the display below is that of 'top'. I'm not familiar with your Unix version so maybe netstat is different on your system. Neither netstat nor top are that much use for delving into linux memory issues, never mind about java issues. It is extremely difficult to correlate the way that linux uses memory to what is actually happening inside the JVM. The next issue is that thousands of people all over the world use tomcat and don't suffer from the problem that you are seeing, so it is most likely that the problem is with your application. I notice that you have 535 process listed - that's an awful lot. You need to investigate what is actually happening inside the application - the best way of doing that is to use either jconsole or a profiling tool. You are using java 6 which means that you can easily set up a jconsole session to display what is happening. If you use jconsole you will be able to check the real JVM memory usage over time and force a GC to see how much of that memory is actually held by a strong reference. And of course what do the log files show? Are you getting out of memory errors? Are you getting RTEs? Another thing to check is that is the system swapping? I notice that in the statistics below you don't show the swap usage. This is often a trap for the unwary. HTH karthikn wrote: Hi >> is running a 64bit JVM. JDK used is 32 bit probably by running java -version on this Unix 11 server bash-3.2# ./java -version java version "1.6.0.00" Java(TM) SE Runtime Environment (build 1.6.0.00-jinteg_12_nov_2007_21_58-b00) Java HotSpot(TM) Server VM (build 1.6.0.00 jinteg:11.12.07-21:02 PA2.0 (aCC_AP), mixed mode) This version is avaliable from HP Unix site http://h18012.www1.hp.com/java/ As per HP site 64 bit is the version HP is distributing I also have some more netstats report for last 4 hrs before we did the restart of the web container ( TOMCAT 5523) Tue Mar 25 15:52:06 2008 System: voise1 Load averages: 1.45, 2.23, 2.28 535 processes: 509 sleeping, 25 running, 1 zombie Cpu states: CPU LOAD USER NICESYS IDLE BLOCK SWAIT INTR SSYS 01.23 2.0% 0.0% 6.9% 91.2% 0.0% 0.0% 0.0% 0.0% 11.66 6.8% 0.0% 3.9% 89.3% 0.0% 0.0% 0.0% 0.0% --- - - - - - - - - avg 1.45 4.9% 0.0% 4.9% 90.3% 0.0% 0.0% 0.0% 0.0% Memory: 3894732K (1548140K) real, 7334992K (3049776K) virtual, 829420K free Page# 1/36 CPU TTYPID USERNAME PRI NISIZE RES STATETIME % WCPU% CPU COMMAND 0pts/3 16406 root 152 20 2412M1218M run 103:21 31.77 31.72 java 1 ?12779 root 152 24 282M 163M run 2:27 9.90 9.88 xas 1 pts/4 8251 root 178 20 11608K 9628K run 0:02 1.12 1.12 top 1 ?12733 root 152 24 79064K 29540K run 3:28 1.06 1.05 voiseBLG 0 ?37root 152 20 8832K8832K run 3:44 0.55 0.55 vxfsd Over a period of 12 hrs the RES STAT reaches 2000M when CPU for java is 100% The web application refuses to display the main page,even though we have the GC configured as "JAVA_OPTS=-server -Xms350m -Xmx350m -XX:+UseParallelGC -verbose:gc" with regards Karthik sterling wrote: I am sorry, if this is the wrong list, for submitting questions about apache on linux and router issues. This problem generated from alcatel to zoom x6 router change. Traceroute to webpage reports router addres
Re: [OT] Comodo as a CA
Have never used Comodo but have used Thawte which is in the same general price range as Comodo. Good experience, support was helpful when we made some dumb errors. Thanks, Carl - Original Message - From: "Christopher Schultz" <[EMAIL PROTECTED]> To: "Tomcat Users List" Sent: Saturday, April 26, 2008 8:40 AM Subject: Re: [OT] Comodo as a CA -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Anyone? Christopher Schultz wrote: | All, | | I'm asking the Tomcat community this because everyone always surprises | me with their wide customer experiences. | | I need to get new SSL certs for a domain name switch we're doing, soon. | VeriSign's SGC/EV SSL certs are ridiculously expensive and so I'm | looking for alternatives. Comodo offers a product that is | indistinguishable (to me) to VeriSign's SGC/EV cert, but it costs 1/4 as | much. | | Does anyone have any experience with Comodo? Specifically, has anyone | run across any customers whose browsers do not trust Comodo as a CA? | | Thanks, | -chris - - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkgTMMcACgkQ9CaO5/Lv0PASDgCePtVQFzSF9gFlphqc1HHWvh2O fecAoIW7BC7ns5hqCBYnzj6aoH6cX0Co =J9yo -END PGP SIGNATURE- - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- No virus found in this incoming message. Checked by AVG. Version: 7.5.524 / Virus Database: 269.23.5/1398 - Release Date: 4/25/2008 2:31 PM - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
Re: [OT] Comodo as a CA
Aggressive marketing, aggressive pricing, no functional advantages. Their support was not quite as good as Thawte (Thawte's doc's are very clear) as Verisign's response time was a little longer than Thawte's (neither was awful.) Thanks, Carl - Original Message - From: "Martin Gainty" <[EMAIL PROTECTED]> To: "Tomcat Users List" Sent: Saturday, April 26, 2008 9:39 AM Subject: Re: [OT] Comodo as a CA Hi Carl Thanks for Thawte feedback Whats your take on Verisign? this is O/T Feel free to ping me offline Thanks Martin - Original Message - From: "Carl" <[EMAIL PROTECTED]> To: "Tomcat Users List" Sent: Saturday, April 26, 2008 11:33 AM Subject: Re: [OT] Comodo as a CA Have never used Comodo but have used Thawte which is in the same general price range as Comodo. Good experience, support was helpful when we made some dumb errors. Thanks, Carl - Original Message - From: "Christopher Schultz" <[EMAIL PROTECTED]> To: "Tomcat Users List" Sent: Saturday, April 26, 2008 8:40 AM Subject: Re: [OT] Comodo as a CA -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Anyone? Christopher Schultz wrote: | All, | | I'm asking the Tomcat community this because everyone always surprises | me with their wide customer experiences. | | I need to get new SSL certs for a domain name switch we're doing, soon. | VeriSign's SGC/EV SSL certs are ridiculously expensive and so I'm | looking for alternatives. Comodo offers a product that is | indistinguishable (to me) to VeriSign's SGC/EV cert, but it costs 1/4 as | much. | | Does anyone have any experience with Comodo? Specifically, has anyone | run across any customers whose browsers do not trust Comodo as a CA? | | Thanks, | -chris - - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.9 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkgTMMcACgkQ9CaO5/Lv0PASDgCePtVQFzSF9gFlphqc1HHWvh2O fecAoIW7BC7ns5hqCBYnzj6aoH6cX0Co =J9yo -END PGP SIGNATURE- - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- No virus found in this incoming message. Checked by AVG. Version: 7.5.524 / Virus Database: 269.23.5/1398 - Release Date: 4/25/2008 2:31 PM - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED] -- No virus found in this incoming message. Checked by AVG. Version: 7.5.524 / Virus Database: 269.23.5/1398 - Release Date: 4/25/2008 2:31 PM - To start a new topic, e-mail: users@tomcat.apache.org To unsubscribe, e-mail: [EMAIL PROTECTED] For additional commands, e-mail: [EMAIL PROTECTED]
SSL problem
Fresh Tomact 6.0.20 install on a new Slackware (version 13 - 64bit) linux box. Access to 8080 works fine but neither 8443 nor 443 work. I believe 443 and 8443 are up because a nmap gives: Discovered open port 443/tcp on 127.0.0.1 Discovered open port 8080/tcp on 127.0.0.1 Discovered open port 8443/tcp on 127.0.0.1 The relevant portions of server.xml are: The cert is from Thawte and has been used in an existing server for the past year: r...@tomcat2:/usr/local/certs# -rw-r--r-- 1 root root 2493 2008-12-12 17:33 tomcat_keystore.ks Yes, I have googled (for the last day) and found many ideas but nothing seems to fit so I am asking if anyone sees anything that might be the cause. TIA, Carl
Re: SSL problem
Chuck, When I try to go to either 8443 or 443, with: https://10.10.10.30/ or https://10.10.10.30:8443/ I get the following: Secure Connection Failed An error occurred during a connection to 10.10.10.30. SSL received a record that exceeded the maximum permissible length. (Error code: ssl_error_rx_record_too_long) * The page you are trying to view can not be shown because the authenticity of the received data could not be verified. * Please contact the web site owners to inform them of this problem. Alternatively, use the command found in the help menu to report this broken site. but when I use https://10.10.10.30:8080/, I get the Tomcat page. nstat -an gives nothing, no output. The entire server.xml is: (rather large as I haven't started pruning out the things I don't need): SSLEngine="on" /> className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" /> keystoreFile="/usr/local/certs/tomcat_keystore.ks" keystorePass="jellybean"/> keystoreFile="/usr/local/certs/tomcat_keystore.ks" keystorePass="jellybean"/> INFO: The APR based Apache Tomcat Native library which allows optimal performanc e in production environments was not found on the java.library.path: /usr/lib64/ java/lib/amd64/server:/usr/lib64/java/lib/amd64:/usr/lib64/java/../lib/amd64:/us r/java/packages/lib/amd64:/lib:/usr/lib Dec 15, 2009 2:43:02 PM org.apache.coyote.http11.Http11Protocol init INFO: Initializing Coyote HTTP/1.1 on http-8080 Dec 15, 2009 2:43:02 PM org.apache.coyote.http11.Http11Protocol init INFO: Initializing Coyote HTTP/1.1 on http-8443 Dec 15, 2009 2:43:02 PM org.apache.coyote.http11.Http11Protocol init INFO: Initializing Coyote HTTP/1.1 on http-443 Dec 15, 2009 2:43:02 PM org.apache.catalina.startup.Catalina load INFO: Initialization processed in 550 ms Dec 15, 2009 2:43:02 PM org.apache.catalina.core.StandardService start INFO: Starting service Catalina Dec 15, 2009 2:43:02 PM org.apache.catalina.core.StandardEngine start INFO: Starting Servlet Engine: Apache Tomcat/6.0.20 RequestFilter initialized RequestFilter initialized RequestFilter initialized Dec 15, 2009 2:43:04 PM org.apache.coyote.http11.Http11Protocol start INFO: Starting Coyote HTTP/1.1 on http-8080 Dec 15, 2009 2:43:05 PM org.apache.coyote.http11.Http11Protocol start INFO: Starting Coyote HTTP/1.1 on http-8443 Dec 15, 2009 2:43:05 PM org.apache.coyote.http11.Http11Protocol start INFO: Starting Coyote HTTP/1.1 on http-443 Dec 15, 2009 2:43:05 PM org.apache.jk.common.ChannelSocket init INFO: JK: ajp13 listening on /0.0.0.0:8009 Dec 15, 2009 2:43:05 PM org.apache.jk.server.JkMain start Hope this helps solve the mystery. TIA, Carl - Original Message - From: "Caldarale, Charles R" To: "Tomcat Users List" Sent: Tuesday, December 15, 2009 1:26 PM Subject: RE: SSL problem From: Carl [mailto:c...@etrak-plus.com] Subject: SSL problem Access to 8080 works fine but neither 8443 nor 443 work. What does that mean? Does Tomcat return a message saying "not working"? (I doubt it.) I believe 443 and 8443 are up because a nmap gives: Do a netstat -an while Tomcat is running and make sure it's actually listening on those ports. The relevant portions of server.xml are: Did you perhaps snip out the comment marker that the default server.xml has around the SSL ? Post all of your server.xml. Are you using the APR library (probably named tcnative-1.so in Tomcat's bin directory)? If so, the SSL configuration is quite different. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: SSL problem
Mark and Chuck, I am so embarrassed, I should have caught that. It works properly with that one little addition. You guys (and others also) provide so much help. All I can say is thanks. Carl - Original Message - From: "Mark Thomas" To: "Tomcat Users List" Sent: Tuesday, December 15, 2009 1:28 PM Subject: Re: SSL problem On 15/12/2009 18:18, Carl wrote: keystoreFile="/usr/local/certs/tomcat_keystore.ks" keystorePass="jellybean"/> You are missing SSLEnabled="true" to tell Tomcat to actually use SSL for that connector. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
JVM goes away
This is a new server, a Dell T110 with a Xeon 3440 processor and 4GB memory. I have turned off both the turbo mode and hyperthreading. The environment: 64 bit Slackware Linux java version "1.6.0_17" Java(TM) SE Runtime Environment (build 1.6.0_17-b04) Java HotSpot(TM) 64-Bit Server VM (build 14.3-b01, mixed mode) Tomcat: apache-tomcat-6.0.20 JAVA_OPTS="-Xms2400m -Xmx2400m -XX:PermSize=512m -XX:MaxPermSize=512m" I have watched observed the memory usage and general performance with Java VisualVM and have seen nothing strange. GC seems to be performing well and the memory rarely gets anywhere near the max. The server runs well, idling along at 2-5% load, serving jsp's, etc. at a reasonable speed. Without warning and with no tracks in any log (Tomcat or system) or to the console, the JVM will just go away, disappear. Sometimes, the system will run for a week, sometimes for only several hours. Initially, I thought the problem was the turbo or hyperthreading but, no, the problem persists. When the JVM goes away, the memory that it held is still being held (as seen from top) but it is nowhere near the machine physical memory. The application has been running on an older server (Dell 600SC, 32 bit Slackware, 2GB memory) for several years and, while the application will throw exceptions now and then, it never crashed the JVM. This leads me to believe the problem has something to do with the 64 bit JVM but, with errors, I can't be certain and don't know what I can do about it except go back to 32 bit. I plan to reinstall Java tonight but, it would seem if the JVM were corrupted, it simply would not run. Any ideas are welcome. TIA, Carl
Re: JVM goes away
Peter and Andy, Thanks for your quick responses. Memory: Physical - $GB Used - 2.4GB to 3.0 GB (according to top... have never seen it above 3GB) Swap - 19GB, none ever used (or, at least I have never seen any used.) The above are all from top. The 2.4GB is after the JVM just crashed (after running less than an hour after having run for five days with nary a blip) and I just restarted Tomcat (customers are running right now) so it is a little higher than normal because it has perhaps .5GB+ unrecovered from the point at which the JVM crashed. I checked dmsg but saw nothing that looked out of the ordinary. I will cut back on the heap and permgen tonight (gonna be a long one.) Any ideas are welcome. Thanks, Carl - Original Message - From: "Peter Crowther" To: "Tomcat Users List" Sent: Monday, January 11, 2010 6:06 PM Subject: Re: JVM goes away 2010/1/11 Carl : This is a new server, a Dell T110 with a Xeon 3440 processor and 4GB memory. I have turned off both the turbo mode and hyperthreading. The environment: 64 bit Slackware Linux java version "1.6.0_17" Java(TM) SE Runtime Environment (build 1.6.0_17-b04) Java HotSpot(TM) 64-Bit Server VM (build 14.3-b01, mixed mode) Tomcat: apache-tomcat-6.0.20 JAVA_OPTS="-Xms2400m -Xmx2400m -XX:PermSize=512m -XX:MaxPermSize=512m" I have watched observed the memory usage and general performance with Java VisualVM and have seen nothing strange. GC seems to be performing well and the memory rarely gets anywhere near the max. The server runs well, idling along at 2-5% load, serving jsp's, etc. at a reasonable speed. Without warning and with no tracks in any log (Tomcat or system) or to the console, the JVM will just go away, disappear. Sometimes, the system will run for a week, sometimes for only several hours. Initially, I thought the problem was the turbo or hyperthreading but, no, the problem persists. When the JVM goes away, the memory that it held is still being held (as seen from top) but it is nowhere near the machine physical memory. The application has been running on an older server (Dell 600SC, 32 bit Slackware, 2GB memory) for several years and, while the application will throw exceptions now and then, it never crashed the JVM. This leads me to believe the problem has something to do with the 64 bit JVM but, with errors, I can't be certain and don't know what I can do about it except go back to 32 bit. I plan to reinstall Java tonight but, it would seem if the JVM were corrupted, it simply would not run. Any ideas are welcome. I'm with Andy: the Linux OOM killer would show those symptoms. With those settings, you're not leaving a lot of memory for the OS. How much swap do you have, and does the same thing happen if you reduce the Java heap and permgen space? - Peter - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: JVM goes away
Andy, Yes, that is 4GB... just a little stressed. I did a 'find' for all 'hs_err*.pid' files and turned up nothing, no files were found. I am using catalina.sh to start Tomcat and I had always assumed that the java JVM was started in that process somewhere. I apologize for my ignorance here but I don't see any java processes other than Tomcat and Visual JVM (using ps -ef.) I do have a little java listener also running that serves some data to client applets. Neither the little listener nor Visual JVM went down when Tomcat stopped. Tomcat is in /usr/local/tomcat (bin, conf, etc.) Java is in usr/local/java. Is the 'cwd of the java process' the directory where the application (Tomcat) is running or the bin directory of java? (I don't see anything in any of those areas that looks odd or informative.) How can I tell if the JVM is or is not running as a daemon? TIA, Carl - Original Message - From: "Andy Wang" To: Sent: Monday, January 11, 2010 6:42 PM Subject: Re: JVM goes away I assume $GB means 4GB :) With that kind of memory use it doesn't sound entirely like the OOM killer. Have you looked around the filesystem for hs_err[pid].pid files? This usually is written to the cwd of the java process. That might give you ideas if it's a native crash. If so, it'll have the java stack, and other native information that might shed some light. Otherwise, if the Tomcat JVM isn't running as a daemon, is it nohup'ed? Andy On 01/11/2010 05:33 PM, Carl wrote: Peter and Andy, Thanks for your quick responses. Memory: Physical - $GB Used - 2.4GB to 3.0 GB (according to top... have never seen it above 3GB) Swap - 19GB, none ever used (or, at least I have never seen any used.) The above are all from top. The 2.4GB is after the JVM just crashed (after running less than an hour after having run for five days with nary a blip) and I just restarted Tomcat (customers are running right now) so it is a little higher than normal because it has perhaps .5GB+ unrecovered from the point at which the JVM crashed. I checked dmsg but saw nothing that looked out of the ordinary. I will cut back on the heap and permgen tonight (gonna be a long one.) Any ideas are welcome. Thanks, Carl - Original Message - From: "Peter Crowther" To: "Tomcat Users List" Sent: Monday, January 11, 2010 6:06 PM Subject: Re: JVM goes away 2010/1/11 Carl : This is a new server, a Dell T110 with a Xeon 3440 processor and 4GB memory. I have turned off both the turbo mode and hyperthreading. The environment: 64 bit Slackware Linux java version "1.6.0_17" Java(TM) SE Runtime Environment (build 1.6.0_17-b04) Java HotSpot(TM) 64-Bit Server VM (build 14.3-b01, mixed mode) Tomcat: apache-tomcat-6.0.20 JAVA_OPTS="-Xms2400m -Xmx2400m -XX:PermSize=512m -XX:MaxPermSize=512m" I have watched observed the memory usage and general performance with Java VisualVM and have seen nothing strange. GC seems to be performing well and the memory rarely gets anywhere near the max. The server runs well, idling along at 2-5% load, serving jsp's, etc. at a reasonable speed. Without warning and with no tracks in any log (Tomcat or system) or to the console, the JVM will just go away, disappear. Sometimes, the system will run for a week, sometimes for only several hours. Initially, I thought the problem was the turbo or hyperthreading but, no, the problem persists. When the JVM goes away, the memory that it held is still being held (as seen from top) but it is nowhere near the machine physical memory. The application has been running on an older server (Dell 600SC, 32 bit Slackware, 2GB memory) for several years and, while the application will throw exceptions now and then, it never crashed the JVM. This leads me to believe the problem has something to do with the 64 bit JVM but, with errors, I can't be certain and don't know what I can do about it except go back to 32 bit. I plan to reinstall Java tonight but, it would seem if the JVM were corrupted, it simply would not run. Any ideas are welcome. I'm with Andy: the Linux OOM killer would show those symptoms. With those settings, you're not leaving a lot of memory for the OS. How much swap do you have, and does the same thing happen if you reduce the Java heap and permgen space? - Peter - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org ---
Re: JVM goes away
Konstantin, Yes, it was started using startup.sh in tomcat/bin and used the same ports (8080, 8443, 443) as the tomcat that died. The fact that the OS did not recover the memory implied to me (could be wrong, even very wrong) that the JVM just died. However, as you point out, how did the ports get freed up. So, now it looks more like I am somehow killing Tomcat because that is the only way those ports could be freed. Shouldn't I see some tracks in one of Tomcat's logs then? Thanks, Carl - Original Message - From: "Konstantin Kolinko" To: "Tomcat Users List" Sent: Monday, January 11, 2010 7:31 PM Subject: Re: JVM goes away 2010/1/12 Carl : Peter and Andy, Thanks for your quick responses. Memory: Physical - $GB Used - 2.4GB to 3.0 GB (according to top... have never seen it above 3GB) Swap - 19GB, none ever used (or, at least I have never seen any used.) The above are all from top. The 2.4GB is after the JVM just crashed (after running less than an hour after having run for five days with nary a blip) and I just restarted Tomcat (customers are running right now) so it is a little higher than normal because it has perhaps .5GB+ unrecovered from the point at which the JVM crashed. You started the new Tomcat instance using the same port numbers that were used by the old one? If so, the old one has really died, but how come that the memory was not freed? I checked dmsg but saw nothing that looked out of the ordinary. dmesg they say In OOM killer scenario it can be sometimes caused by some maintenance task scheduled with cron. But an OOM killer which does not free memory would be very much useless. Best regards, Konstantin Kolinko - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: JVM goes away
Aha, for some reason, I thought perm gen was included in the general heap so the maximum for the two combined was constrained by the 2400m I had defined for the heap. Somewhere around 2:00AM (I am US east coast), I can restart the server with the new settings. I have taken several heap dumps (using Visual JVM) and nothing looked odd. Also, I can see (from Visual JVM) that the GC runs reasonably frequently when the heap grows (from users working) but the total heap is generally under 1GB. I wonder if the sneaky little bugger, under load, just pushes to OOM and I am running so close to the edge that I don't see it. Odd though, I have forced OOM issues in the past and they always showed up in catalina.out. Thanks for your thoughts and help. Carl - Original Message - From: "Pid" To: Sent: Monday, January 11, 2010 8:07 PM Subject: Re: JVM goes away On 11/01/2010 23:06, Peter Crowther wrote: 2010/1/11 Carl: This is a new server, a Dell T110 with a Xeon 3440 processor and 4GB memory. I have turned off both the turbo mode and hyperthreading. The environment: 64 bit Slackware Linux java version "1.6.0_17" Java(TM) SE Runtime Environment (build 1.6.0_17-b04) Java HotSpot(TM) 64-Bit Server VM (build 14.3-b01, mixed mode) Tomcat: apache-tomcat-6.0.20 JAVA_OPTS="-Xms2400m -Xmx2400m -XX:PermSize=512m -XX:MaxPermSize=512m" I have watched observed the memory usage and general performance with Java VisualVM and have seen nothing strange. GC seems to be performing well and the memory rarely gets anywhere near the max. The server runs well, idling along at 2-5% load, serving jsp's, etc. at a reasonable speed. Without warning and with no tracks in any log (Tomcat or system) or to the console, the JVM will just go away, disappear. Sometimes, the system will run for a week, sometimes for only several hours. Initially, I thought the problem was the turbo or hyperthreading but, no, the problem persists. When the JVM goes away, the memory that it held is still being held (as seen from top) but it is nowhere near the machine physical memory. The application has been running on an older server (Dell 600SC, 32 bit Slackware, 2GB memory) for several years and, while the application will throw exceptions now and then, it never crashed the JVM. This leads me to believe the problem has something to do with the 64 bit JVM but, with errors, I can't be certain and don't know what I can do about it except go back to 32 bit. I plan to reinstall Java tonight but, it would seem if the JVM were corrupted, it simply would not run. Any ideas are welcome. I'm with Andy: the Linux OOM killer would show those symptoms. With those settings, you're not leaving a lot of memory for the OS. How much swap do you have, and does the same thing happen if you reduce the Java heap and permgen space? - Peter Despite later posts, I'm leaning towards agreeing with the above, based on the information provided. N.B. Maximum heap size does not equal the maximum memory a JVM can/will use. The Perm generation is in addition to the heap so you're effectively saying that the memory you want to use is 2400 + 512 (+ other stuff falling into the non-heap category). So you may be using more than 3Gb; jmap -heap will provide more information, you could regularly dump the output to file to see what's happening with the JVM. http://java.sun.com/javase/6/docs/technotes/guides/management/jconsole.html (Confession: I'm not sure I've got my head round it yet) An OOM should leave a trace somewhere on your system, it might be a single log entry saying that a given process id has been terminated. Google for specific info for your OS. p - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: JVM goes away
Perhaps not the best design but basically, yes. We have stored a fair amount of information in memory (session) to try to improve the speed of the application (so we don't go back to the disk as often.) At some point (probably never happen) I would like to take a look at the balance between storing information in session and the penalty for disk access but, like most people, just too busy right now. So, I have tried to cover this up with (relatively) cheap memory. Thanks for your thoughts. Carl - Original Message - From: "Pid" To: "Carl" Sent: Tuesday, January 12, 2010 4:41 AM Subject: Re: JVM goes away On 12/01/2010 01:30, Carl wrote: Aha, for some reason, I thought perm gen was included in the general heap so the maximum for the two combined was constrained by the 2400m I had defined for the heap. Somewhere around 2:00AM (I am US east coast), I can restart the server with the new settings. I have taken several heap dumps (using Visual JVM) and nothing looked odd. Also, I can see (from Visual JVM) that the GC runs reasonably frequently when the heap grows (from users working) but the total heap is generally under 1GB. I wonder if the sneaky little bugger, under load, just pushes to OOM and I am running so close to the edge that I don't see it. Odd though, I have forced OOM issues in the past and they always showed up in catalina.out. Does your app actually need all that memory? p Thanks for your thoughts and help. Carl - Original Message - From: "Pid" To: Sent: Monday, January 11, 2010 8:07 PM Subject: Re: JVM goes away On 11/01/2010 23:06, Peter Crowther wrote: 2010/1/11 Carl: This is a new server, a Dell T110 with a Xeon 3440 processor and 4GB memory. I have turned off both the turbo mode and hyperthreading. The environment: 64 bit Slackware Linux java version "1.6.0_17" Java(TM) SE Runtime Environment (build 1.6.0_17-b04) Java HotSpot(TM) 64-Bit Server VM (build 14.3-b01, mixed mode) Tomcat: apache-tomcat-6.0.20 JAVA_OPTS="-Xms2400m -Xmx2400m -XX:PermSize=512m -XX:MaxPermSize=512m" I have watched observed the memory usage and general performance with Java VisualVM and have seen nothing strange. GC seems to be performing well and the memory rarely gets anywhere near the max. The server runs well, idling along at 2-5% load, serving jsp's, etc. at a reasonable speed. Without warning and with no tracks in any log (Tomcat or system) or to the console, the JVM will just go away, disappear. Sometimes, the system will run for a week, sometimes for only several hours. Initially, I thought the problem was the turbo or hyperthreading but, no, the problem persists. When the JVM goes away, the memory that it held is still being held (as seen from top) but it is nowhere near the machine physical memory. The application has been running on an older server (Dell 600SC, 32 bit Slackware, 2GB memory) for several years and, while the application will throw exceptions now and then, it never crashed the JVM. This leads me to believe the problem has something to do with the 64 bit JVM but, with errors, I can't be certain and don't know what I can do about it except go back to 32 bit. I plan to reinstall Java tonight but, it would seem if the JVM were corrupted, it simply would not run. Any ideas are welcome. I'm with Andy: the Linux OOM killer would show those symptoms. With those settings, you're not leaving a lot of memory for the OS. How much swap do you have, and does the same thing happen if you reduce the Java heap and permgen space? - Peter Despite later posts, I'm leaning towards agreeing with the above, based on the information provided. N.B. Maximum heap size does not equal the maximum memory a JVM can/will use. The Perm generation is in addition to the heap so you're effectively saying that the memory you want to use is 2400 + 512 (+ other stuff falling into the non-heap category). So you may be using more than 3Gb; jmap -heap will provide more information, you could regularly dump the output to file to see what's happening with the JVM. http://java.sun.com/javase/6/docs/technotes/guides/management/jconsole.html (Confession: I'm not sure I've got my head round it yet) An OOM should leave a trace somewhere on your system, it might be a single log entry saying that a given process id has been terminated. Google for specific info for your OS. p - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: JVM goes away
One more reason why we use this much memory: we run 2-4 contexts most of the time. This was originally done to separate certain customer data while keeping the code base the same and to allow us to have a test environment exactly like the production environment. If each context requires 50-60MB for perm gen, then the Tomcat perm gen is upwards of 256MB. Also, if each context requires 200-300MB of heap, then we could require upwards heap of 500-700MB (allowing for GC, timing, etc.) I have probably allocated more heap memory than needed (due to my faulty understanding that the heap constrained perm gen, etc.) Thanks, Carl - Original Message - From: "Pid" To: "Carl" Sent: Tuesday, January 12, 2010 4:41 AM Subject: Re: JVM goes away On 12/01/2010 01:30, Carl wrote: Aha, for some reason, I thought perm gen was included in the general heap so the maximum for the two combined was constrained by the 2400m I had defined for the heap. Somewhere around 2:00AM (I am US east coast), I can restart the server with the new settings. I have taken several heap dumps (using Visual JVM) and nothing looked odd. Also, I can see (from Visual JVM) that the GC runs reasonably frequently when the heap grows (from users working) but the total heap is generally under 1GB. I wonder if the sneaky little bugger, under load, just pushes to OOM and I am running so close to the edge that I don't see it. Odd though, I have forced OOM issues in the past and they always showed up in catalina.out. Does your app actually need all that memory? p Thanks for your thoughts and help. Carl - Original Message - From: "Pid" To: Sent: Monday, January 11, 2010 8:07 PM Subject: Re: JVM goes away On 11/01/2010 23:06, Peter Crowther wrote: 2010/1/11 Carl: This is a new server, a Dell T110 with a Xeon 3440 processor and 4GB memory. I have turned off both the turbo mode and hyperthreading. The environment: 64 bit Slackware Linux java version "1.6.0_17" Java(TM) SE Runtime Environment (build 1.6.0_17-b04) Java HotSpot(TM) 64-Bit Server VM (build 14.3-b01, mixed mode) Tomcat: apache-tomcat-6.0.20 JAVA_OPTS="-Xms2400m -Xmx2400m -XX:PermSize=512m -XX:MaxPermSize=512m" I have watched observed the memory usage and general performance with Java VisualVM and have seen nothing strange. GC seems to be performing well and the memory rarely gets anywhere near the max. The server runs well, idling along at 2-5% load, serving jsp's, etc. at a reasonable speed. Without warning and with no tracks in any log (Tomcat or system) or to the console, the JVM will just go away, disappear. Sometimes, the system will run for a week, sometimes for only several hours. Initially, I thought the problem was the turbo or hyperthreading but, no, the problem persists. When the JVM goes away, the memory that it held is still being held (as seen from top) but it is nowhere near the machine physical memory. The application has been running on an older server (Dell 600SC, 32 bit Slackware, 2GB memory) for several years and, while the application will throw exceptions now and then, it never crashed the JVM. This leads me to believe the problem has something to do with the 64 bit JVM but, with errors, I can't be certain and don't know what I can do about it except go back to 32 bit. I plan to reinstall Java tonight but, it would seem if the JVM were corrupted, it simply would not run. Any ideas are welcome. I'm with Andy: the Linux OOM killer would show those symptoms. With those settings, you're not leaving a lot of memory for the OS. How much swap do you have, and does the same thing happen if you reduce the Java heap and permgen space? - Peter Despite later posts, I'm leaning towards agreeing with the above, based on the information provided. N.B. Maximum heap size does not equal the maximum memory a JVM can/will use. The Perm generation is in addition to the heap so you're effectively saying that the memory you want to use is 2400 + 512 (+ other stuff falling into the non-heap category). So you may be using more than 3Gb; jmap -heap will provide more information, you could regularly dump the output to file to see what's happening with the JVM. http://java.sun.com/javase/6/docs/technotes/guides/management/jconsole.html (Confession: I'm not sure I've got my head round it yet) An OOM should leave a trace somewhere on your system, it might be a single log entry saying that a given process id has been terminated. Google for specific info for your OS. p - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe,
Tomcat dies suddenly (was JVM goes away)
>From the original posting: This is a new server, a Dell T110 with a Xeon 3440 processor and 4GB memory. I have turned off both the turbo mode and hyperthreading. The environment: 64 bit Slackware Linux java version "1.6.0_17" Java(TM) SE Runtime Environment (build 1.6.0_17-b04) Java HotSpot(TM) 64-Bit Server VM (build 14.3-b01, mixed mode) Tomcat: apache-tomcat-6.0.20 These are the current JAVA_OPTS="-Xms1024m -Xmx1024m -XX:PermSize=368m -XX:MaxPermSize=368m" In the previous posting, I noted that I have observed the memory usage and general performance with Java VisualVM and have seen nothing strange. GC seems to be performing well and the memory rarely gets anywhere near the max. New information: I thought I was seeing GC as memory usage was going up and down but in fact it was mostly people coming onto the system and leaving it. After several hours, the memory settles to a baseline of about 375MB. Forced GC never takes it below that value and the ups and downs from the people coming onto and leaving the system also returns it to pretty much that value. The maximum memory used never was above 700MB for the entire day. The server runs well, idling along at 2-5% load, except for a quick spike during GC, serving jsp's, etc. at a reasonable speed. Without warning and with no tracks in any log (Tomcat or system) or to the console, the JVM will just go away, disappear. New information: The JVM does not just go away but somehow Tomcat shutsdown as the ports used by Tomcat are closed (pointed out by Konstantin.) Sometimes, the system will run for a week, sometimes for only several hours. Initially, I thought the problem was the turbo or hyperthreading but, no, the problem persists. When Tomcat shuts down, the memory that it held is still being held (as seen from top) but it is nowhere near the machine physical memory. The application has been running on an older server (Dell 600SC, 32 bit Slackware, 2GB memory) for several years and, while the application will throw exceptions now and then, it never crashed. This lead me to believe the problem had something to do with the 64 bit JVM but, with without seeing errors anywhere, I can't be certain and don't know what I can do about it except go back to 32 bit. New information. Last evening, I observed the heap and permGen memory usage with Visual JVM. It was running around 600MB before I forced a GC and 375MB afterward. Speed was good. Memory usage from top was 2.4GB. Five minutes later, Tomcat stopped leaving no tracks that I could find. The memory usage from top was around 2.4GB. The memory usage from Visual JVM was still showing 400MB+ although the Tomcat process was gone. I restarted Tomcat (did not reboot) so Tomcat had been shutdown gracefully enough to close the ports (8080, 8443, 443.) Tomcat stayed up for less than an hour (under light load) and stopped again. The memory used according to top was less than 3GB but I didn't get the exact number. I restarted it again (no server reboot) and it ran for the rest of the night (light load) and top was showing 3.3GB for memory this morning. I brought up a new server last night and have switched to that server for production (same Linux, JDK, server.xml, JAVA_OPTS, etc.). It would seem if the problem is with my application or the JVM, that the problem will follow me to the new server. Anyone have any ideas how I might track this problem down? Thanks, Carl
Re: Tomcat dies suddenly (was JVM goes away)
In process... thanks for the suggestion. Carl - Original Message - From: "Peter Crowther" To: "Tomcat Users List" Sent: Wednesday, January 13, 2010 8:49 AM Subject: Re: Tomcat dies suddenly (was JVM goes away) Very difficult to know what the problem is. One thing you can now do (as you've switched to another production server) is to run a memory test across the "bad" server. A T110 doesn't use error-correcting memory, as I recall, so a dodgy bit could cause problems. Give it a couple of hours with memtest86+ and you'll at least know whether you've been chasing phantoms due to a hardware error. (I'm perhaps biased - I've had memory errors on three low-end servers now) - Peter 2010/1/13 Carl : From the original posting: This is a new server, a Dell T110 with a Xeon 3440 processor and 4GB memory. I have turned off both the turbo mode and hyperthreading. The environment: 64 bit Slackware Linux java version "1.6.0_17" Java(TM) SE Runtime Environment (build 1.6.0_17-b04) Java HotSpot(TM) 64-Bit Server VM (build 14.3-b01, mixed mode) Tomcat: apache-tomcat-6.0.20 These are the current JAVA_OPTS="-Xms1024m -Xmx1024m -XX:PermSize=368m -XX:MaxPermSize=368m" In the previous posting, I noted that I have observed the memory usage and general performance with Java VisualVM and have seen nothing strange. GC seems to be performing well and the memory rarely gets anywhere near the max. New information: I thought I was seeing GC as memory usage was going up and down but in fact it was mostly people coming onto the system and leaving it. After several hours, the memory settles to a baseline of about 375MB. Forced GC never takes it below that value and the ups and downs from the people coming onto and leaving the system also returns it to pretty much that value. The maximum memory used never was above 700MB for the entire day. The server runs well, idling along at 2-5% load, except for a quick spike during GC, serving jsp's, etc. at a reasonable speed. Without warning and with no tracks in any log (Tomcat or system) or to the console, the JVM will just go away, disappear. New information: The JVM does not just go away but somehow Tomcat shutsdown as the ports used by Tomcat are closed (pointed out by Konstantin.) Sometimes, the system will run for a week, sometimes for only several hours. Initially, I thought the problem was the turbo or hyperthreading but, no, the problem persists. When Tomcat shuts down, the memory that it held is still being held (as seen from top) but it is nowhere near the machine physical memory. The application has been running on an older server (Dell 600SC, 32 bit Slackware, 2GB memory) for several years and, while the application will throw exceptions now and then, it never crashed. This lead me to believe the problem had something to do with the 64 bit JVM but, with without seeing errors anywhere, I can't be certain and don't know what I can do about it except go back to 32 bit. New information. Last evening, I observed the heap and permGen memory usage with Visual JVM. It was running around 600MB before I forced a GC and 375MB afterward. Speed was good. Memory usage from top was 2.4GB. Five minutes later, Tomcat stopped leaving no tracks that I could find. The memory usage from top was around 2.4GB. The memory usage from Visual JVM was still showing 400MB+ although the Tomcat process was gone. I restarted Tomcat (did not reboot) so Tomcat had been shutdown gracefully enough to close the ports (8080, 8443, 443.) Tomcat stayed up for less than an hour (under light load) and stopped again. The memory used according to top was less than 3GB but I didn't get the exact number. I restarted it again (no server reboot) and it ran for the rest of the night (light load) and top was showing 3.3GB for memory this morning. I brought up a new server last night and have switched to that server for production (same Linux, JDK, server.xml, JAVA_OPTS, etc.). It would seem if the problem is with my application or the JVM, that the problem will follow me to the new server. Anyone have any ideas how I might track this problem down? Thanks, Carl - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly (was JVM goes away)
Done. Thanks for the suggestion. Plan to place this machine back on the firing line after running the memory test suggested by Peter. Thanks, Carl - Original Message - From: "Paolo Santarsiero" To: "Tomcat Users List" Sent: Wednesday, January 13, 2010 8:58 AM Subject: Re: Tomcat dies suddenly (was JVM goes away) In order to monitor java memory at chrash time you can add to JAVA_OPTS these directives -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/your/tomcat/folder/memorydump.hprof In this way, if tomcat goes in out of memory, you have an image of memory (memorydump.hprof) that you can analyze by an external application like MemoryAnalyzer [ http://www.eclipse.org/mat/ ]. 2010/1/13 Carl From the original posting: This is a new server, a Dell T110 with a Xeon 3440 processor and 4GB memory. I have turned off both the turbo mode and hyperthreading. The environment: 64 bit Slackware Linux java version "1.6.0_17" Java(TM) SE Runtime Environment (build 1.6.0_17-b04) Java HotSpot(TM) 64-Bit Server VM (build 14.3-b01, mixed mode) Tomcat: apache-tomcat-6.0.20 These are the current JAVA_OPTS="-Xms1024m -Xmx1024m -XX:PermSize=368m -XX:MaxPermSize=368m" In the previous posting, I noted that I have observed the memory usage and general performance with Java VisualVM and have seen nothing strange. GC seems to be performing well and the memory rarely gets anywhere near the max. New information: I thought I was seeing GC as memory usage was going up and down but in fact it was mostly people coming onto the system and leaving it. After several hours, the memory settles to a baseline of about 375MB. Forced GC never takes it below that value and the ups and downs from the people coming onto and leaving the system also returns it to pretty much that value. The maximum memory used never was above 700MB for the entire day. The server runs well, idling along at 2-5% load, except for a quick spike during GC, serving jsp's, etc. at a reasonable speed. Without warning and with no tracks in any log (Tomcat or system) or to the console, the JVM will just go away, disappear. New information: The JVM does not just go away but somehow Tomcat shutsdown as the ports used by Tomcat are closed (pointed out by Konstantin.) Sometimes, the system will run for a week, sometimes for only several hours. Initially, I thought the problem was the turbo or hyperthreading but, no, the problem persists. When Tomcat shuts down, the memory that it held is still being held (as seen from top) but it is nowhere near the machine physical memory. The application has been running on an older server (Dell 600SC, 32 bit Slackware, 2GB memory) for several years and, while the application will throw exceptions now and then, it never crashed. This lead me to believe the problem had something to do with the 64 bit JVM but, with without seeing errors anywhere, I can't be certain and don't know what I can do about it except go back to 32 bit. New information. Last evening, I observed the heap and permGen memory usage with Visual JVM. It was running around 600MB before I forced a GC and 375MB afterward. Speed was good. Memory usage from top was 2.4GB. Five minutes later, Tomcat stopped leaving no tracks that I could find. The memory usage from top was around 2.4GB. The memory usage from Visual JVM was still showing 400MB+ although the Tomcat process was gone. I restarted Tomcat (did not reboot) so Tomcat had been shutdown gracefully enough to close the ports (8080, 8443, 443.) Tomcat stayed up for less than an hour (under light load) and stopped again. The memory used according to top was less than 3GB but I didn't get the exact number. I restarted it again (no server reboot) and it ran for the rest of the night (light load) and top was showing 3.3GB for memory this morning. I brought up a new server last night and have switched to that server for production (same Linux, JDK, server.xml, JAVA_OPTS, etc.). It would seem if the problem is with my application or the JVM, that the problem will follow me to the new server. Anyone have any ideas how I might track this problem down? Thanks, Carl - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly (was JVM goes away)
Chris,
Carl: when the JVM "dies" and you use top to see free memory, does it
say that 2.4GB of memory is in use by a particular process,
It shows the 2.4GB as 'Used' but does not show it attached to any process
(remember that the Tomcat process has disappeared... ps aux | grep tomcat
yields nothing.) My observation is that the server has 500MB 'used' when it
starts and moves to 2.4GB after Tomcat is started. However, the server does
not appear to reclaim the memory after the process dies as the 'used' stays
right at 2.4GB. Visual LVM continues to report that the now dead Tomcat
instance is still holding onto the memory but I am not certain whether this
reflects some variable(s) set in Visual JVM or the actual memory something is
still holding onto.
or does it
just appear that the memory is not "available"?
The 2.4GB is just shown as 'Used' by top.
If it's by a particular
process, which one?
No process but I expected that as the Tomcat process (ps aux | grep tomcat) no
longer exists (after the 'crash'.)
The JVM process ("/usr/bin/java" or whatever) either
does or does not exist, and if it does not exist, is it retaining
memory?
I don't know how I could tell if the Tomcat java process/JVM was holding onto
the memory if the process no longer exists.
If the Tomcat connectors have shut down (thereby releasing the
TCP/IP ports), but not the java process, then there should be some
indication in catalina.out
No indication at all... just comes to a stop. (I had a problem a while ago
with not properly releasing database connections an I still have a good deal of
stuff going to catalina.out (because I have been too busy to comment out the
debugging messages.)
that the connectors have been shut down
explicitly.
The whole thing sounds weird. :(
That has been a good deal of my frustration... I thought it would leave some
tracks somewhere.
All thoughts and ideas are appreciated.
Thanks,
Carl
- Original Message -
From: "Christopher Schultz"
To: "Tomcat Users List"
Sent: Wednesday, January 13, 2010 11:50 AM
Subject: Re: Tomcat dies suddenly (was JVM goes away)
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA1
>
> Peter,
>
> On 1/13/2010 8:49 AM, Peter Crowther wrote:
>> Very difficult to know what the problem is. One thing you can now do
>> (as you've switched to another production server) is to run a memory
>> test across the "bad" server.
>
> Usually, I would agree that physical memory problems are likely to be a
> problem, but every time I've had a physical memory problem (much more
> common than I'd like to admit!), the JVM has crashed in a more classic
> way: that is, with an hs_log file and almost always with a SIGSEGV,
> rather than this phantom thing described by Carl.
>
> The Linux OOM killer might be a suspect, except that the process is
> apparently not dying, which is very strange.
>
> Carl: when the JVM "dies" and you use top to see free memory, does it
> say that 2.4GB of memory is in use by a particular process, or does it
> just appear that the memory is not "available"? If it's by a particular
> process, which one? The JVM process ("/usr/bin/java" or whatever) either
> does or does not exist, and if it does not exist, is it retaining
> memory? If the Tomcat connectors have shut down (thereby releasing the
> TCP/IP ports), but not the java process, then there should be some
> indication in catalina.out that the connectors have been shut down
> explicitly.
>
> The whole thing sounds weird. :(
>
> - -chris
> -BEGIN PGP SIGNATURE-
> Version: GnuPG v1.4.10 (MingW32)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iEYEARECAAYFAktN+ecACgkQ9CaO5/Lv0PAZzgCgsZaU16RGcs5pgsgzgLVX7q0W
> 8xcAnRUb1Zl+0PY6+Umk8nQAEagfl/Su
> =RA9e
> -END PGP SIGNATURE-
>
> -
> To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
> For additional commands, e-mail: users-h...@tomcat.apache.org
>
>
Re: Tomcat dies suddenly (was JVM goes away)
Peter, The memTest is still running but clean so far. Thanks, Carl - Original Message - From: "Peter Crowther" To: "Tomcat Users List" Sent: Wednesday, January 13, 2010 12:00 PM Subject: Re: Tomcat dies suddenly (was JVM goes away) 2010/1/13 Christopher Schultz : On 1/13/2010 8:49 AM, Peter Crowther wrote: Very difficult to know what the problem is. One thing you can now do (as you've switched to another production server) is to run a memory test across the "bad" server. Usually, I would agree that physical memory problems are likely to be a problem, but every time I've had a physical memory problem (much more common than I'd like to admit!), the JVM has crashed in a more classic way: that is, with an hs_log file and almost always with a SIGSEGV, rather than this phantom thing described by Carl. The Linux OOM killer might be a suspect, except that the process is apparently not dying, which is very strange. [...] The whole thing sounds weird. :( Oh, I agree entirely - usually something will turn a reference bad and you'll get a memory access somewhere off in hyperspace during a GC. But it's an easy thing to check, and there is an (admittedly small) possibility of seeing these symptoms. Heck, with hardware errors there's a small probability of seeing pretty much *any* symptoms. - Peter - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly (was JVM goes away)
David, Will do... thanks for the heads up. Carl - Original Message - From: "David kerber" To: "Tomcat Users List" Sent: Wednesday, January 13, 2010 1:17 PM Subject: Re: Tomcat dies suddenly (was JVM goes away) Carl wrote: Peter, The memTest is still running but clean so far. Make sure you let it run for quite a while. I've had memory failures show up as late as 11 passes into a test run. D - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly (was JVM goes away)
David, What do you use for your mem testing? I am using the memTest suggested by Peter... after six tests, it still shows all memory is OK. Probably call Dell this morning. TIA, Carl - Original Message - From: "David Kerber" To: "Tomcat Users List" Sent: Wednesday, January 13, 2010 6:26 PM Subject: Re: Tomcat dies suddenly (was JVM goes away) Peter Crowther wrote: 2010/1/13 David kerber : Make sure you let it run for quite a while. I've had memory failures show up as late as 11 passes into a test run. That's dedication - I usually end up stopping it after a couple of runs. Thanks David, I've learned something! - Peter - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org I just start it and let it go for a day or four, until I get around to checking it again. I try to get at least 24 hours of memtest testing on new machines, and 48 hrs on used/older ones. D - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly (was JVM goes away)
David, I am such a dufuss... didn't even notice it cycled after it finished a test. After almost 24 hours, showing no failures. Time to call Dell. Thanks, Carl - Original Message - From: "David kerber" To: "Tomcat Users List" Sent: Thursday, January 14, 2010 8:48 AM Subject: Re: Tomcat dies suddenly (was JVM goes away) Memtest86, which I believe is the same one Peter suggested (or at least a variation of it). It just loops continuously until stopped. Carl wrote: David, What do you use for your mem testing? I am using the memTest suggested by Peter... after six tests, it still shows all memory is OK. Probably call Dell this morning. TIA, Carl - Original Message - From: "David Kerber" To: "Tomcat Users List" Sent: Wednesday, January 13, 2010 6:26 PM Subject: Re: Tomcat dies suddenly (was JVM goes away) Peter Crowther wrote: 2010/1/13 David kerber : Make sure you let it run for quite a while. I've had memory failures show up as late as 11 passes into a test run. That's dedication - I usually end up stopping it after a couple of runs. Thanks David, I've learned something! - Peter - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org I just start it and let it go for a day or four, until I get around to checking it again. I try to get at least 24 hours of memtest testing on new machines, and 48 hrs on used/older ones. D - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly (was JVM goes away)
Yes, Slackware, version 13, 64bit. I had done this manually (looked through each log for any evidence of a failure) but had not done it your automated way. Just did your automated way and it found nothing (I included all the messages logs)... bummer. The server I brought up Tuesday is using the same Slackware, Tomcat, JDK. This server is a Dell T105 (it was destined to be used in a smaller setting) which has an AMD processor instead of the Xeon. This server is a little slower (the users don't notice) and has yet to have any problem. Of course, the T110 ran for a week before it had a problem. Thanks, Carl - Original Message - From: "Pid" To: Sent: Thursday, January 14, 2010 9:55 AM Subject: Re: Tomcat dies suddenly (was JVM goes away) On 14/01/2010 14:36, Carl wrote: David, I am such a dufuss... didn't even notice it cycled after it finished a test. After almost 24 hours, showing no failures. Time to call Dell. If there's no memory hardware issue, then we're back to software. You were on linux right? Did you search the OS logs for evidence of an OOM kill? cat /var/log/messages | grep --ignore-case "killed process" p - Original Message - From: "David kerber" To: "Tomcat Users List" Sent: Thursday, January 14, 2010 8:48 AM Subject: Re: Tomcat dies suddenly (was JVM goes away) Memtest86, which I believe is the same one Peter suggested (or at least a variation of it). It just loops continuously until stopped. Carl wrote: David, What do you use for your mem testing? I am using the memTest suggested by Peter... after six tests, it still shows all memory is OK. Probably call Dell this morning. TIA, Carl - Original Message - From: "David Kerber" To: "Tomcat Users List" Sent: Wednesday, January 13, 2010 6:26 PM Subject: Re: Tomcat dies suddenly (was JVM goes away) Peter Crowther wrote: 2010/1/13 David kerber : Make sure you let it run for quite a while. I've had memory failures show up as late as 11 passes into a test run. That's dedication - I usually end up stopping it after a couple of runs. Thanks David, I've learned something! - Peter - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Tomcat dies suddenly
6-7 weeks ago, we built up some new servers and started having sudden failures... Tomcat just stops with no error message, no system error messages, nothing that I have been able to find so far. To refresh everyone's memory, this is a new server, a Dell T110 with a Xeon 3440 processor and 4GB memory. I have turned off both the turbo mode and hyperthreading. The environment: 64 bit Slackware Linux java version "1.6.0_17" Java(TM) SE Runtime Environment (build 1.6.0_17-b04) Java HotSpot(TM) 64-Bit Server VM (build 14.3-b01, mixed mode) Tomcat: apache-tomcat-6.0.20 These are the current JAVA_OPTS="-Xms1024m -Xmx1024m -XX:PermSize=368m -XX:MaxPermSize=368m" I have observed the memory usage and general performance with Java VisualVM and have seen nothing strange. I thought I was seeing GC as memory usage was going up and down but in fact it was mostly people coming onto the system and leaving it. After several hours, the memory settles to a baseline of about 375MB. Forced GC never takes it below that value and the ups and downs from the people coming onto and leaving the system also returns it to pretty much that value. The maximum memory used never was above 700MB for the entire day. The server runs well, idling along at 2-5% load, except for a quick spike during GC, serving jsp's, etc. at a reasonable speed. Without warning and with no tracks in any log (Tomcat or system) or to the console, Tomcat just shuts down. I can usually simply restart it as the ports used by Tomcat are closed... today, I needed to run shutdown.sh before I could run startup.sh (startup.sh gave no errors but would not start Tomcat until I ran shutdown.sh and that process put nothing in the logs... this is the first time this has happened.) Sometimes, the system will run for a week, sometimes for only several hours, sometimes only for a few minutes. Today, it ran until about 1:00PM and has been down four times since then. The failure (Tomcat shutting down) is not always the same place in the code (I have some debugging messages going to catalina.out.) Load does not seem to make a difference. I have tried another sever (Dell T105, AMD processor, 6GB memory) and have observed the same results. I have run memTest86 on the T110 for about 30 hours and it showed nothing. I rebuilt the T110 with SUSE linux, Java 1.6.18 and Tomcat 6.0.24... it lasted 15 minutes. I have used the same server.xml on all the installs: When Tomcat shuts down, the memory that it was using seems to still be held (as seen from top) but it is nowhere near the machine physical memory. The application has been running on an older server (Dell 600SC, 32 bit Slackware, 2GB memory) for several years and, while the application will throw exceptions now and then, it never crashed. This lead me to believe the problem had something to do with the 64 bit JVM but, with without seeing errors anywhere, I can't be certain and don't know what I can do about it except go back to 32 bit. One time, I observed the heap and permGen memory usage with Visual JVM. It was running around 600MB before I forced a GC and 375MB afterward. Speed was good. Memory usage from top was 2.4GB. Five minutes later, Tomcat stopped leaving no tracks that I could find. The memory usage from top was around 2.4GB. The memory usage from Visual JVM was still showing 400MB+ although the Tomcat process was gone. I restarted Tomcat (did not reboot) so Tomcat had been shutdown gracefully enough to close the ports (8080, 8443, 443.) Tomcat stayed up for less than an hour (under light load) and stopped again. The memory used according to top was less than 3GB but I didn't get the exact number. I restarted it again (no server reboot) and it ran for the rest of the night (light load) and top was showing 3.3GB for memory in the morning. Anyone have any ideas how I might track this problem down? Thanks, Carl
Re: Tomcat dies suddenly
Mark, Thanks for your quick reply. Can I run the 32 bit JVM on the 64 bit linux (I think I can but just wanted to confirm)? Thanks, Carl - Original Message - From: "Mark Thomas" To: "Tomcat Users List" Sent: Friday, January 29, 2010 3:18 PM Subject: Re: Tomcat dies suddenly On 29/01/2010 19:54, Carl wrote: 6-7 weeks ago, we built up some new servers and started having sudden failures... Tomcat just stops with no error message, no system error messages, nothing that I have been able to find so far. To refresh everyone's memory, this is a new server, a Dell T110 with a Xeon 3440 processor and 4GB memory. I have turned off both the turbo mode and hyperthreading. The environment: 64 bit Slackware Linux java version "1.6.0_17" Java(TM) SE Runtime Environment (build 1.6.0_17-b04) Java HotSpot(TM) 64-Bit Server VM (build 14.3-b01, mixed mode) Tomcat: apache-tomcat-6.0.20 These are the current JAVA_OPTS="-Xms1024m -Xmx1024m -XX:PermSize=368m -XX:MaxPermSize=368m" I have observed the memory usage and general performance with Java VisualVM and have seen nothing strange. I thought I was seeing GC as memory usage was going up and down but in fact it was mostly people coming onto the system and leaving it. After several hours, the memory settles to a baseline of about 375MB. Forced GC never takes it below that value and the ups and downs from the people coming onto and leaving the system also returns it to pretty much that value. The maximum memory used never was above 700MB for the entire day. The server runs well, idling along at 2-5% load, except for a quick spike during GC, serving jsp's, etc. at a reasonable speed. Without warning and with no tracks in any log (Tomcat or system) or to the console, Tomcat just shuts down. I can usually simply restart it as the ports used by Tomcat are closed... today, I needed to run shutdown.sh before I could run startup.sh (startup.sh gave no errors but would not start Tomcat until I ran shutdown.sh and that process put nothing in the logs... this is the first time this has happened.) Sometimes, the system will run for a week, sometimes for only several hours, sometimes only for a few minutes. Today, it ran until about 1:00PM and has been down four times since then. The failure (Tomcat shutting down) is not always the same place in the code (I have some debugging messages going to catalina.out.) Load does not seem to make a difference. I have tried another sever (Dell T105, AMD processor, 6GB memory) and have observed the same results. I have run memTest86 on the T110 for about 30 hours and it showed nothing. I rebuilt the T110 with SUSE linux, Java 1.6.18 and Tomcat 6.0.24... it lasted 15 minutes. I have used the same server.xml on all the installs: SSLEngine="on" /> /> className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener" /> keystoreFile="/usr/local/certs/tomcat_keystore.ks" keystorePass="jellybean"/> keystoreFile="/usr/local/certs/tomcat_keystore.ks" keystorePass="jellybean"/> When Tomcat shuts down, the memory that it was using seems to still be held (as seen from top) but it is nowhere near the machine physical memory. The application has been running on an older server (Dell 600SC, 32 bit Slackware, 2GB memory) for several years and, while the application will throw exceptions now and then, it never crashed. This lead me to believe the problem had something to do with the 64 bit JVM but, with without seeing errors anywhere, I can't be certain and don't know what I can do about it except go back to 32 bit. Given the circumstances, that seems like a reasonable next step. Anyone have any ideas how I might track this problem down? It sounds like it is going to be trial an error. The closest I have seen to this was a web application that hung on Tomcat start. It appeared to be one or more gc bugs in the 64-bit JVM. I had one set options gc options that worked on Solaris and failed on linux and another set that worked on linux but failed on Solaris. I never did find a set that worked on both. When Tomcat shuts down, I assume there is no java process left running. i.e. the process dies rather than hangs. Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
OK, that's what I thought. Died 8 times so far this afternoon... different than the past although we have unusally heavy volume this afternoon. Don't understand but 32 bit here I come. Thanks, Carl - Original Message - From: "Caldarale, Charles R" To: "Tomcat Users List" Sent: Friday, January 29, 2010 3:40 PM Subject: RE: Tomcat dies suddenly From: Carl [mailto:c...@etrak-plus.com] Subject: Re: Tomcat dies suddenly Can I run the 32 bit JVM on the 64 bit linux Yes (but not the other way around, of course). - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Slackware 64
Has anyone successfully used Tomcat with Slackware 64 and Sun's 64 bit Java? If so, what were your Java options, particularly the ones relating to GC? TIA, Carl
Re: Slackware 64
Chuck, Thanks for the reply. I was attempting to see if anyone had problems with 64 bit Java and 64 bit Slackware like I have been having and figured out a work around. I agree the settings are heavily dependent on the application. We have plenty of memory. It was suggested on a Java list that I add 'System.addShutdownHook(Thread)' to see if I can distinguish between exit() and halt() or some form of JVM abort. I should get to that later today but am looking for all the ammunition I can get to beat this problem. Thanks, Carl - Original Message - From: "Caldarale, Charles R" To: "Tomcat Users List" Sent: Saturday, January 30, 2010 10:19 AM Subject: RE: Slackware 64 From: Carl [mailto:c...@etrak-plus.com] Subject: Slackware 64 Has anyone successfully used Tomcat with Slackware 64 and Sun's 64 bit Java? If so, what were your Java options, particularly the ones relating to GC? Appropriate heap sizing, choice of GC algorithm, and other GC-related parameters are almost entirely dependent on the characteristics of *your* webapps and processing requirements; unless someone has an identical set of webapps and identical load patterns, anyone else's settings are irrelevant. You will need to insure that the total active process space (of which the Java heap is only a part) does not exceed the available RAM on the system, or you'll get into page thrashing pretty quickly. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
George, Thanks for your reply. I will check that IPV6 is disabled... can't remember for certain. I have had the problem on both openSuse and Slackware which has lead me to believe the problem is in the JVM. I guess I could try compiling the JVM from source on the target machine. I ran memTest86 for 30 hours on the first server I encountered the problem with (the first server with Slackware and 64 bit java that I brought up) and nothing showed up. I then, redid that machine with openSuse (thought the problem might be Slackware) but openSuse failed as quickly. Thanks, Carl - Original Message - From: "George Sexton" To: "'Tomcat Users List'" Sent: Saturday, January 30, 2010 11:46 AM Subject: RE: Tomcat dies suddenly I've had this happen. Finally, I got a stack trace. In my case, there appears to be a bug in GLIBC, so when a reverse IP address lookup is done AND there is an IPV6 entry, it causes a problem. The solution appeared to be disabling IPV6 since I'm not using it. This is on OpenSuSE. It would be worth checking. You might also run MemTest86+ on it if you haven't already. George Sexton MH Software, Inc. http://www.mhsoftware.com/ Voice: 303 438 9585 -Original Message- From: Carl [mailto:c...@etrak-plus.com] Sent: Friday, January 29, 2010 12:54 PM To: Tomcat Users List Subject: Tomcat dies suddenly 6-7 weeks ago, we built up some new servers and started having sudden failures... Tomcat just stops with no error message, no system error messages, nothing that I have been able to find so far. To refresh everyone's memory, this is a new server, a Dell T110 with a Xeon 3440 processor and 4GB memory. I have turned off both the turbo mode and hyperthreading. The environment: 64 bit Slackware Linux java version "1.6.0_17" Java(TM) SE Runtime Environment (build 1.6.0_17-b04) Java HotSpot(TM) 64-Bit Server VM (build 14.3-b01, mixed mode) Tomcat: apache-tomcat-6.0.20 These are the current JAVA_OPTS="-Xms1024m -Xmx1024m -XX:PermSize=368m -XX:MaxPermSize=368m" I have observed the memory usage and general performance with Java VisualVM and have seen nothing strange. I thought I was seeing GC as memory usage was going up and down but in fact it was mostly people coming onto the system and leaving it. After several hours, the memory settles to a baseline of about 375MB. Forced GC never takes it below that value and the ups and downs from the people coming onto and leaving the system also returns it to pretty much that value. The maximum memory used never was above 700MB for the entire day. The server runs well, idling along at 2-5% load, except for a quick spike during GC, serving jsp's, etc. at a reasonable speed. Without warning and with no tracks in any log (Tomcat or system) or to the console, Tomcat just shuts down. I can usually simply restart it as the ports used by Tomcat are closed... today, I needed to run shutdown.sh before I could run startup.sh (startup.sh gave no errors but would not start Tomcat until I ran shutdown.sh and that process put nothing in the logs... this is the first time this has happened.) Sometimes, the system will run for a week, sometimes for only several hours, sometimes only for a few minutes. Today, it ran until about 1:00PM and has been down four times since then. The failure (Tomcat shutting down) is not always the same place in the code (I have some debugging messages going to catalina.out.) Load does not seem to make a difference. I have tried another sever (Dell T105, AMD processor, 6GB memory) and have observed the same results. I have run memTest86 on the T110 for about 30 hours and it showed nothing. I rebuilt the T110 with SUSE linux, Java 1.6.18 and Tomcat 6.0.24... it lasted 15 minutes. I have used the same server.xml on all the installs: When Tomcat shuts down, the memory that it was using seems to still be held (as seen from top) but it is nowhere near the machine physical memory. The application has been running on an older server (Dell 600SC, 32 bit Slackware, 2GB memory) for several years and, while the application will throw exceptions now and then, it never crashed. This lead me to believe the problem had something to do with the 64 bit JVM but, with without seeing errors anywhere, I can't be certain and don't know what I can do about it except go back to 32 bit. One time, I observed the heap and permGen memory usage with Visual JVM. It was running around 600MB before I forced a GC and 375MB afterward. Speed was good. Memory usage from top was 2.4GB. Five minutes later, Tomcat stopped leaving no tracks that I could find. The memory usage from top was around 2.4GB. The memory usage from Visual JVM was still showing 400MB+ although the Tomcat process was gone. I restarted Tomcat (did not reboot) s
Re: Tomcat dies suddenly
Chris, There appears to be some bling that openSuse has that Slackware doesn't but otherwise they seem to be the same. I tried openSuse because 1) Dell stated they had tested their servers with that distribution and 2) Sun stated that the Java distribution had been tested using openSuse. My thought was there might be a subtle difference that could either explain what I was seeing or give me a clue to what was happening. No luck with either. However, I have found a couple of things that might account for the failures: 1. Slackware, by default, installs a version of Java (1.6.0_16.) I knew this but apparently neglected to remove it using the package manager (I noticed it was still installed because nepomuk was starting a java process which was showing up in VisualJVM.) So, there were two versions of Java running, one from /usr/bin/java and one from /usr/local/java. Don't remember what JAVA_HOME was set to and not certain how Tomcat actually started but both JAVA_HOME and PATH are correct now. 2. One of the servers had no swap. Should not have been a problem since I don't recall ever getting to the maximum physical memory (4GB) on that machine. However, I suspect if the system ever hit the physical memory limit, we could well see the OOM killer. I did restrict the heap (yesterday) to 500MB and the garbage collection was hit frequently and performed properly so it seems my hypothesis that it might be the GC process is false. So, the monitoring continues. Thanks, Carl - Original Message - From: "Christopher Schultz" To: "Tomcat Users List" Sent: Monday, February 01, 2010 4:31 PM Subject: Re: Tomcat dies suddenly -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Carl, On 1/30/2010 12:32 PM, Carl wrote: I have had the problem on both openSuse and Slackware which has lead me to believe the problem is in the JVM. ...or the kernel. Is there an appreciable difference between OpenSUSE and Slackware in your environment? They may be on slightly different kernel patch levels, or glibc levels, etc. but it always boils down to the Linux kernel and GNU libc. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAktnSEkACgkQ9CaO5/Lv0PBkqQCeOzFkYmDqCtMQpfmB7q9nZrBD a+IAoKLzHfv6T16FMZrC84aO6v4d7P5p =7wgJ -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Dan, Does Tomcat do an orderly shutdown? Normally, when Tomcat shuts down, it shows the process in catalina.out. When this issue happens, there are no entries in catalina.out (or any other log that I could locate.) On the other hand, it seems to clean up the ports because I can just start Tomcat (with startup.sh). So, I am not certain whether or not it is an 'orderly' shutdown. Does Tomcat appear to freeze (stop answering new requests) Using ps aux | grep Tomcat shows no process. Also, I can just restart Tomcat with startup.sh which implies the ports are cleared which seems to mean it is not 'frozen'. Does the JVM hosting Tomcat crash? I do not think so because that should produce an entry in a file or a file that shows the cause (somewhat) at the time the JVM crashed. I have not been able to find any such entry or file. You can use these options on your JVM in your tomcat start sequence to control where the JVM crash log will be written (And to force one for an out of memory error) I have added these switches to the startup (catalina.sh.) Thanks for the suggestions. Carl - Original Message - From: "Dan Armbrust" To: "Tomcat Users List" Sent: Tuesday, February 02, 2010 1:36 PM Subject: Re: Tomcat dies suddenly Carl, Maybe I missed it, but you still haven't clarified what you mean by tomcat stops. Does Tomcat do an orderly shutdown? Does Tomcat appear to freeze (stop answering new requests) Does the JVM hosting Tomcat crash? Assuming you have a JVM crash issue here... you should try other JVMs (32 bit, and/or other versions). You should try different JVM switches. And you should find the JVM crash log file - you should have one somewhere. You can use these options on your JVM in your tomcat start sequence to control where the JVM crash log will be written (And to force one for an out of memory error) -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/logFolder Dan - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: tomcat memory usage
Dan, These are the Javs opts currently set in catalina.sh: JAVA_OPTS="-Xms512m -Xmx512m -XX:PermSize=384m -XX:MaxPermSize=384m -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/local/tomcat/logs" Do you see anything dangerous/wrong/not good? Thanks, Carl - Original Message - From: "Dan Armbrust" To: "Tomcat Users List" ; Sent: Tuesday, February 02, 2010 2:07 PM Subject: Re: tomcat memory usage FYI - look out for this WRT MaxNewSize and NewRatio: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6862534 - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
George, I scanned the code for 'InetAddress.getHostName' and didn't find any instances (that is what I thought but wanted to be certain.) It is frustrating how it will run very well for days and then suddenly fail every 15 minutes. Last Friday was a 'fail every 15 minutes' day. The only thing that was different last Friday was it was the last day of registration for several organizations and, while the total number of users was still around 80, the amount of work they were doing was substantially more than usual. The only server I was using that day was a Dell T105. When I switched back to an older server (P4 dinosaur running 32 bit 5.5 java and an old Tomcat), it never failed. On the weekend, I was load testing both the rebuilt T110 and the T105 that was running Friday and found that the T105 had zero swap although it has 6GB of memory and I don't think it ever got close to that amount used (I also now rebuilt that server.) At any rate, no swap could, if I understand it, provoke the OOM killer which would give the type of failure we were seeing. The load testing showed nothing. I could overload the server, run it out of memory, etc. and it still behaved as expected (a little weird at times but never the type of failure we were seeing.) I have now added some java options: -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/local/tomcat/logs and am monitoring it on a continual basis using VisualJVM (which shows a lot of information but has never shown anything at the time the system fails.) So far, the system is running great: quick response, memory in the 300-400MB range, GC working as it should, plenty of free memory in the machine (not close to using swap) and no dying. If you can think of any rock I might look under, I would like to hear about it. TIA, Carl - Original Message - From: "George Sexton" To: "'Tomcat Users List'" Sent: Wednesday, February 03, 2010 1:55 PM Subject: RE: Tomcat dies suddenly -Original Message- From: Carl [mailto:c...@etrak-plus.com] Sent: Saturday, January 30, 2010 10:32 AM To: Tomcat Users List Subject: Re: Tomcat dies suddenly George, Thanks for your reply. I will check that IPV6 is disabled... can't remember for certain. I have had the problem on both openSuse and Slackware which has lead me to believe the problem is in the JVM. I guess I could try compiling the JVM from source on the target machine. Are you doing any kind of reverse DNS lookup? E.G. InetAddress.getHostName()? I've really seen this problem more than once with Linux. On both 32-bit and 64-bit systems. What makes it crazy is that the system might run for weeks, but then a certain user can trigger the error and it will happen twice in an hour. If the defect is in GLIBC, it would happen on both distributions. Really, if you're using InetAddress.getHostName(), this could be a problem. I ran memTest86 for 30 hours on the first server I encountered the problem with (the first server with Slackware and 64 bit java that I brought up) and nothing showed up. I then, redid that machine with openSuse (thought the problem might be Slackware) but openSuse failed as quickly. Thanks, Carl - Original Message - From: "George Sexton" To: "'Tomcat Users List'" Sent: Saturday, January 30, 2010 11:46 AM Subject: RE: Tomcat dies suddenly > I've had this happen. Finally, I got a stack trace. In my case, there > appears to be a bug in GLIBC, so when a reverse IP address lookup is done > AND there is an IPV6 entry, it causes a problem. > > The solution appeared to be disabling IPV6 since I'm not using it. This is > on OpenSuSE. > > It would be worth checking. > > You might also run MemTest86+ on it if you haven't already. > > George Sexton > MH Software, Inc. > http://www.mhsoftware.com/ > Voice: 303 438 9585 > > >> -Original Message- >> From: Carl [mailto:c...@etrak-plus.com] >> Sent: Friday, January 29, 2010 12:54 PM >> To: Tomcat Users List >> Subject: Tomcat dies suddenly >> >> 6-7 weeks ago, we built up some new servers and started having sudden >> failures... Tomcat just stops with no error message, no system error >> messages, nothing that I have been able to find so far. >> >> To refresh everyone's memory, this is a new server, a Dell T110 with a >> Xeon 3440 processor and 4GB memory. I have turned off both the turbo >> mode and hyperthreading. >> >> The environment: >> >> 64 bit Slackware Linux >> >> java version "1.6.0_17" >> Java(TM) SE Runtime Environment (build 1.6.0_17-b04) >> Java HotSpot(TM) 64-Bit Server VM
Re: Tomcat dies suddenly
Chuck, Thanks for your reply. I only added those options recently (over the weekend) so they were not there at the time of the last failure. Without them, the GC pauses are insignificant. I added the print options in the hope that I could see something right at the time of failure. The concurrent options can probably go as I don't see they are of any value. Thanks, Carl - Original Message - From: "Caldarale, Charles R" To: "Tomcat Users List" Sent: Wednesday, February 03, 2010 3:28 PM Subject: RE: Tomcat dies suddenly From: Carl [mailto:c...@etrak-plus.com] Subject: Re: Tomcat dies suddenly I have now added some java options: -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/local/tomcat/logs If you can think of any rock I might look under I would try running without concurrent GC and the PrintGC options and see if the failures disappear. (Obviously, that will take some time before any confidence can be had.) Hopefully, the resulting GC pause times won't be a significant problem. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Chuck, Yes, it seems to point in that direction as we have ruled out most of the rest of the universe. I am hoping I can see failure at the point of failure with the heap dump options -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/local/tomcat/logs but I have no experience with them so I am not certain what I might see. On Saturday, when I was stress testing it (JMeter), I was able to run it out of memory but it still didn't die, just kept logging that it was out of memory. Not quite square one because we have eliminated some possibilities. Thanks, Carl - Original Message - From: "Caldarale, Charles R" To: "Tomcat Users List" Sent: Wednesday, February 03, 2010 4:04 PM Subject: RE: Tomcat dies suddenly From: Carl [mailto:c...@etrak-plus.com] Subject: Re: Tomcat dies suddenly I added the print options in the hope that I could see something right at the time of failure. Since the failures occurred before the print options were used, I guess that rules out any possibility of the OS getting upset at the JVM calling for timing information. Back to square one, where the most likely culprit is still the Linux OOM killer. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
George, I did and found one instance but it is running on an applet on the client side so I don't think it should impact the server. Thanks for the thoughts. Carl - Original Message - From: "George Sexton" To: "'Tomcat Users List'" Sent: Wednesday, February 03, 2010 4:09 PM Subject: RE: Tomcat dies suddenly You might want to search for getHostName by itself since it's an instance method and the variable name for the object would most certainly not be InetAddress. George Sexton MH Software, Inc. http://www.mhsoftware.com/ Voice: 303 438 9585 -Original Message- From: Carl [mailto:c...@etrak-plus.com] Sent: Wednesday, February 03, 2010 12:55 PM To: Tomcat Users List Subject: Re: Tomcat dies suddenly George, I scanned the code for 'InetAddress.getHostName' and didn't find any instances (that is what I thought but wanted to be certain.) It is frustrating how it will run very well for days and then suddenly fail every 15 minutes. Last Friday was a 'fail every 15 minutes' day. The only thing that was different last Friday was it was the last day of registration for several organizations and, while the total number of users was still around 80, the amount of work they were doing was substantially more than usual. The only server I was using that day was a Dell T105. When I switched back to an older server (P4 dinosaur running 32 bit 5.5 java and an old Tomcat), it never failed. On the weekend, I was load testing both the rebuilt T110 and the T105 that was running Friday and found that the T105 had zero swap although it has 6GB of memory and I don't think it ever got close to that amount used (I also now rebuilt that server.) At any rate, no swap could, if I understand it, provoke the OOM killer which would give the type of failure we were seeing. The load testing showed nothing. I could overload the server, run it out of memory, etc. and it still behaved as expected (a little weird at times but never the type of failure we were seeing.) I have now added some java options: -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+PrintGCDetails - XX:+PrintGCTimeStamps -XX:+HeapDumpOnOutOfMemoryError - XX:HeapDumpPath=/usr/local/tomcat/logs and am monitoring it on a continual basis using VisualJVM (which shows a lot of information but has never shown anything at the time the system fails.) So far, the system is running great: quick response, memory in the 300- 400MB range, GC working as it should, plenty of free memory in the machine (not close to using swap) and no dying. If you can think of any rock I might look under, I would like to hear about it. TIA, Carl - Original Message - From: "George Sexton" To: "'Tomcat Users List'" Sent: Wednesday, February 03, 2010 1:55 PM Subject: RE: Tomcat dies suddenly >> -Original Message- >> From: Carl [mailto:c...@etrak-plus.com] >> Sent: Saturday, January 30, 2010 10:32 AM >> To: Tomcat Users List >> Subject: Re: Tomcat dies suddenly >> >> George, >> >> Thanks for your reply. >> >> I will check that IPV6 is disabled... can't remember for certain. >> >> I have had the problem on both openSuse and Slackware which has lead me >> to >> believe the problem is in the JVM. I guess I could try compiling the >> JVM >> from source on the target machine. > > Are you doing any kind of reverse DNS lookup? E.G. > InetAddress.getHostName()? > > I've really seen this problem more than once with Linux. On both 32- bit > and > 64-bit systems. > > What makes it crazy is that the system might run for weeks, but then a > certain user can trigger the error and it will happen twice in an hour. > > If the defect is in GLIBC, it would happen on both distributions. Really, > if > you're using InetAddress.getHostName(), this could be a problem. > > > >> >> I ran memTest86 for 30 hours on the first server I encountered the >> problem >> with (the first server with Slackware and 64 bit java that I brought >> up) and >> nothing showed up. I then, redid that machine with openSuse (thought >> the >> problem might be Slackware) but openSuse failed as quickly. >> >> Thanks, >> >> Carl >> >> - Original Message - >> From: "George Sexton" >> To: "'Tomcat Users List'" >> Sent: Saturday, January 30, 2010 11:46 AM >> Subject: RE: Tomcat dies suddenly >> >> >> > I've had this happen. Finally, I got a stack trace. In my case, there >> > appears to be a bug in GLIBC, so when a reverse IP address lookup is >> done >> > AND there is an IPV6 entry, it causes a problem. >> > >> &
Re: Tomcat dies suddenly
Chuck, One more thing. Sunday, I ran with a max heap of 500MB in the hope I could force an OOM but it never happened... just tooled along GC'ing very happily. Thanks, Carl - Original Message - From: "Caldarale, Charles R" To: "Tomcat Users List" Sent: Wednesday, February 03, 2010 4:04 PM Subject: RE: Tomcat dies suddenly From: Carl [mailto:c...@etrak-plus.com] Subject: Re: Tomcat dies suddenly I added the print options in the hope that I could see something right at the time of failure. Since the failures occurred before the print options were used, I guess that rules out any possibility of the OS getting upset at the JVM calling for timing information. Back to square one, where the most likely culprit is still the Linux OOM killer. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Dan, I looked at the messages log but there was never anything of interest in it, just the normal stuff. Haven't tried disabling the oom-killer... might give that a go. I thought it was a failure in the JVM but have never found any indications of anything... I will keep looking. Back in the old days, C language buffer overruns were hard to find but this is worse because I don't seem to be able to find a test case to force the failure... I will keep looking. Thanks for the suggestions. Carl - Original Message - From: "Dan Armbrust" To: "Tomcat Users List" Sent: Wednesday, February 03, 2010 4:58 PM Subject: Re: Tomcat dies suddenly Since the failures occurred before the print options were used, I guess that rules out any possibility of the OS getting upset at the JVM calling for timing information. Back to square one, where the most likely culprit is still the Linux OOM killer. - Chuck Have you checked: /var/log/messages for this: Out of Memory: Killed process [PID] [process name] Or tried disabling oom-killer? # echo "0" > /proc/sys/vm/oom-kill # echo "1" > /proc/sys/vm/oom-kill From my own experiences with JVMs, it seems more likely that you have a JVM segfault occurring - which is supposed to be giving you a log. Not that the log is always that helpful. But it might be :) I've had a heck of a time finding stable JVMs on Linux running my workload - especially a 1.6 JVM. I was stuck on 1.5 for ages because none of the 1.6's would run without randomly segfaulting. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Chuck, Yes, I run the system monitor included with Slackweare. Currently showing 1.9GB used out of 3.8GB available and no swap being used... that's about where it has been tracking. The machine has nothing but Tomcat and a small server (for reports and applet data... could/should have been a servlet but was originally developed this way) running on it. The VisualJVM display shows heap usage in the 400-500MB range... that's a little higher than it usually runs but no where near the maximum of 1GB. PermGen is running around 100-120MB. The other parts of the heap are printed periodically in catalina.out and look OK as far as I can tell, i.e., nothing seems to be pushing a limit. Still puzzled but I know there is a reason sitting out there somewhere. Thanks for your suggestions and comments. Carl - Original Message - From: "Caldarale, Charles R" To: "Tomcat Users List" Sent: Wednesday, February 03, 2010 5:00 PM Subject: RE: Tomcat dies suddenly From: Carl [mailto:c...@etrak-plus.com] Subject: Re: Tomcat dies suddenly I am hoping I can see failure at the point of failure with the heap dump options That won't have any effect on the Linux OOM killer. If it is the OOM killer taking out Tomcat, the JVM will not receive any notification - all threads will just be terminated with extreme prejudice. There won't be a chance for the heap dump to happen. From what you've observed, it sounds like something other than the heap may be eating up process (and real) memory space. Have you monitored actual real memory usage of the process as it's running? (Can't be done from inside the JVM; you'll need Linux-specific tools for this.) - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Chris, Interesting idea. I tried over the weekend to force that situation with JMeter hitting a simple jsp that did some data stuff and created a small display. I pushed it to the point that there were entries in the log stating it was out of memory (when attempting to GC, I think) but it just slowed way down and never crashed. I could see from VisualJVM that it had used the entire heap but, again, I could never get it to crash. Strange because it doesn't have the classic signs (slowing down or throwing out of memory exceptions or freezing), it just disappears without any tracks. I am certain there is a reason somewhere, I just haven't found it yet. Thanks for your suggestions, Carl - Original Message - From: "Christopher Schultz" To: "Tomcat Users List" Sent: Wednesday, February 03, 2010 5:45 PM Subject: Re: Tomcat dies suddenly -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Chuck, On 2/3/2010 5:00 PM, Caldarale, Charles R wrote: From: Carl [mailto:c...@etrak-plus.com] Subject: Re: Tomcat dies suddenly I am hoping I can see failure at the point of failure with the heap dump options That won't have any effect on the Linux OOM killer. If it is the OOM killer taking out Tomcat, the JVM will not receive any notification - all threads will just be terminated with extreme prejudice. There won't be a chance for the heap dump to happen. Maybe what Carl should try is to actually cause an OOME without triggering the "OOM Killer". Carl: you said that everything is copacetic when you run with a smaller heap, right? Well, run with that smaller heap, but then write a simply servlet/JSP/whatever that just allocates huge, say, byte[] buffers until it gets an OOME and verify that you get a heap dump when that happens. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAktp/JEACgkQ9CaO5/Lv0PBI0ACgmDLJVxM5xDiN5UIFsqqKRj7i 2J8AoL8ENGsYxwudtFrLy+tZTzmr/rxC =4yC9 -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Mark, This was both helpful and intriguing. 1. I had always used top to see memory used until I saw the system monitor tools in Slackware. Had not compared the two. At this moment, the system monitor is reporting .96GB of memory used while top and vmstat are reporting 3.6GB... quite a difference. From now on, top/vmstat it is. Further, the fact that this machine is running that close to the 4GB physical memory would seem to make it a candidate for failure with a fair amount of activity. Today could be interesting and revealing. 2. The only reference to 'RunTime' I could find in the code was in a try-catch in the ASTranslatorFactory where it throws a RunTimeException. We use this package in the process for communicating with Flash applications (part of our application uses Flash to provide a richer environment.) The ASTranslator jars are the latest ones and have not been changed since the middle of 2007. I am not certain how the process works inside but I would have thought the jars would have been updated if there were problems. 3. I am not certain I understood your explanation of potential DNS problems. This server is very simple: it receives requests from the outside, processes those (usually accessing a data server which is in the /etc/hosts file) and sends the response on its way. During the processing, there is no accessing the outside world that I know of. I would think if there was a request to the outside world that was causing a problem, we would see failure in a specific are of the overall system but we are not seeing anything like that. I did cut the Xms, Xmx in half in an attempt to force the problem but nothing happened (the system worked just fine) and I have since moved it back to it's old setting (1024m.) Thanks for your ideas and comments. Carl - Original Message - From: "Mark Eggers" To: "Tomcat Users List" Sent: Wednesday, February 03, 2010 11:46 PM Subject: Re: Tomcat dies suddenly Carl, A couple of random thoughts . . . I'm not familiar with the Slackware monitoring tools, but I am with the various tools that come with Fedora / Redhat. One of the things that I've noticed with those GUI tools is that they add cache and buffers to the free memory total. Tools like top and vmstat should give a more complete picture of your memory. With vmstat you can watch free, cache, buffers, and swap conveniently. With top, you can actually do a command line monitor and watch a particular PID. From the taroon-list: If you're running a 32 bit Linux and run out of low memory, it doesn't matter how much high memory you have, the OOM killer will start killing processes off. Since you're running a 64 bit Linux, this should not be the problem. A discussion on stackoverflow.com may be more relevant to your situation. It turns out (according to the discussion) that calling Runtime.getRuntime().exec() on a busy system can lead to transient memory shortages which trigger the OOM killer. If Runtime.getRuntime().exec() or similar calls do not exist in your application, then please skip the following speculation. I've made some comments concerning host resolution at the end of this message which might be helpful. If Runtime.getRuntime().exec() is used, the scenario goes like this: 1. call Runtime.getRuntime().exec() 2. fork() gets called and makes a copy of the parent process 3. System runs a different process At this point you have two processes with largish memory requirements At this point the OOM killer may get triggered 4. exec() gets called on the child process and memory requirements go back down. At least that's how I read the this reference: http://stackoverflow.com/questions/209875/from-what-linux-kernel-libc-version-is-java-runtime-exec-safe-with-regards-to-m Since processes that fork a lot of child processes are high on OOM killer's kill list, Tomcat gets killed. See for example: http://prefetch.net/blog/index.php/2009/09/30/how-the-linux-oom-killer-works/ As to why it would happen on the newer production systems and not the older system, my only idea concerns the version of the kernel you're using. Memory management has been significantly reworked between the 2.4 and 2.6 kernels. If you use a 2.4 kernel on your older system, this could explain some of the differences with memory allocation. So, if Runtime.getRuntime().exec() is used, what are some possible solutions? 1. Reducing Xms, Xmx while adding physical memory If you do this, then the fork() call without the exec() being called directly afterwards won't be as expensive. Your application will be able to serve more clients without potentially triggering the OOM killer. Garbage collection may be an issue if this is done, so tuning with JMeter is probably a good idea. 2. Create a lightweight process that forks what Runtime.getRuntime().exec()
Re: Tomcat dies suddenly
Kees Jan, Actually, since the first machine I brought it up was brand new, my first thought was a bad memory stick that only had a problem in a certain temperature range. However, running memtest86 for 30 hours showed nothing and the problem happened on a second machine (also brand new but a different processor, motherboard, etc. but same OS, Java and Tomcat) so I don't think it is a problem with the hardware. I did try a different OS (openSuse) and it showed the same problem. I also tried the latest 32 bit Java on a fresh install of 32 bit Slackware but that also showed the problem (even though an older version of Slackware 32 bit running Java 5.5 from several years ago does not... this setup has run flawlessly for 3-4 years.) Thanks for you suggestions. Carl - Original Message - From: "Kees Jan Koster" To: "Tomcat Users List" Sent: Thursday, February 04, 2010 12:47 AM Subject: Re: Tomcat dies suddenly Dear Carl, So this problem happens on one machine or on more than one? If it is limited to one machine, I would be inclined to run memtest86 on it to see if the RAM is good. Kees Jan On 4 feb 2010, at 02:07, "Carl" wrote: Chris, Interesting idea. I tried over the weekend to force that situation with JMeter hitting a simple jsp that did some data stuff and created a small display. I pushed it to the point that there were entries in the log stating it was out of memory (when attempting to GC, I think) but it just slowed way down and never crashed. I could see from VisualJVM that it had used the entire heap but, again, I could never get it to crash. Strange because it doesn't have the classic signs (slowing down or throwing out of memory exceptions or freezing), it just disappears without any tracks. I am certain there is a reason somewhere, I just haven't found it yet. Thanks for your suggestions, Carl - Original Message - From: "Christopher Schultz" > To: "Tomcat Users List" Sent: Wednesday, February 03, 2010 5:45 PM Subject: Re: Tomcat dies suddenly -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Chuck, On 2/3/2010 5:00 PM, Caldarale, Charles R wrote: From: Carl [mailto:c...@etrak-plus.com] Subject: Re: Tomcat dies suddenly I am hoping I can see failure at the point of failure with the heap dump options That won't have any effect on the Linux OOM killer. If it is the OOM killer taking out Tomcat, the JVM will not receive any notification - all threads will just be terminated with extreme prejudice. There won't be a chance for the heap dump to happen. Maybe what Carl should try is to actually cause an OOME without triggering the "OOM Killer". Carl: you said that everything is copacetic when you run with a smaller heap, right? Well, run with that smaller heap, but then write a simply servlet/JSP/whatever that just allocates huge, say, byte[] buffers until it gets an OOME and verify that you get a heap dump when that happens. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAktp/JEACgkQ9CaO5/Lv0PBI0ACgmDLJVxM5xDiN5UIFsqqKRj7i 2J8AoL8ENGsYxwudtFrLy+tZTzmr/rxC =4yC9 -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Chuck, An interesting situation. Of the 4028MB of physical memory, top was showing usage 4001MB used... now, that would be really inviting. (Note that none of the 19GB swap was being used.) I had X windows, the system monitor, VisualJVM and a browser running. Further, the memory used by the system was constantly increasing (according to top) even though Tomcat is essentially the only application (other than those I mentioned above) running and, if I understand it correctly, Tomcat is (mostly) running in the heap and should, therefore, not be requiring more memory (of any consequence.) I stopped X windows and the applications it was running... that recovered about 400MB (according to top.) However, that is slowly being eaten away by something (the pattern is that the used memory goes up a bit, flattens and then goes up a bit more... it never goes down except for a mment here and there.) I would have expected the usage to flatten out and then fluctuate about that number. This implies a memory leak someplace in the system. I did the search for 'runtime' as case-insensitive. If I have a memory leak, cutting the heap in half will just put off the problem, won't it? Also, it would seem the leak has to be outside my application as leaks from the application should show up by blowing the heap... is this correct? (permGen is set as 384m, both max and min and is currently using around 100m according to VisualJVM.) Thanks for your insights. Carl - Original Message - From: "Caldarale, Charles R" To: "Tomcat Users List" Sent: Thursday, February 04, 2010 9:14 AM Subject: RE: Tomcat dies suddenly From: Carl [mailto:c...@etrak-plus.com] Subject: Re: Tomcat dies suddenly top and vmstat are reporting 3.6GB... That must look quite inviting to the OOM killer. The only reference to 'RunTime' I could find It's "Runtime", not "RunTime", if you were doing case-sensitive searches. I did cut the Xms, Xmx in half in an attempt to force the problem That is most likely to eliminate the problem, not force it. Increasing Xms and Xmx should trigger it more frequently. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
, Charles R" To: "Tomcat Users List" Sent: Thursday, February 04, 2010 12:46 PM Subject: RE: Tomcat dies suddenly From: Carl [mailto:c...@etrak-plus.com] Subject: Re: Tomcat dies suddenly if I understand it correctly, Tomcat is (mostly) running in the heap and should, therefore, not be requiring more memory Not necessarily. The real memory used by a Java process can continue to climb until all the pages assigned to the heap and associated spaces have actually been touched. That might not happen for quite some time, especially in PermGen. However, that is slowly being eaten away by something. I would have expected the usage to flatten out and then fluctuate about that number. This implies a memory leak someplace in the system. Probably not. As Dan explained, that's normal file caching behavior. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Tsirkin, I doubt that will help as the server already has 19GB of swap and the swap is never touched. Thanks for the suggestion. Carl - Original Message - From: "Tsirkin Evgeny" To: "Tomcat Users List" Sent: Thursday, February 04, 2010 3:11 PM Subject: Re: Tomcat dies suddenly Just an advise . I don't know what the problem is,however suppose that this is OOM killer - why not just add some GB of swap disk spae is even chiper then RAM Evgeny On Thu, Feb 4, 2010 at 9:04 PM, Carl wrote: Chuck and Dan, First, some housekeeping to respond to your points and questions: 1. The 3.6GB I stated in my prior email to Dan was the total memory used. The Tomcat process (from top) was 1.7-1.8GB. 2. I did not know that Linux cached disk accesses in RAM. I have worked with Linux (and Unix and Aix before that) for a number of years but never understood all the pieces... as long as it worked, I was happy (and, in general, I didn't ask much of the systems.) To sum up your observations: The memory usage I was seeing was not unusual and not likely the source of the problem. The system failed again about 30 minutes ago. The overall memory dropped to 2.9GB (from top.) The last entries in catalina.out were: 41608.326: [GC 41608.326: [ParNew: 72021K->4186K(76672K), 0.0674360 secs] 376022K->309373K(1040064K) icms_dc=0 , 0.0675430 secs] [Times: user=0.25 sys=0.00, real=0.07 secs] 41610.048: [GC 41610.048: [ParNew: 72346K->8512K(76672K), 0.0408460 secs] 377533K->318233K(1040064K) icms_dc=0 , 0.0409480 secs] [Times: user=0.15 sys=0.00, real=0.04 secs] The GC information in catalina.out before that was: 41391.749: [GC 41391.749: [ParNew: 70179K->2466K(76672K), 0.0064220 secs] 374180K->306467K(1040064K) icms_dc=0 , 0.0065240 secs] [Times: user=0.02 sys=0.00, real=0.01 secs] 41393.383: [GC 41393.383: [ParNew: 70626K->2571K(76672K), 0.0059940 secs] 374627K->306572K(1040064K) icms_dc=0 , 0.0060940 secs] [Times: user=0.03 sys=0.00, real=0.01 secs] and before that: 41216.669: [GC 41216.669: [ParNew: 69204K->2621K(76672K), 0.0068300 secs] 367414K->300832K(1040064K) icms_dc=0 , 0.0069380 secs] [Times: user=0.02 sys=0.00, real=0.00 secs] 41216.710: [GC 41216.710: [ParNew: 70720K->3097K(76672K), 0.0064720 secs] 368930K->301364K(1040064K) icms_dc=0 , 0.0065740 secs] [Times: user=0.03 sys=0.01, real=0.00 secs] 41216.750: [GC 41216.750: [ParNew: 71117K->3241K(76672K), 0.0063760 secs] 369384K->301618K(1040064K) icms_dc=0 , 0.0064780 secs] [Times: user=0.03 sys=0.00, real=0.00 secs] The /var/log/messages from that machine for today were: Feb 4 00:08:05 tomcat2 -- MARK -- Feb 4 00:28:05 tomcat2 -- MARK -- Feb 4 00:48:05 tomcat2 -- MARK -- Feb 4 01:08:05 tomcat2 -- MARK -- Feb 4 01:28:05 tomcat2 -- MARK -- Feb 4 01:48:05 tomcat2 -- MARK -- Feb 4 02:08:05 tomcat2 -- MARK -- Feb 4 02:28:05 tomcat2 -- MARK -- Feb 4 02:48:05 tomcat2 -- MARK -- Feb 4 03:08:05 tomcat2 -- MARK -- Feb 4 03:28:05 tomcat2 -- MARK -- Feb 4 03:48:05 tomcat2 -- MARK -- Feb 4 04:08:05 tomcat2 -- MARK -- Feb 4 04:28:06 tomcat2 -- MARK -- Feb 4 04:48:06 tomcat2 -- MARK -- Feb 4 05:08:06 tomcat2 -- MARK -- Feb 4 05:28:06 tomcat2 -- MARK -- Feb 4 05:48:06 tomcat2 -- MARK -- Feb 4 06:08:06 tomcat2 -- MARK -- Feb 4 06:28:06 tomcat2 -- MARK -- Feb 4 06:48:06 tomcat2 -- MARK -- Feb 4 07:08:06 tomcat2 -- MARK -- Feb 4 07:28:06 tomcat2 -- MARK -- Feb 4 07:48:06 tomcat2 -- MARK -- Feb 4 08:08:06 tomcat2 -- MARK -- Feb 4 08:28:06 tomcat2 -- MARK -- Feb 4 08:48:06 tomcat2 -- MARK -- Feb 4 09:08:06 tomcat2 -- MARK -- Feb 4 09:28:06 tomcat2 -- MARK -- Feb 4 09:33:32 tomcat2 /usr/sbin/gpm[3414]: *** info [mice.c(1766)]: Feb 4 09:33:32 tomcat2 /usr/sbin/gpm[3414]: imps2: Auto-detected intellimouse PS/2 Feb 4 09:48:06 tomcat2 -- MARK -- Feb 4 10:08:06 tomcat2 -- MARK -- Feb 4 10:28:06 tomcat2 -- MARK -- Feb 4 10:48:06 tomcat2 -- MARK -- Feb 4 11:08:06 tomcat2 -- MARK -- Feb 4 11:28:06 tomcat2 -- MARK -- Feb 4 11:48:06 tomcat2 -- MARK -- Feb 4 12:08:06 tomcat2 -- MARK -- Feb 4 12:28:06 tomcat2 -- MARK -- Feb 4 12:48:06 tomcat2 -- MARK -- Feb 4 13:08:07 tomcat2 -- MARK -- Feb 4 13:28:07 tomcat2 -- MARK -- Feb 4 13:48:07 tomcat2 -- MARK -- I don't see anything here and this file has looked pretty much like this every failure. JAVA_OPTS in catalina.sh are: JAVA_OPTS="-Xms1024m -Xmx1024m -XX:PermSize=384m -XX:MaxPermSize=384m -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/local/tomcat/logs" I checked /usr/local/tomcat/logs but found only the usual files. The information was in catalina.2010-02-04.log showed our nightly restart of Tomcat but nothing more recent. There were, however, several entries like the following that indicate some kind of problem: Feb 4, 2010 1:10:03 AM org.apache.catalina.loader.WebappClassLoad
Re: Tomcat dies suddenly
Mark, Since I don't understand what is causing the problem, all information is helpful and I appreciate you taking time to think about what could be wrong. 1. The application runs fine on an older system. Do we have the glibc and kernel versions for all systems? The old system: P4. 1GB memory, 1.3GB swap. Uses swap on a regular basis. kernel is 2.4.25. Java is 1.5.0_01-b08. Tomcat is 5.5.23. Glibc is version 2.3.1. New systems: Server A (Dell T110) is a Xeon 3440, sever B (Dell T105) is an AMD. A has 4GB memory and 19GB swap which is never used. B has 6GB memory and 10GB swap which is never used. A and B both use kernel version 2.6.29.6, Java 1.6.0_18-b07 and Tomcat 6.0.24. Glibc version is 4.3.3 for both A and B. 2. Different usage patterns (?) seem to cause the outages at different rates (if I remember an account of one Friday). What paths in the application were being exercised most heavily during that time? The outages appear to be most frequent in times of heavy transaction processing. This part of the application is basically a shopping cart although the path from start to 'in the cart' has many variations depending on the type of item being registered for, i.e., the registration process steps through 20+ classes each of which could request additional processing, display a screen for user input, etc. It seems during periods of heavy transactions, the system fails more frequently but it may be that the application requires a certain cumulative amount of activity before it fails and that activity can be spread over several hours or several days. However, since Tomcat is restarted once a day (around 1:00AM after rolling out changes), it would seem that the application would not be able to carry activity from one day into the next. Therefore, it would seem that the failure is triggered by something on the day it occurs. Thanks for your help. Carl - Original Message - From: "Mark Eggers" To: "Tomcat Users List" Sent: Thursday, February 04, 2010 7:10 PM Subject: RE: Tomcat dies suddenly --- On Thu, 2/4/10, Caldarale, Charles R wrote: > > 6. Carl was using 32-bit Linux, which he isn't :( > > Correct, which made the whole point moot, so I'm not sure > why Dan even brought it up. > I just mentioned the 32-bit Linux behavior for completeness. I did state that I realized 32-bit Linux is not in play. > > AFAIK, 64-bit Linux has a wide-open memory addressing > scheme. Maybe it > > considers everything under 17 billion GiB to be "low > memory", now :) > > No, the hardware restrictions don't exist in 64-bit mode. This is what I've read as well. If you use 64-bit Linux, this problem goes away. There are also some ways to build the 32-bit kernel in order to reduce this problem. All this is moot since a 64-bit Linux kernel is being used. As to the copy-on-write behavior for fork()d processes, it would help if I read the man pages: Under Linux, fork() is implemented using copy-on-write pages, so the only penalty that it incurs is the time and memory required to duplicate the parent’s page tables, and to create a unique task structure for the child. It turns out that things are a little bit more complicated than that, in that since version 2.3.3 fork is actually a wrapper to clone(2) with the appropriate flags to give the same result as a traditional fork(2) call. All of this is moot however if there is no Runtime.exec() call in the application. I'm a bit curious though about several points: 1. The application runs fine on an older system. Do we have the glibc and kernel versions for all systems? 2. Different usage patterns (?) seem to cause the outages at different rates (if I remember an account of one Friday). What paths in the application were being exercised most heavily during that time? As for cache / buffer / free - I've seen cases where the cache did not go to 0, but swap was in play (slow disk, small amount of memory). Sorry for chasing down the rabbit hole . . . /mde/ - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Bill, ulimit -a shows the following: core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited file size (blocks, -f) unlimited pending signals (-i) 40960 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size(512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 40960 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited This looks fine except the 'open files'. I don't think we would exceed that number but it might be possible. Thanks for your interest. Carl - Original Message - From: "Bill Stoddard" To: "Tomcat Users List" Sent: Friday, February 05, 2010 9:09 AM Subject: Re: Tomcat dies suddenly On 2/5/10 6:31 AM, Carl wrote: Mark, Since I don't understand what is causing the problem, all information is helpful and I appreciate you taking time to think about what could be wrong. Could one of the log files written by Tomcat be bumping into your file size ulimit? ulimit -a to check file size ulimit If the filesize ulimit is not unlimited, compare the value with the size of the files being written by Tomcat. Bill - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Peter, Do you see any harm in just doubling the number (to 2048) just to see if it has an impact? Thanks, Carl - Original Message - From: "Peter Crowther" To: "Tomcat Users List" Sent: Friday, February 05, 2010 9:58 AM Subject: Re: Tomcat dies suddenly On 5 February 2010 14:41, Carl wrote: Bill, ulimit -a shows the following: [...] open files (-n) 1024 [...] This looks fine except the 'open files'. I don't think we would exceed that number but it might be possible. Note that a socket uses a file descriptor in UNIX, so you should include open sockets in that calculation. - Peter - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Peter, OK, I raised the value from 1024 to 4096. Thanks for the tip... we'll see over the next 10 days if it had any impact. Carl - Original Message - From: "Peter Crowther" To: "Tomcat Users List" Sent: Friday, February 05, 2010 10:29 AM Subject: Re: Tomcat dies suddenly On 5 February 2010 15:16, Carl wrote: Do you see any harm in just doubling the number (to 2048) just to see if it has an impact? I wouldn't expect any problems, but I don't know your server as well as you do. Just don't hold me responsible if your Tomcat fails / the dog eats the hamster / someone holds you hostage for the release of the unknown soldier's grandmother (delete all that don't apply). I'd suggest checking about limits that may be compiled into the kernel, too. File descriptor usage masks used to be stored as bitmasks in arrays, and the kernel used to have fixed maximum sizes. But then, this is coming from someone who every day walks past a museum containing a computer with a total of 1,024 bits of memory... things may have improved since then. - Peter - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Chris, I agree but this seemed like a low risk change. I also understand that changing too many things at once sometimes confuses the issue. Thanks, Carl - Original Message - From: "Christopher Schultz" To: "Tomcat Users List" Sent: Friday, February 05, 2010 10:48 AM Subject: Re: Tomcat dies suddenly -BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Carl, On 2/5/2010 10:16 AM, Carl wrote: Do you see any harm in just doubling the number (to 2048) just to see if it has an impact? We would expect to see an OOME (strange, I know) if you were running out of file descriptors. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAktsPbYACgkQ9CaO5/Lv0PBWnwCfXyZVJNsd/mMCgJWVamRBVZ+F AMkAn1H2romyi8eFunc3lF+eDiQa0Hlq =muVu -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Jonathon, I have yet to do any sampling, I have been visually monitoring the servers with the VisualJVM, the Slackware system monitor, top and vmstat. I was using JMeter to stress test the application. The old server is Slackware 12.1 (32 bit), Tomcat 5.5, Java 1.5.0_01. The new servers are Slackware 13 (64 bit), Tomcat 6.0.24, Java 1.6_18. The old server has run flawlessly for several years. The new servers are kicking my butt: - They will run for a period of time (could be 10 days, could be 15 minutes) and silently fail, no error logs, no memory dump, nothing in catalina.out, no tracks that I have been able to find. - The issue seems load related and cumulative. A week ago was a very heavy processing day and the server failed repeatedly starting about noon. Otherwise, it would usually fail late in the day. - I have tried different heap sizes. My first inclination was a larger heap but, as Chuck pointed out, that was probably the wrong thing. I am currently running with a 500MB heap (384MB permGen.) Been up for two days and looks great. - I have quite a bit of logging to catalina.out because I was, at one point, looking for a connection leak. I also started incremental GC and printing the GC instances using the appropriate java flags. The last entry for the last failure was a GC. - I tried provoking the failure by stressing the application using JMeter... wouldn't fail even when I forced an OOM. Your idea of sampling is interesting... may try that next. Thanks, Carl - Original Message - From: "Jonathan Mast" To: "Tomcat Users List" Sent: Friday, February 05, 2010 7:11 PM Subject: Re: Tomcat dies suddenly Hi Carl, I've skimmed thru all your posts, please excuse me if these questions are redundant. You've mentioned VisualJVM, JMeter, and Slackware monitoring tools, it seems to me that you are using these to occasionally to monitor you Tomcat instance, when things seem to be going badly (thats my perception from reading your posts) Have you done any long term sampling (Tomcat threads, memory, etc) of your system? I use Slackware 12.2 (32-bit), Httpd 2.2, Java 1.6, Tomcat 6.0.18 and had Tomcat die in a manner similiar to yours: First it died silently (nothing in the catalina.out), the next time (about a week later) it became non-responsive complaining about PermGen, OOMException and so on. I haven't determined the cause yet, I rolled over to another server (exact same hardware/software btw). I ran Memtest on the problematic server, no problems detected. Anyways, I've been sampling the httpd process count, tomcat thread count, and free memory (sans cache and buffers) every minute via a cron job and writing the results to a file on both machines since this issue arose about 2 weeks ago. My hope is that the next time it bugs out, i'll be able to say "oh the thread count exploded, but the memory use only increased incrementally" or whatever the cause actually is. Have you done any monitoring at that fine grained a level on your systems? It would be especially interesting to see how your old HW setup compares to the new ones. Hope I've helped a little bit :) On Fri, Feb 5, 2010 at 11:26 AM, Carl wrote: Chris, I agree but this seemed like a low risk change. I also understand that changing too many things at once sometimes confuses the issue. Thanks, Carl - Original Message - From: "Christopher Schultz" < ch...@christopherschultz.net> To: "Tomcat Users List" Sent: Friday, February 05, 2010 10:48 AM Subject: Re: Tomcat dies suddenly -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Carl, On 2/5/2010 10:16 AM, Carl wrote: Do you see any harm in just doubling the number (to 2048) just to see if it has an impact? We would expect to see an OOME (strange, I know) if you were running out of file descriptors. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAktsPbYACgkQ9CaO5/Lv0PBWnwCfXyZVJNsd/mMCgJWVamRBVZ+F AMkAn1H2romyi8eFunc3lF+eDiQa0Hlq =muVu -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Jonathon, My system is a little different as I don't run Apache and I have another java process running but your script is certainly helpful. Thanks, Carl - Original Message - From: "Jonathan Mast" To: "Tomcat Users List" Sent: Saturday, February 06, 2010 4:38 PM Subject: Re: Tomcat dies suddenly Carl, Here's what I have on my system, you'll obviously need to adjust for your setup, especially the httpd part as I don't believe you're using it: #db-style timestamp STAMP=`date +%F' '%T`; # count the number of httpd child processes AP_COUNT=`ps auxf | grep -c "httpd"`; # count the number of Tomcat threads. # NOTE: this assumes that the only program that is using java is Tomcat TC_COUNT=`ps auxHS | grep -c "bin/java"`; # pipe the output of free into grep seaching for the second line MEM_CHECK=`free -m | grep "buffers/"`; #use a tab delimiter to simplify importing results into a spreadsheet or db TAB=`echo -e '\t'`; MESSAGE=$STAMP$TAB--$TAB$AP_COUNT$TAB$TC_COUNT$TAB$MEM_CHECK$TAB[httpd,tomcat,memory]; echo $MESSAGE; hope you find it helpful On Fri, Feb 5, 2010 at 10:57 PM, chadwickbailey71 < chadwickbaile...@yahoo.com> wrote: There is no hardware restrictions in 64-bit mode. use 64-bit Linux, this will fix the problem - Learn an http://automatedsocialnetworking.com/ Automated Social Marketing technique WITHOUT Spending More than 5 Minutes Per Month at your Computer :working: -- View this message in context: http://old.nabble.com/Tomcat-dies-suddenly-tp27377457p27476911.html Sent from the Tomcat - User mailing list archive at Nabble.com. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Mark, Thanks for your information. I am not certain where the glibc versions I previously gave came from because... as you noted, they are not correct. The correct glibc version is 2.9 and the threading version (I hope I am stating this correctly) is NPTL 2.9. The kernel version is 2.6.29.6. From the Slackware 13.0 release notes, 'We've used the well-tested and recently patched 2.6.29.6 kernel'. I assumed that was about as good a kernel as I could get... was I wrong? I will try LD_ASSUME_KERNEL=2.4.1 (the Slackware site seems to indicate my version supports this setting) on one of the servers (so I can quickly revert to the other one if the setting doesn't work.) I will add the 'XX:ParallelGCThreads=1 option to one of the servers and see how that works. I am moving a litlle slowly because I don't want to make a nbad situation worse and want to be certain I can account for any improvements or screwups. Thanks for your insights. Carl - Original Message - From: "Mark Eggers" To: "Tomcat Users List" Sent: Saturday, February 06, 2010 9:46 PM Subject: Re: Tomcat dies suddenly --- On Fri, 2/5/10, Carl wrote: Carl, 1. The application runs fine on an older system. Do we have the glibc and kernel versions for all systems? The old system: P4. 1GB memory, 1.3GB swap. Uses swap on a regular basis. kernel is 2.4.25. Java is 1.5.0_01-b08. Tomcat is 5.5.23. Glibc is version 2.3..1. New systems: Server A (Dell T110) is a Xeon 3440, sever B (Dell T105) is an AMD. A has 4GB memory and 19GB swap which is never used. B has 6GB memory and 10GB swap which is never used. A and B both use kernel version 2.6.29.6, Java 1.6.0_18-b07 and Tomcat 6.0.24.. Glibc version is 4.3.3 for both A and B. A couple of observations here: Both the old new kernels end in odd numbers. From memory, I thought the odd kernel numbers were experimental, while the even numbers were production or mainline. I don't remember when this numbering system took place, but certainly by the time the 2.6 kernels were released. From kernel.org, I didn't see a 2.6.29 release marked as stable. The thread implementation has changed between the 2.4 and 2.6 kernels. You can see the thread implementation change by running: getconf GNU_LIBPTHREAD_VERSION I'd be interested in knowing the result of that and getconf GNU_LIBC_VERSION on both systems, since I don't recognize 4.3.3 as a glibc version (latest stable is 2.11.1, so I'm assuming 2.4.3.3?). glibc has some thread bugs that were fixed, but not until 2.8 or 2.9. There was also a persistent bug for 32-bit systems that bites Java applications (not your concern since you're running 64-bit) that wasn't fixed until 2.10.1. So in short, I'm guessing this may be a glibc NPTL issue. There are some observations that don't match, in that you're using Java 6 (most problems are reported with Java 1.4 and Java 5), and that you've used OpenSuSE (kernel, glibc version?) with the same Tomcat failure. However: For some of the earlier 2.6 kernels, you could get around NPTL problems by setting this environment variable: export LD_ASSUME_KERNEL=2.4.1 which forces the use of the old linuxthreads model. I don't know if that option is available with the 2.6 kernel that you are using. Another work-around has been posted on the Java bugs forum, albeit for a different threading problem and Java 5: -XX:ParallelGCThreads=1 sets GC to single threads. It's not fixed in the Java bugs database, because later versions of RedHat Linux don't exhibit the SIGSEGV problem. Some people report that single-threading GC solves their problems, while other people report that it doesn't. Some things to try I guess: 1. export LD_ASSUME_KERNEL=2.4.1 (maybe in startup.sh?) if your kernel supports this.. 2. set -XX:ParallelGCThreads=1 in catalina.sh (JAVA_OPTS). This is an experimental switch, not documented here: http://java.sun.com/javase/technologies/hotspot/vmoptions.jsp, but documented here: http://java.sun.com/javase/technologies/hotspot/gc/gc_tuning_6.html 3. Move to an even-numbered kernel with a glibc of 2.10.1 or better. 2.10 might be OK for your environment since the bug fixed in 2.10.1 causes problems for 32-bit systems only. just my two cents . . . . /mde/ - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
This problem continues to plague me. A quick recap so you don't have to search your memory or archives. The 10,000 foot view: new Dell T105 and T110, Slackware 13.0 (64 bit), latest Java (64 bit) and latest Tomcat. Machines only run Tomcat and a small, special purpose Java server (which I have also moved to another machine to make certain it wasn't causing any problems.) Periodically, Tomcat just dies leaving no tracks in any log that I have been able to find. The application has run on a Slackware 12.1 (32 bit) for several years without problems (except for application bugs.) I have run memTest86 for 30 hours on the T110 with no problems reported. More details: the Dell 105 has an AMD processor and (currently) 8 GB memory. The T110 has a Xeon 3440 processor and 4 GB memory. The current Java version is 1.6.0_18-b07. The current Tomcat version is 6.0.24. The servers are lightly loaded with less than 100 sessions active at any one time. All of the following trials have produced the same results: 1. Tried openSuse 64 bit. 2. Tried 32 bit Slackware 13. 3. Increased the memory in the T105 from 4GB to 6 GB and finally to 8 GB. 4. Have fiddled with the JAVA_OPTS settings in catalina.sh. The current settings are: JAVA_OPTS="-Xms512m -Xmx512m -XX:PermSize=384m -XX:MaxPermSize=384m -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/local/tomcat/logs" I can see the incremental GC effects in both catalina.out and VisualJVM. Note the fairly small (512MB) heap but watching VisualJVM indicates this is sufficient (when a failure occurs, VisualJVM will report the last amount of memory used and this is always well under the max in both heap and permGen.) More information about the failures: 1. They are clean kills as I can restart Tomcat immediately after failure and there is no port conflict. As I understand it, this implies the linux process was killed (I have manually killed the java process with kill -9 and had the same result that I have observed when the system fails) or Tomcat was shut down normally, e.g., using shutdown.sh (this always leaves tracks in catalina.out and I am not seeing any so I do not believe this is the case.) 2. They appear to be load related. On heavy processing days, the system might fail every 15 minutes but it could also run for up to 10 days without failure but with lighter processing. I have found a way to force a more frequent failure. We have four war's deployed (I will call them A, B, C and D.) They are all the same application but we use this process to enable access to different databases. A user accesses the correct application by https://xx.com/A or B, etc. A is used for production while the others have specific purposes. Thus, A is always used while the others are used periodically. If users start coming in on B, C and/or D, within hours the failure occurs (Tomcat shuts down bringing all of the users down, of course.) Note that the failure still does not happen immediately. 3. They do not appear to be caused by memory restrictions as 1) the old server had only 2 GB of memory and ran well, 2) I have tried adding memory to the new servers with no change in behavior and 3) the indications from top and the Slackware system monitor are that the system is not starved for memory. In fact, yesterday, running on the T105 with 8 GB of memory, top never reported over 6 GB being used (0 swap being used) yet it failed at about 4:00PM. 4. Most of the failures will occur after some amount of processing. We update the war's and restart the Tomcats each morning at 1:00AM. Most of the failures will occur toward the end of the day although heavy processing (or using multiple 'applications') may force it to happen earlier (the earliest failure has been around 1:00PM... it was the heaviest processing day ever.) It is almost as if there is a bucket somewhere that gets filled up and, when filled, causes the failure. (So there is no misunderstanding, there has never been an OOM condition reported anywhere that I can find.) Observations (or random musings): The fact that the failures occur after some amount of processing implies that the issue is related to memory usage, and, potentially, caused by a memory leak in the application. However, 1) I have never seen (from VisualJVM) any issue with either heap or permGen and the incremental GC's reported in catalina.out look pretty normal and 2) top, vmstat, system monitor, etc. are not showing any issues with memory. The failures look a lot like the linux OOM killer (which Mark or Chris said way back at the beginning which is now 2-3 months ago.) Does anyone have an idea where I could get information on tracking the linux signals that could cause this? Thanks, Carl ---
Re: Tomcat dies suddenly
alina.startup.HostConfig deployDirectory INFO: Deploying web application directory ROOT Feb 12, 2010 7:43:31 AM org.apache.coyote.http11.Http11Protocol start INFO: Starting Coyote HTTP/1.1 on http-8080 Feb 12, 2010 7:43:31 AM org.apache.coyote.http11.Http11Protocol start INFO: Starting Coyote HTTP/1.1 on http-8443 Feb 12, 2010 7:43:31 AM org.apache.coyote.http11.Http11Protocol start INFO: Starting Coyote HTTP/1.1 on http-443 Feb 12, 2010 7:43:31 AM org.apache.jk.common.ChannelSocket init INFO: JK: ajp13 listening on /0.0.0.0:8009 Feb 12, 2010 7:43:32 AM org.apache.jk.server.JkMain start INFO: Jk running ID=0 time=0/131 config=null Feb 12, 2010 7:43:32 AM org.apache.catalina.startup.Catalina start INFO: Server startup in 3617 ms 76.034: [GC [1 CMS-initial-mark: 4975K(481728K)] 25384K(520064K), 0.0183320 secs] [Times: user=0.02 sys=0.00, real=0.02 secs] 76.057: [CMS-concurrent-mark-start] 76.117: [CMS-concurrent-mark: 0.053/0.059 secs] [Times: user=0.12 sys=0.00, real=0.06 secs] 76.117: [CMS-concurrent-preclean-start] 76.124: [CMS-concurrent-preclean: 0.007/0.007 secs] [Times: user=0.02 sys=0.00, real=0.01 secs] 76.124: [CMS-concurrent-abortable-preclean-start] CMS: abort preclean due to time 81.142: [CMS-concurrent-abortable-preclean: 0.266/5.018 secs] [Times: user=0.68 sys=0.01, real=5.02 secs] 81.142: [GC[YG occupancy: 21090 K (38336 K)]81.142: [Rescan (parallel) , 0.0154810 secs]81.158: [weak refs processing, 0.0001730 secs] [1 CMS-remark: 4975K(481728K)] 26066K(520064K), 0.0157760 secs] [Times: user=0.02 sys=0.00, real=0.02 secs] 81.158: [CMS-concurrent-sweep-start] 81.164: [CMS-concurrent-sweep: 0.005/0.005 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 81.164: [CMS-concurrent-reset-start] 81.169: [CMS-concurrent-reset: 0.006/0.006 secs] [Times: user=0.00 sys=0.00, real=0.01 secs] Does anyone see anything in the above that looks odd (or maybe like a clue?) Thanks, Carl - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Chuck, Thanks for your quick reply. I don't think any of the file systems are in danger (we purposely spec'd large disks because they were cheap and we wouldn't have to deal with space shortages.) df gives: Filesystem 1K-blocks Used Available Use% Mounted on /dev/root 76896348468464 72521680 1% / /dev/sda4 66682840 8093552 55201984 13% /usr tmpfs 4085376 0 4085376 0% /dev/shm Tomcat is in /usr/local/tomcat. We have no quotas on any of the file systems (checked with 'quotacheck' just in case some rogue quota was set.) No logs (other than the standard Tomcat logs.) tomcat/temp currently has files totaling 272K... not likely that is a problem. Thanks, Carl - Original Message - From: "Caldarale, Charles R" To: "Tomcat Users List" Sent: Friday, February 12, 2010 8:39 AM Subject: RE: Tomcat dies suddenly From: Carl [mailto:c...@etrak-plus.com] Subject: Re: Tomcat dies suddenly The fact that the failures occur after some amount of processing implies that the issue is related to memory usage, and, potentially, caused by a memory leak in the application. Actually, it's looking less and less like a memory problem here. What about exhaustion of some other resource, such as a disk quota? Do the applications create any kind of log files or otherwise use increasing amounts of disk as they run? - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Chris, Thanks for your response. I'll try to find a memtest for 64 bit. On the 32 bit topic. I have run successfully for several years on a Slackware 12.1 32 bit system (Java 1.5 something, Tomcat 5.5.25.) When I brought up Slackware 13.0 32 bit, and the latest Tomcat and Java (32 bit, of course), it suffered from the same problem which surprised me. Also, one of the reasons for going to 64 bit was that we have had problems when some people were running B, C and D (permGen, very clear) so I was trying to get a little extra memory. No, we are not doing remote logging (the servers are 10' away from me.) Me stumped also... has always been so simple to set up a Tomcat server. Do I gain anything by trying Glassfish? Thanks, Carl - Original Message - From: "Christopher Schultz" To: "Tomcat Users List" Sent: Friday, February 12, 2010 10:13 AM Subject: Re: Tomcat dies suddenly -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Carl, On 2/12/2010 7:20 AM, Carl wrote: The 10,000 foot view: new Dell T105 and T110, Slackware 13.0 (64 bit), latest Java (64 bit) and latest Tomcat. Machines only run Tomcat and a small, special purpose Java server (which I have also moved to another machine to make certain it wasn't causing any problems.) Periodically, Tomcat just dies leaving no tracks in any log that I have been able to find. The application has run on a Slackware 12.1 (32 bit) for several years without problems (except for application bugs.) I have run memTest86 for 30 hours on the T110 with no problems reported. All of the following trials have produced the same results: One last check: have you tried a 32-bit JVM running on either a 32-bit or 64-bit OS? Your memory needs are quire modest, so a 32-bit JVM should suffice. I think you mentioned that, in the past, on lesser hardware, a 32-bit JVM was being used. The failures look a lot like the linux OOM killer (which Mark or Chris said way back at the beginning which is now 2-3 months ago.) While the Linux OOM killer /is/ possible, I agree with Chuck that it's sounding less like a memory problem. I would have bet on a hardware problem, but memtest86 seems to have ruled that out (any x86 system I've suspected of having flaky hardware has always failed on it's first round of memtest86 testing). I wonder if there's a memtest_x64 that you should be trying, instead . I honestly can't think of any other reason why Tomcat would be just dying with no trace: a clean shutdown would produce logs. The Linux OOM killer would say something in the system logs (you're not doing remote syslogging, and the message is somehow getting lost, are you?). A JVM bug would likely generate an hs_pid.log file with details of the crash. I have to admit, I'm stumped. :( - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkt1cAcACgkQ9CaO5/Lv0PB6lgCfc2s6FpRt+LNqzgNV2KG76FiZ VeQAn3utd641tdOet+lQkG+QjpUg/txt =QLU+ -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Jeffrey, Thanks for taking the time to give this some thought. 1. There is no place in the code that we intentionally put an exit(). I have grepped for exit() and found nothing. The system stops in a different place every time... the last entry in catalina.out has never been the same (over 15-20 failures.) 2. No, we do not share jars or classes... our thought was this was a potential for screwups and not really gaining anything. 3. Good idea... I will try this. Thanks, Carl - Original Message - From: "Jeffrey Janner" To: "Tomcat Users List" Sent: Friday, February 12, 2010 10:31 AM Subject: RE: Tomcat dies suddenly I've been following this thread with interest, but haven't weighed in since I'm doing much development these days. I have to say that I'm agreeing with Chuck and Chris that it is a resource issue - especially since it doesn't appear to be a problem unless under load. Also, the OOP mentioned that it doesn't seem to occur if only the production app is being hit. It takes someone using one of the other copies first to generate the problem. So the questions occur: 1) Are you absolutely positive that there is no code that could be calling exit()? 2) Are you sharing jar files or classes between the apps? If so, place a separate copy in the WEB-INF/lib directory of each webapp, and remove them from the "shared" location, to insure that you're not having an issue with one app stomping on another. The combined load on a resource in a shared class could be overloading some built-in limit, and that class could be causing the fail without logging an error, or you're not catching the error to log it. 3) You have the disk space, try turning on "debug level" in all your logging utilities and see what the apps were last doing. Jeff *** NOTICE * This message is intended for the use of the individual or entity to which it is addressed and may contain information that is privileged, confidential, and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient or the employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by reply or by telephone (call us collect at 512-343-9100) and immediately delete this message and all its attachments. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Tony, Thanks for your response. Way back (several months ago), I did start with an older version of Tomcat. When it showed the failures, I moved forward to the current version with a step in between. I did not try to go back to 5.5 or Java 1.5 (and I think that would be counter productive.) Thanks, Carl - Original Message - From: "Anthony J. Biacco" To: "Tomcat Users List" Sent: Friday, February 12, 2010 10:36 AM Subject: RE: Tomcat dies suddenly You haven't said if you tried a previous java version or tomcat version (6.0.20)? -Tony Sent from my Windows® phone. -Original Message- From: Carl Sent: Friday, February 12, 2010 5:22 AM To: Tomcat Users List Subject: Re: Tomcat dies suddenly This problem continues to plague me. A quick recap so you don't have to search your memory or archives. The 10,000 foot view: new Dell T105 and T110, Slackware 13.0 (64 bit), latest Java (64 bit) and latest Tomcat. Machines only run Tomcat and a small, special purpose Java server (which I have also moved to another machine to make certain it wasn't causing any problems.) Periodically, Tomcat just dies leaving no tracks in any log that I have been able to find. The application has run on a Slackware 12.1 (32 bit) for several years without problems (except for application bugs.) I have run memTest86 for 30 hours on the T110 with no problems reported. More details: the Dell 105 has an AMD processor and (currently) 8 GB memory. The T110 has a Xeon 3440 processor and 4 GB memory. The current Java version is 1.6.0_18-b07. The current Tomcat version is 6.0.24. The servers are lightly loaded with less than 100 sessions active at any one time. All of the following trials have produced the same results: 1. Tried openSuse 64 bit. 2. Tried 32 bit Slackware 13. 3. Increased the memory in the T105 from 4GB to 6 GB and finally to 8 GB. 4. Have fiddled with the JAVA_OPTS settings in catalina.sh. The current settings are: JAVA_OPTS="-Xms512m -Xmx512m -XX:PermSize=384m -XX:MaxPermSize=384m -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/local/tomcat/logs" I can see the incremental GC effects in both catalina.out and VisualJVM. Note the fairly small (512MB) heap but watching VisualJVM indicates this is sufficient (when a failure occurs, VisualJVM will report the last amount of memory used and this is always well under the max in both heap and permGen.) More information about the failures: 1. They are clean kills as I can restart Tomcat immediately after failure and there is no port conflict. As I understand it, this implies the linux process was killed (I have manually killed the java process with kill -9 and had the same result that I have observed when the system fails) or Tomcat was shut down normally, e.g., using shutdown.sh (this always leaves tracks in catalina.out and I am not seeing any so I do not believe this is the case.) 2. They appear to be load related. On heavy processing days, the system might fail every 15 minutes but it could also run for up to 10 days without failure but with lighter processing. I have found a way to force a more frequent failure. We have four war's deployed (I will call them A, B, C and D.) They are all the same application but we use this process to enable access to different databases. A user accesses the correct application by https://xx.com/A or B, etc. A is used for production while the others have specific purposes. Thus, A is always used while the others are used periodically. If users start coming in on B, C and/or D, within hours the failure occurs (Tomcat shuts down bringing all of the users down, of course.) Note that the failure still does not happen immediately. 3. They do not appear to be caused by memory restrictions as 1) the old server had only 2 GB of memory and ran well, 2) I have tried adding memory to the new servers with no change in behavior and 3) the indications from top and the Slackware system monitor are that the system is not starved for memory. In fact, yesterday, running on the T105 with 8 GB of memory, top never reported over 6 GB being used (0 swap being used) yet it failed at about 4:00PM. 4. Most of the failures will occur after some amount of processing. We update the war's and restart the Tomcats each morning at 1:00AM. Most of the failures will occur toward the end of the day although heavy processing (or using multiple 'applications') may force it to happen earlier (the earliest failure has been around 1:00PM... it was the heaviest processing day ever.) It is almost as if there is a bucket somewhere that gets filled up and, when filled, causes the failure. (So there is no misunderstanding, there has never been an OOM condition reported anywhere that I can find.) Observations (or random musings): The fact that the failure
Re: Tomcat dies suddenly
Chris,
Thank you... I would never have thought about this script. I'll fire that
baby up tonight.
Thanks,
Carl
- Original Message -
From: "Christopher Schultz"
To: "Tomcat Users List"
Sent: Friday, February 12, 2010 10:49 AM
Subject: Re: Tomcat dies suddenly
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Jeffrey,
On 2/12/2010 10:31 AM, Jeffrey Janner wrote:
Also, the OP mentioned that it doesn't seem to occur if only the
production app is being hit. It takes someone using one of the other
copies first to generate the problem. So the questions occur:
1) Are you absolutely positive that there is no code that could be
calling exit()?
One way to check this would be to launch the JVM using a script that
captures the exit code of the process:
$!/bin/sh
# Grab all these variable definitions from bin/catalina.sh
"$_RUNJAVA" "$LOGGING_CONFIG" $JAVA_OPTS $CATALINA_OPTS \
-Djava.endorsed.dirs="$JAVA_ENDORSED_DIRS" -classpath "$CLASSPATH" \
-Dcatalina.base="$CATALINA_BASE" \
-Dcatalina.home="$CATALINA_HOME" \
-Djava.io.tmpdir="$CATALINA_TMPDIR" \
org.apache.catalina.startup.Bootstrap "$@" start \
>> "$CATALINA_BASE"/logs/catalina.out 2>&1
result=$?
echo "JVM existed with status code=${result}"
If it's 0, then the changes that this is due to a System.exit(0) call
are very high.
You could also try running under a SecurityManager, and prohibit
System.exit() calls.
- -chris
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAkt1eKMACgkQ9CaO5/Lv0PBMIACgkZAP4UQQqVpELU2Ej4HQFOLI
7GEAn1rsUxRxvyXBrKGBYlomkaI0WN8C
=hSGU
-END PGP SIGNATURE-
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Peter, Great ideas (did you see Chris's response with a way of testing the exit code?) Of course I am willing to add some debug code (I'll do almost anything at this point)... I will look at the links you provided and try it out. I've contented right along that the major issue is that the failure leaves no tracks... I'm hopeful the 'debug' code you have suggested will allow me to start to understand the underlying cause. I will leave the security manager to last as I don't know that stuff very well. There is no native code in the application (used to do a lot of work in C and I am familiar with mayhem of buffer overruns, pointer screwups, etc.) Thanks, Carl - Original Message - From: "Peter Crowther" To: "Tomcat Users List" Sent: Friday, February 12, 2010 12:05 PM Subject: Re: Tomcat dies suddenly On 12 February 2010 16:43, Carl wrote: 1. There is no place in the code that we intentionally put an exit(). I have grepped for exit() and found nothing. The system stops in a different place every time... the last entry in catalina.out has never been the same (over 15-20 failures.) I'm wondering about a concurrency issue, given that the failure occurs more frequently under load but can occur at other times. But it's difficult to think of one that would cause a silent crash like this, unless you're using a library somewhere that makes use of the "crash and burn" school of error handling! ... actually, that's a thought. There *is* a way of you distinguishing the VM exiting in an orderly fashion, quitting, or being terminated, and that's to add a tiny webapp that just registers a shutdown hook (http://java.sun.com/j2se/1.5.0/docs/api/java/lang/Runtime.html#addShutdownHook%28java.lang.Thread%29). If you're willing to add that debug code, then you could log a message (or simply touch a file). If there's no message/file, the VM is shutting down due to an error or someone's calling halt (http://java.sun.com/j2se/1.5.0/docs/api/java/lang/Runtime.html#halt%28int%29). As another small piece of debugging, it would be very interesting to capture the exit status of the JVM. How are you starting it, and is there any chance of inspecting the code on exit? A non-zero exit code, in particular, would be interesting. As a third, rather larger, piece of debugging, you could consider running Tomcat under a security manager that allowed all operations except exit. This may be tough to get right, especially on a production server, but it would definitely tell you whether there were any Java calls to Runtime.exit() or Runtime.halt(). Finally, is there any native code in any part of your application? This is, of course, outside of any of the JVM handlers; a failure of native code can (and occasionally does!) cause mayhem. None of this is a solution, I'm afraid. It's all just more debugging and gathering more information. But the problem is sufficiently unusual that I think you're going to have to keep on debugging it :-(. - Peter - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Tony, I tried stressing it with JMeter and came up no results. I could push it hard enough to force an OOM but it performed/failed as expected leaving tracks all over the place. The stressing was not very sophisticated (just a couple of the production jsp's) but, like I said, it didn't show anything (I was really testing to see if the problem was in GC... it wasn't.) Might rig up a more comprehensive test... will see after I try Chris and Peter's ideas. Thanks, Carl - Original Message - From: To: "Tomcat Users List" Sent: Friday, February 12, 2010 12:07 PM Subject: Re: Tomcat dies suddenly Is it possible to run this server with a basic tomcat application under load to rule out the application causing the crash? On Fri, Feb 12, 2010 at 4:20 AM, Carl wrote: This problem continues to plague me. A quick recap so you don't have to search your memory or archives. The 10,000 foot view: new Dell T105 and T110, Slackware 13.0 (64 bit), latest Java (64 bit) and latest Tomcat. Machines only run Tomcat and a small, special purpose Java server (which I have also moved to another machine to make certain it wasn't causing any problems.) Periodically, Tomcat just dies leaving no tracks in any log that I have been able to find. The application has run on a Slackware 12.1 (32 bit) for several years without problems (except for application bugs.) I have run memTest86 for 30 hours on the T110 with no problems reported. More details: the Dell 105 has an AMD processor and (currently) 8 GB memory. The T110 has a Xeon 3440 processor and 4 GB memory. The current Java version is 1.6.0_18-b07. The current Tomcat version is 6.0.24. The servers are lightly loaded with less than 100 sessions active at any one time. All of the following trials have produced the same results: 1. Tried openSuse 64 bit. 2. Tried 32 bit Slackware 13. 3. Increased the memory in the T105 from 4GB to 6 GB and finally to 8 GB. 4. Have fiddled with the JAVA_OPTS settings in catalina.sh. The current settings are: JAVA_OPTS="-Xms512m -Xmx512m -XX:PermSize=384m -XX:MaxPermSize=384m -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/usr/local/tomcat/logs" I can see the incremental GC effects in both catalina.out and VisualJVM. Note the fairly small (512MB) heap but watching VisualJVM indicates this is sufficient (when a failure occurs, VisualJVM will report the last amount of memory used and this is always well under the max in both heap and permGen.) More information about the failures: 1. They are clean kills as I can restart Tomcat immediately after failure and there is no port conflict. As I understand it, this implies the linux process was killed (I have manually killed the java process with kill -9 and had the same result that I have observed when the system fails) or Tomcat was shut down normally, e.g., using shutdown.sh (this always leaves tracks in catalina.out and I am not seeing any so I do not believe this is the case.) 2. They appear to be load related. On heavy processing days, the system might fail every 15 minutes but it could also run for up to 10 days without failure but with lighter processing. I have found a way to force a more frequent failure. We have four war's deployed (I will call them A, B, C and D.) They are all the same application but we use this process to enable access to different databases. A user accesses the correct application by https://xx.com/A or B, etc. A is used for production while the others have specific purposes. Thus, A is always used while the others are used periodically. If users start coming in on B, C and/or D, within hours the failure occurs (Tomcat shuts down bringing all of the users down, of course.) Note that the failure still does not happen immediately. 3. They do not appear to be caused by memory restrictions as 1) the old server had only 2 GB of memory and ran well, 2) I have tried adding memory to the new servers with no change in behavior and 3) the indications from top and the Slackware system monitor are that the system is not starved for memory. In fact, yesterday, running on the T105 with 8 GB of memory, top never reported over 6 GB being used (0 swap being used) yet it failed at about 4:00PM. 4. Most of the failures will occur after some amount of processing. We update the war's and restart the Tomcats each morning at 1:00AM. Most of the failures will occur toward the end of the day although heavy processing (or using multiple 'applications') may force it to happen earlier (the earliest failure has been around 1:00PM... it was the heaviest processing day ever.) It is almost as if there is a bucket somewhere that gets filled up and, when filled, causes the failure. (So there is no misunderstanding, there has never been an OOM
Re: Tomcat dies suddenly
Konstantin, Your response provides some interesting ideas. Remember that I had tried shutting Tomcat down by killing (kill -9 xxx where xxx is the process id) the java process and saw the same ending in the logs that I have been seeing when the system 'fails'. So it would seem that I could test the implementation of Chris' and Peter's ideas by simply killing the java process and see what the debugging code gives me. At least that looks like a baseline. Thanks for the help. Carl - Original Message - From: "Konstantin Kolinko" To: "Tomcat Users List" Sent: Friday, February 12, 2010 1:17 PM Subject: Re: Tomcat dies suddenly 2010/2/12 Carl : More details: the Dell 105 has an AMD processor and (currently) 8 GB memory. The T110 has a Xeon 3440 processor and 4 GB memory. The current Java version is 1.6.0_18-b07. The current Tomcat version is 6.0.24. Actually, 6.0.24 version *will* exit silently if it is terminated by a signal. To reproduce (on Windows): 1. Run Tomcat in a console window. 2. Press Ctrl+C 3. Tomcat shutdowns in several seconds, cleanly, but silently. That is because in 6.0.24 a VM shutdown hook was added that cleanly terminates logging subsystem (flushing all cached messages). The VM starts all shutdown hook threads at the same time and they run in parallel. Thus if Tomcat is being stopped through a hook, the logging subsystem has a chance to be already shut down at that time. I think that the same silent shutdown will occur if calling System.exit(), by the same reason. I do not know, what will happen if running Tomcat with jsvc on a Unix. (IIRC, jsvc calls the stop method in Bootstrap/Catalina, so Tomcat shuts down not through a hook and all messages should be present). The above is specific to 6.0.24. In 6.0.20 logs are forcibly flushed after each message is written, and log files are never explicitly closed. Best regards, Konstantin Kolinko - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Chuck, Oooo, that's interesting. Just tried kill -9 xxx (the process id of the java process) on one the servers not currently in production and it put nothing in the catalina.out file and nothing in the Slackware messages file. Time to test the ideas from Chris and Peter. Thanks for your insights. Carl - Original Message - From: "Caldarale, Charles R" To: "Tomcat Users List" Sent: Friday, February 12, 2010 1:57 PM Subject: RE: Tomcat dies suddenly From: Konstantin Kolinko [mailto:knst.koli...@gmail.com] Subject: Re: Tomcat dies suddenly Actually, 6.0.24 version *will* exit silently if it is terminated by a signal. To reproduce (on Windows): 1. Run Tomcat in a console window. 2. Press Ctrl+C 3. Tomcat shutdowns in several seconds, cleanly, but silently. Not for me; I get the following in catalina.2010-02-12.log after entering CTRL-C: Feb 12, 2010 12:53:42 PM org.apache.coyote.http11.Http11Protocol pause INFO: Pausing Coyote HTTP/1.1 on http-8080 This is Tomcat 6.0.24, JDK 1.6.0_14, Windows XP Pro 32-bit. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Andre, Thanks for the response. I have read almost all of your posts and realy enjoy the way to take problems apart. Keep on thinking. Thanks, Carl - Original Message - From: "André Warnier" To: "Tomcat Users List" Sent: Friday, February 12, 2010 2:49 PM Subject: Re: Tomcat dies suddenly Carl wrote: .. a lot of messages, and I have not yet read all of them, but I sympathise with the predicament. I am very far from any kind of Java expertise, so just something that I vaguely seem to remember, although it may be from a very long time ago, not applicable, already mentioned by someone, and maybe not relevant at all.. I thus vaguely seem to remember that there existed some option to the JVM to provide a last resort amount of memory, only used if the JVM really ran out of memory, for the purpose of allowing at least some last-ditch desperate action to take place (like logging the catastrophe maybe). I don't remember how it's called or how it works, but maybe it will raise something in some expert's memory. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Chris, Another great idea. I can deal with huge log files a whole lot better than continued failures. Thanks, Carl - Original Message - From: "Christopher Schultz" To: "Tomcat Users List" Sent: Friday, February 12, 2010 3:01 PM Subject: Re: Tomcat dies suddenly -BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Carl, On 2/12/2010 2:42 PM, Carl wrote: Great ideas (did you see Chris's response with a way of testing the exit code?) [snip] There is no native code in the application (used to do a lot of work in C and I am familiar with mayhem of buffer overruns, pointer screwups, etc.) If you get really desperate, you can also run the jvm inside of strace and get ready for a huge log file. It's possible that you'll see the jvm fail on the same function call every time, and you'll get more information about the problem. strace will show you if a signal terminated the process, or if some other call killed it (like exec(), which would sure do a number on your JVM). - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkt1s58ACgkQ9CaO5/Lv0PB8YQCgq0kuu87WVbPy0P40vFFHyPJW RUsAn1dzTKz2NTm7HAKAK7xeAWJS/2hd =2HBr -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Chuck, Didn't know that but there is nothing in the catalina.2010-02-12.log except the lines from stopping Tomcat at 1:10AM this morning (the scripted restart after rolling out the changes from yesterday.) There is nothing in any of the other logs in the ~/tomcat/logs directory either. Thank you for the help. Carl - Original Message - From: "Caldarale, Charles R" To: "Tomcat Users List" Sent: Friday, February 12, 2010 3:09 PM Subject: RE: Tomcat dies suddenly From: Carl [mailto:c...@etrak-plus.com] Subject: Re: Tomcat dies suddenly Just tried kill -9 xxx (the process id of the java process) on one the servers not currently in production and it put nothing in the catalina.out file and nothing in the Slackware messages file. The message won't be in catalina.out - look in catalina.-MM-dd.log. Also, what Windows does may not be the same as Linux. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Chris, That script would also (I think) kill the VisualJVM and my small java server, wouldn't it. That is not happening... those are staying very much alive, just Tomcat goes down. Thanks, Carl - Original Message - From: "Christopher Schultz" To: "Tomcat Users List" Sent: Friday, February 12, 2010 3:10 PM Subject: Re: Tomcat dies suddenly -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Carl, On 2/12/2010 3:03 PM, Carl wrote: Oooo, that's interesting. Just tried kill -9 xxx (the process id of the java process) on one the servers not currently in production and it put nothing in the catalina.out file and nothing in the Slackware messages file. Time to test the ideas from Chris and Peter. Maybe you should look around for a script like this: #!/bin/sh while true do sleep `random` killall -KILL java done :) - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkt1tb4ACgkQ9CaO5/Lv0PArjgCeJ5HUImbpQCTyBgqVFt16bqUB NtMAn2kuTh159ad00+Y059p3XqCj5tZr =cLAj -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Konstantin, When I shut Tomcat down using shutdown.sh, I see these messages (from 1:10AM this morning): Feb 12, 2010 1:10:02 AM org.apache.coyote.http11.Http11Protocol pause INFO: Pausing Coyote HTTP/1.1 on http-8080 Feb 12, 2010 1:10:02 AM org.apache.coyote.http11.Http11Protocol pause INFO: Pausing Coyote HTTP/1.1 on http-8443 Feb 12, 2010 1:10:02 AM org.apache.coyote.http11.Http11Protocol pause INFO: Pausing Coyote HTTP/1.1 on http-443 Feb 12, 2010 1:10:03 AM org.apache.catalina.core.StandardService stop INFO: Stopping service Catalina Feb 12, 2010 1:10:03 AM org.apache.coyote.http11.Http11Protocol destroy INFO: Stopping Coyote HTTP/1.1 on http-8080 Feb 12, 2010 1:10:03 AM org.apache.coyote.http11.Http11Protocol destroy INFO: Stopping Coyote HTTP/1.1 on http-8443 Feb 12, 2010 1:10:03 AM org.apache.coyote.http11.Http11Protocol destroy INFO: Stopping Coyote HTTP/1.1 on http-443 in catalina.-MM-DD.log but I never see any messages when I do kill -9 xxx to kill the java process. Thanks, Carl - Original Message - From: "Konstantin Kolinko" To: "Tomcat Users List" Sent: Friday, February 12, 2010 3:19 PM Subject: Re: Tomcat dies suddenly 2010/2/12 Carl : Chuck, Oooo, that's interesting. Just tried kill -9 xxx (the process id of the java process) on one the servers not currently in production and it put nothing in the catalina.out file and nothing in the Slackware messages file. Time to test the ideas from Chris and Peter. Thanks for your insights. Carl - Original Message - From: "Caldarale, Charles R" To: "Tomcat Users List" Sent: Friday, February 12, 2010 1:57 PM Subject: RE: Tomcat dies suddenly From: Konstantin Kolinko [mailto:knst.koli...@gmail.com] Subject: Re: Tomcat dies suddenly Actually, 6.0.24 version *will* exit silently if it is terminated by a signal. To reproduce (on Windows): 1. Run Tomcat in a console window. 2. Press Ctrl+C 3. Tomcat shutdowns in several seconds, cleanly, but silently. Not for me; I get the following in catalina.2010-02-12.log after entering CTRL-C: Feb 12, 2010 12:53:42 PM org.apache.coyote.http11.Http11Protocol pause INFO: Pausing Coyote HTTP/1.1 on http-8080 This is Tomcat 6.0.24, JDK 1.6.0_14, Windows XP Pro 32-bit. That depends on your luck. As I said, all hooks run in parallel. You can compare those messages with the ones printed when you shutdown through "catalina.bat stop" (aka shutdown.bat). Also, note that to disable buffering in log subsystem in 6.0.24 you can configure "bufferSize=-1" for all your FileHandlers in logging.properties, see https://issues.apache.org/bugzilla/show_bug.cgi?id=48614#c5 6.0.25 and later will have log buffering turned off by default. Best regards, Konstantin Kolinko - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Andre, Take my comment as a compliment because that is the way it was meant... you have helped a lot of people on this least and I, for one, really appreciate that. I was waiting to see if someone could give me an idea how to implement what you remembered and, if not, then I would google around to see if I could find it myself. Thanks, Carl - Original Message - From: "André Warnier" To: "Tomcat Users List" Sent: Friday, February 12, 2010 3:34 PM Subject: Re: Tomcat dies suddenly Carl wrote: Andre, Thanks for the response. I have read almost all of your posts and realy enjoy the way to take problems apart. Keep on thinking. I'm not quite sure how to take the above answer.. So, just in case, and maybe to my own ultimate embarassment, I want to indicate that I was serious. I seem to remember cases where an application at the point of dying, would have very much liked to log a last desperate message to indicate the situation, but did not even have the resources left to be able to do so. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Chris, I was preparing to try out your checking the jvm exit code and noticed (drum roll) that this morning's failure left a core dump in the tomcat/bin directory. Opening it with gdb -c yielded: This GDB was configured as "x86_64-slackware-linux". (no debugging symbols found) Core was generated by `/usr/local/java/bin/java -Djava.util.logging.config.file=/usr/local/tomcat/conf'. Program terminated with signal 11, Segmentation fault. This Tomcat is straight out of the box except for some modifications to JAVA_OPTS in tomcat/bin/catalina.sh and opening up ports and turning on SSL in tomcat/conf/server.xml. There is a logging.properties file in tomcat/conf. How does this line: `/usr/local/java/bin/java -Djava.util.logging.config.file=/usr/local/tomcat/conf' know that it is supposed to use the logging.properties file if that is the problem? Or, is there some other problem? Now, this is embarrassing: I just checked the other server and it also has a core file with the date and time of the last failure in the tomcat/bin directory. And, it shows a seg fault at exactly the same code. This might be a winner... we certainly know it killed the java process. Thanks, Carl - Original Message - From: "Christopher Schultz" To: "Tomcat Users List" Sent: Friday, February 12, 2010 3:01 PM Subject: Re: Tomcat dies suddenly -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Carl, On 2/12/2010 2:42 PM, Carl wrote: Great ideas (did you see Chris's response with a way of testing the exit code?) [snip] There is no native code in the application (used to do a lot of work in C and I am familiar with mayhem of buffer overruns, pointer screwups, etc.) If you get really desperate, you can also run the jvm inside of strace and get ready for a huge log file. It's possible that you'll see the jvm fail on the same function call every time, and you'll get more information about the problem. strace will show you if a signal terminated the process, or if some other call killed it (like exec(), which would sure do a number on your JVM). - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkt1s58ACgkQ9CaO5/Lv0PB8YQCgq0kuu87WVbPy0P40vFFHyPJW RUsAn1dzTKz2NTm7HAKAK7xeAWJS/2hd =2HBr -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Chuck, Darn, I thought we were onto something here but, as you suspected, the line contains a lot of parameters and was truncated. So, now, I think we know the JVM is seg faulting, we just don't know where or why. I'm guessing we have to somehow get a stack trace at the point of failure... any idea how I can get one? Or, is there a better way? Thanks, Carl - Original Message - From: "Caldarale, Charles R" To: "Tomcat Users List" Sent: Friday, February 12, 2010 4:52 PM Subject: RE: Tomcat dies suddenly From: Carl [mailto:c...@etrak-plus.com] Subject: Re: Tomcat dies suddenly How does this line: `/usr/local/java/bin/java - Djava.util.logging.config.file=/usr/local/tomcat/conf' know that it is supposed to use the logging.properties file if that is the problem? It doesn't know. The command line parameter should be: -Djava.util.logging.config.file=/usr/local/tomcat/conf/logging.properties Perhaps the display is just a truncation, since there should be several additional command line parameters beyond that one. Use JConsole to see if the full set of properties are present when Tomcat is running normally. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Konstantin, Both servers (T105 and T110) have temperature monitoring. I pulled and sent some information (don't remember how I got it) to Dell and they said the system was operating in a normal range. Also, the fact that it occurs on two different, brand new boxes makes me think it is unlikely to be the culprit. Thanks, Carl - Original Message - From: "Konstantin Kolinko" To: "Tomcat Users List" Sent: Friday, February 12, 2010 5:08 PM Subject: Re: Tomcat dies suddenly Can it be hardware? Do you have ways to monitor temperature in your box? http://www.bitwizard.nl/sig11/ Best regards, Konstantin Kolinko 2010/2/13 Carl : Chuck, Darn, I thought we were onto something here but, as you suspected, the line contains a lot of parameters and was truncated. So, now, I think we know the JVM is seg faulting, we just don't know where or why. I'm guessing we have to somehow get a stack trace at the point of failure... any idea how I can get one? Or, is there a better way? Thanks, Carl - Original Message - From: "Caldarale, Charles R" To: "Tomcat Users List" Sent: Friday, February 12, 2010 4:52 PM Subject: RE: Tomcat dies suddenly From: Carl [mailto:c...@etrak-plus.com] Subject: Re: Tomcat dies suddenly How does this line: `/usr/local/java/bin/java - Djava.util.logging.config.file=/usr/local/tomcat/conf' know that it is supposed to use the logging.properties file if that is the problem? It doesn't know. The command line parameter should be: -Djava.util.logging.config.file=/usr/local/tomcat/conf/logging.properties Perhaps the display is just a truncation, since there should be several additional command line parameters beyond that one. Use JConsole to see if the full set of properties are present when Tomcat is running normally. - Chuck - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Chris and Andre, Andre's note that it was always code that was not meant for the platform triggered a thought that it might be remnants of the jre Slackware includes in their distribution. Let me explain. I have been installing Slackware by just saying 'load everything'. Then, I would remove the jre 'package' using the package manager. My thought was what if the package manager is not removing everything? So, I am rebuilding one of the servers eliminating unwanted packages before they are installed (take less than 30 minutes... not certain how I can get a 10 minute test to see if I accomplished anything.) I agree with Chris that the only definitive way of finding the problem is to get a stack trace. It seems to me we have two stack traces that we need to know about: 1) the jvm stack trace and 2) the java stack trace. Running gdb against the core dump only tells me the problem was in the jvm because there is no debugging info in the jvm. So, the only way to get the details would seem to be to build the jvm from source (I have downloaded the source but haven't built the jvm yet.) I don't know how to force a java stack dump at point of failure, not even certain it is possible because it would seem the the failure in the C code in the jvm would mean the jvm would stop before it could give a stack trace. Understand that this is my best guess and that this area is removed from my usual mundane Java application development. If anyone has suggestions, I am open to them because I know I know very little. Thanks, Carl - Original Message - From: "Christopher Schultz" To: "Tomcat Users List" Sent: Saturday, February 13, 2010 9:46 AM Subject: Re: Tomcat dies suddenly -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 André, On 2/12/2010 7:29 PM, André Warnier wrote: I would just like to mention that in 90% of the cases where I have seen a Seg Fault, it was due to the attempted execution of a piece of binary code not meant for the current platform. (It's been a while since I've seen one though.) In a Java context, for me it's always been either misbehaving native code (/not/ from Sun... this would be "application" code), or bad hardware. Maybe another run through memtest86+ would be a good idea. I'd love to see a stack trace from a few crashes, though. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkt2u18ACgkQ9CaO5/Lv0PBGsQCgvhnXtIby1uP47o3BmjN8Hlyh USAAn1P/xLbv3tDhsTto6lWXDfwd4lM7 =xovn -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Chuck, I don't know. Memtest86 states that it has 'support for execution with multiple CPUs' but I recall that process froze when I tried it and the suggestion (from memtest86) was to use a different option. I will revisit this after I have tried rebuilding the server sans jre and building the jvm from source (so I can use it with gdb.) Thanks, Carl - Original Message - From: "Caldarale, Charles R" To: "Tomcat Users List" Sent: Saturday, February 13, 2010 10:29 AM Subject: RE: Tomcat dies suddenly From: Christopher Schultz [mailto:ch...@christopherschultz.net] Subject: Re: Tomcat dies suddenly Maybe another run through memtest86+ would be a good idea. Does memtest86+ fire up enough threads to heat up all the cores? - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Chuck, Yes, I saw that and am hoping I don't have to go down that path. But, I think that may be the only way to know for certain what is happening. Thanks, Carl - Original Message - From: "Caldarale, Charles R" To: "Tomcat Users List" Sent: Saturday, February 13, 2010 10:51 AM Subject: RE: Tomcat dies suddenly From: Carl [mailto:c...@etrak-plus.com] Subject: Re: Tomcat dies suddenly building the jvm from source Just be aware that Sun is rather explicit about the specific versions of the compilers and libraries used to build the JVM - it can be a major hassle to get it all set up right. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Andre,
I tried this and 1) I am now permamently cross eyed and 2) didn't see
anything that was out of place or looked like a binary that should not be
there.
Thanks,
Carl
- Original Message -
From: "André Warnier"
To: "Tomcat Users List"
Sent: Saturday, February 13, 2010 11:49 AM
Subject: Re: Tomcat dies suddenly
Just a quick way to check if a rogue binary hasn't crept into some
libraries directory :
find . -type f -print -exec file {} \; | more
That should give the same file type most of the time, except the rogue
module.
(the -print may be superfluous)
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Chris,
I have implemented and tested this process (if I kill the java process, I
can see that it was killed and the exit code.).
In the meantime, the system failed again. Light load this time and it was
restarted at 1:00AM this morning so it should not have had any carryover
from yesterday. The only thing I have done is to access application B
several times to see if that would trigger a failure (application A is
production.) Maybe it did and maybe it didn't. The core file showed the
same information as previous failures:
This GDB was configured as "x86_64-slackware-linux".
(no debugging symbols found)
Core was generated by
`/usr/local/java/bin/java -Djava.util.logging.config.file=/usr/local/tomcat/conf'.
Program terminated with signal 11, Segmentation fault.
I have tried attaching gdb to a currently running Tomcat and I get seg fault
after seg fault but it lets me continue. Not certain what that means but it
looks more and more like I will have to build a jvm from source with
debugging symbols. Does anyone know of a way to force a java stack dump
when the jvm seg faults?
The good news is that I have now switched to the server I built this morning
to see how it holds up (eliminated the jre from the install instead of
removing it afterwards.)
Thanks,
Carl
- Original Message -
From: "Christopher Schultz"
To: "Tomcat Users List"
Sent: Friday, February 12, 2010 10:49 AM
Subject: Re: Tomcat dies suddenly
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1
Jeffrey,
On 2/12/2010 10:31 AM, Jeffrey Janner wrote:
Also, the OP mentioned that it doesn't seem to occur if only the
production app is being hit. It takes someone using one of the other
copies first to generate the problem. So the questions occur:
1) Are you absolutely positive that there is no code that could be
calling exit()?
One way to check this would be to launch the JVM using a script that
captures the exit code of the process:
$!/bin/sh
# Grab all these variable definitions from bin/catalina.sh
"$_RUNJAVA" "$LOGGING_CONFIG" $JAVA_OPTS $CATALINA_OPTS \
-Djava.endorsed.dirs="$JAVA_ENDORSED_DIRS" -classpath "$CLASSPATH" \
-Dcatalina.base="$CATALINA_BASE" \
-Dcatalina.home="$CATALINA_HOME" \
-Djava.io.tmpdir="$CATALINA_TMPDIR" \
org.apache.catalina.startup.Bootstrap "$@" start \
>> "$CATALINA_BASE"/logs/catalina.out 2>&1
result=$?
echo "JVM existed with status code=${result}"
If it's 0, then the changes that this is due to a System.exit(0) call
are very high.
You could also try running under a SecurityManager, and prohibit
System.exit() calls.
- -chris
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.10 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
iEYEARECAAYFAkt1eKMACgkQ9CaO5/Lv0PBMIACgkZAP4UQQqVpELU2Ej4HQFOLI
7GEAn1rsUxRxvyXBrKGBYlomkaI0WN8C
=hSGU
-END PGP SIGNATURE-
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Tsirkin, I tried part of that. I brought up a server with openSuse (64 bit), the latest Sun Java and the latest Tomcat. Failed in about 15 minutes with the same indicators (no tracks in any log, didn't know to look for a core file at that time.) Could try this again and check for the core file to see if the failure is the same. I agree building from source and debugging is a very hard road. Have been trying to find a solution for almost three months and everything I have tried has failed. There appear to be only two moving parts: the operating system and the jvm (we now know the failure is when the jvm seg faults.) Maybe I should try a different jvm but I have always believed Sun's was most likely the most stable and bug free. Another option is to create a separate Tomcat for each application. This would require reworking and rethinking the applications with no guarantee of success anyway. It would seem that there is something wrong in my setup because I can't believe every 64 bit Slackware/Tomcat has failed as we would likely see that on this list. I am certainly open to any suggestion and I appreciate your thoughts. Thanks, Carl - Original Message - From: "Tsirkin Evgeny" To: "Tomcat Users List" Sent: Saturday, February 13, 2010 1:24 PM Subject: Re: Tomcat dies suddenly Carl , At this point it is probably would be much simpler for you to just move away from Slackware . Building jvm from source ,debugging with strace - this is a very hard path . And once you find the bug - there is nothing that you can do with it. You are not going to fix jvm/libc bugs ,right? You could report it and wait for new release . Probably your best bet would be use another distro and download Sun's jvm from thier site. Evgeny So, the only way to get the details would seem to be to build the jvm from source (I have downloaded the source but haven't built the jvm yet.) I don't know how to force a java stack dump at point of failure, not even certain it is possible because it would seem the the failure in the C code in the jvm would mean the jvm would stop before it could give a stack trace. Understand that this is my best guess and that this area is removed from my usual mundane Java application development. If anyone has suggestions, I am open to them because I know I know very little. Thanks, Carl - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
The oomParachute does not seem a likely candidate for solving the issue because 1) we have never seem the memory (from JConsole or VisualJVM) fill the heap or approach the max memory in the machine (never uses swap) or come close to blowing the permGem memory. More and more it does not seem like a memory problem. Thanks, Carl - Original Message - From: "Christopher Schultz" To: "Tomcat Users List" Sent: Friday, February 12, 2010 5:28 PM Subject: Re: Tomcat dies suddenly -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 André, On 2/12/2010 3:34 PM, André Warnier wrote: Carl wrote: Andre, Thanks for the response. I have read almost all of your posts and realy enjoy the way to take problems apart. Keep on thinking. I'm not quite sure how to take the above answer.. So, just in case, and maybe to my own ultimate embarassment, I want to indicate that I was serious. I seem to remember cases where an application at the point of dying, would have very much liked to log a last desperate message to indicate the situation, but did not even have the resources left to be able to do so. Are you talking about this? http://tomcat.apache.org/tomcat-6.0-doc/config/http.html (search for "oomParachute") - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkt11hkACgkQ9CaO5/Lv0PA40gCfaqBCAz8wq2k6W3SH3gKIlF7q xMQAnjnynEOuXlosG/v2jWHVx5akaQWo =ODoh -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Chris, This is the only thing I see from gdb: r...@tomcat_liv:/# gdb -c core GNU gdb 6.8 Copyright (C) 2008 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Type "show copying" and "show warranty" for details. This GDB was configured as "x86_64-slackware-linux". (no debugging symbols found) Core was generated by `/usr/local/java/bin/java -Djava.util.logging.config.file=/usr/local/tomcat/conf'. Program terminated with signal 11, Segmentation fault. [New process 3824] [New process 4182] [New process 3811] [New process 3823] [New process 3825] [New process 4325] [New process 3849] [New process 3364] [New process 3850] [New process 3393] [New process 3851] [New process 3395] [New process 3852] [New process 3399] [New process 3401] [New process 3853] [New process 3406] [New process 3859] [New process 3860] [New process 3861] [New process 3862] [New process 3863] [New process 3410] [New process 3864] [New process 3880] [New process 3416] [New process 3939] [New process 3940] [New process 3775] [New process 3986] [New process 3780] [New process 3987] [New process 3388] [New process 4291] [New process 3387] [New process 3403] [New process 3383] [New process 3389] [New process 3396] [New process 3398] [New process 3407] [New process 3408] [New process 3409] [New process 3411] [New process 3412] [New process 3413] [New process 3414] [New process 3415] [New process 3776] [New process 3782] [New process 3818] [New process 3820] #0 0x7fe01f9d359d in ?? () I have thought the reason I am seeing nothing beyond the JVM is that the JVM has no debugging symbols. Did I miss something? Thanks, Carl - Original Message - From: "Christopher Schultz" To: "Tomcat Users List" Sent: Friday, February 12, 2010 5:27 PM Subject: Re: Tomcat dies suddenly -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Carl, On 2/12/2010 4:59 PM, Carl wrote: Darn, I thought we were onto something here but, as you suspected, the line contains a lot of parameters and was truncated. So, now, I think we know the JVM is seg faulting, we just don't know where or why. I'm guessing we have to somehow get a stack trace at the point of failure... any idea how I can get one? Or, is there a better way? I believe you can do roughly this with gdb (from memory): $ gdb core-file gdb) where (boom: your stack trace goes here) - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkt11bwACgkQ9CaO5/Lv0PBUCgCfbR2f2IXwFq7ile8biFNjsXOH opEAnj7gYfb2jfQDtwIcl5atUpyYG8au =im6P -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Chris, I find it hard to believe two brand new machines with different processors, etc. would have a hardware problem that showed itself in exactly the same way. Further, I have run memTest86 for 30 hours on one of the servers and it showed nothing (although, as Chuck pointed out, the test may not have handled the cores correctly or may not have changed the temperature sufficiently to cause the problem we are seeing.) I have not found a mem test specifically for 64 bit processors. Thanks, Carl - Original Message - From: "Christopher Schultz" To: "Tomcat Users List" Sent: Friday, February 12, 2010 5:26 PM Subject: Re: Tomcat dies suddenly -BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Carl, On 2/12/2010 4:44 PM, Carl wrote: Now, this is embarrassing: I just checked the other server and it also has a core file with the date and time of the last failure in the tomcat/bin directory. And, it shows a seg fault at exactly the same code. This might be a winner... we certainly know it killed the java process. So, this now, to me, narrows this down to two possibilities: 1. A JVM bug 2. A hardware problem That is, if you really aren't running any native code. But, if you were, it would be showing up in the code dump, right?! - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkt11YMACgkQ9CaO5/Lv0PC/ZwCcCdB3k1ARO5uhxneWilt3wkox n/4AniRJb/t3Xd9FgKQecXHN7UyFP5RQ =s/EK -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Donn,
It looks like the total files opened are less than 1,000 and the ulimit is
set to 4,096 (this was increased as a way to check if ulimit was a
problem... did not change the behavior of the system.)
We use jdbc with the commons pooling process. We follow the number of open
connections very closely (logging to catalina.out) because we have had
connection leaks (still have a small one) in the past.
We do not use LDAP.
Thanks,
Carl
- Original Message -
From: "Donn Aiken"
To: "Tomcat Users List"
Sent: Friday, February 12, 2010 4:00 PM
Subject: Re: Tomcat dies suddenly
Carl -
I did have something like this happen to me - not with Tomcat but with
another JEE container. The container would run for a while, without
incident, then suddenly simply die, with nothing in any log, and not on any
apparent time schedule.
We had some code that was manipulating LDAP that had a leak in it. For each
leaked connection, we had an open file descriptor that never went away,
until the process went away. If memory serves, we finally found it by
looking at entries in /proc/{pid of jvm}/fd, doing a bunch of find . | wc
and watching that over time.
I hope this is of some help. Good luck.
Donn
On Fri, Feb 12, 2010 at 3:46 PM, Carl wrote:
Andre,
Take my comment as a compliment because that is the way it was meant...
you
have helped a lot of people on this least and I, for one, really
appreciate
that.
I was waiting to see if someone could give me an idea how to implement
what
you remembered and, if not, then I would google around to see if I could
find it myself.
Thanks,
Carl
- Original Message - From: "André Warnier"
To: "Tomcat Users List"
Sent: Friday, February 12, 2010 3:34 PM
Subject: Re: Tomcat dies suddenly
Carl wrote:
Andre,
Thanks for the response.
I have read almost all of your posts and realy enjoy the way to take
problems apart.
Keep on thinking.
I'm not quite sure how to take the above answer..
So, just in case, and maybe to my own ultimate embarassment, I want to
indicate that I was serious. I seem to remember cases where an
application
at the point of dying, would have very much liked to log a last desperate
message to indicate the situation, but did not even have the resources
left
to be able to do so.
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
-
To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org
For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Chris, I will start the newly rebuilt server with strace tomorrow morning before anyone comes on. Hopefully, strace will yield some useful information. Thanks, Carl - Original Message - From: "Christopher Schultz" To: "Tomcat Users List" Sent: Friday, February 12, 2010 3:01 PM Subject: Re: Tomcat dies suddenly -BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Carl, On 2/12/2010 2:42 PM, Carl wrote: Great ideas (did you see Chris's response with a way of testing the exit code?) [snip] There is no native code in the application (used to do a lot of work in C and I am familiar with mayhem of buffer overruns, pointer screwups, etc.) If you get really desperate, you can also run the jvm inside of strace and get ready for a huge log file. It's possible that you'll see the jvm fail on the same function call every time, and you'll get more information about the problem. strace will show you if a signal terminated the process, or if some other call killed it (like exec(), which would sure do a number on your JVM). - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkt1s58ACgkQ9CaO5/Lv0PB8YQCgq0kuu87WVbPy0P40vFFHyPJW RUsAn1dzTKz2NTm7HAKAK7xeAWJS/2hd =2HBr -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Andre, You have the ability to boil things down to the bare essentials. 1) you never saw this issue under a previous JVM 1.5 and Tomcat version 5.5.x Correct. (Running on a P4 with 32 bit Slackware.) 2) the problem happens on two separate servers, which seems to rule out a common server hardware issue Correct. 3) it happens under different versions of Linux, which seems to rule out a problem with one particular Linux distribution Correct... Slackware and openSuse. 4) it seems to be a SegFault in the JVM, leaving a core dump but no traces in the logs. (which SegFaults in my experience happen usually when trying to execute something which is not valid executable code for the platform at hand) Anyway, it does not seem to be due to running out of some resource, nor to a hidden call to system.exit(). Correct... might be some strange code someplace but I can't find any. 5) not quite sure of this anymore, but it seems to happen also on different JVMs, which would tend to rule out a problem with a particular JVM port. No, I have only used Sun's 64 bit. Started with 1.6.0_17 and am now using 1.6.0_18. 6) it does not happen immediately, not in any obvious way related to what is being processsed, except that it seems to happen more readily under load Correct although I am leaning more towards something related to accessing applications B, C and D. Correct that it does not seem to have an issue at any particular point in the code or after some activity by a user. 7) it is obviously not a common problem with either JVM or Tomcat, or we would have had laments from others by now Correct, I think it is something specific to my setup. 8) I don't know how a Java/Tomcat webapp application could trigger a SegFault on its own, other than by having the JVM participate in it. And apparently your apps are working fine up to the moment of the sudden death, so for once they do not appear as being among the usual suspects. Correct. I can see no degradation of speed right up to the moment of failure. 9) This, in one of your earlier posts, triggered my curiosity : quote This Tomcat is straight out of the box except for some modifications to JAVA_OPTS in tomcat/bin/catalina.sh (NDLR: canonically, a better place would be setenv.sh) and opening up ports and turning on SSL in tomcat/conf/server.xml. unquote So, maybe two suggestions, taking into account that I am just making wild guesses here (but that's pretty much what everyone by now is doing too, so I don't feel too bad) : - have you tried running Tomcat from the command-line, with STDOUT/STDERR to the console ? Maybe something shows up there which doesn't show up anywhere else ? I have been starting Tomcat from startup.sh which redirects STDOUT to catalina.out and STDERR to somewhere (I will have to look at it closer.) Starting tomorrow morning, the server which will be running production (I keep the other server in reserve for failures and the old server further back just in case I can't keep up with the failures) will be running under strace to see if that gives us anything (and I will be pounding on applications B, C and D just to see if I can force a failure.) - what about this SSL ? that just seems to me a likely candidate for something that is maybe not used all the time, probably calls stuff which should be native code, and is usually provided separately from Tomcat. Can you turn it off and still be operational ? Also, if it is provided separately, it should probably be relatively "grouped" in some directory, making it easier to check if everything is as it should be. We use SSL for all communications because most of the data we handle is personal data for children. Can't really turn that off. Note also that apart from a direct hardware similarity between the servers on which it happens, another common element seems to be the place at which it happens, namely the server room. This is a long shot, but a power supply issue may also provoke hardware failures. Or if your server room is on top of a mountain, or near a particle accelerator ? (re relativistic gamma rays, dark energy and all that stuff). ;-) I am not certain but I do know I don't have to use any lights at night, I provide enough glowing (light) to see where I am going. All of servers are on UPS's which are tested periodically. Thanks for your thoughts, you have such a great way of analyzing problems. Carl - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Anthony, I have to get through this as quickly as possible and I have never been able to rig up a stress test that duplicates what I am seeing in production so I am basically using the production servers for working the problem out. When a server fails, I just redirect the traffic to the other server and try to analyze what happened. And, if I can't keep the new servers up, I just move back to the old server (thank goodness I didn't rebuild that one when the new ones seemed to work.) Thanks, Carl - Original Message - From: "Anthony J. Biacco" To: "Tomcat Users List" Sent: Saturday, February 13, 2010 4:08 PM Subject: RE: Tomcat dies suddenly If #1 is correct maybe you should just revert back until you can do more testing outside production. Of course that's only if you're not using some tomcat 6/java 1.6 specific features for your apps -Tony Sent from my Windows® phone. -Original Message- From: André Warnier Sent: Saturday, February 13, 2010 1:56 PM To: Tomcat Users List Subject: Re: Tomcat dies suddenly Carl wrote: Chris, I find it hard to believe two brand new machines with different processors, etc. would have a hardware problem that showed itself in exactly the same way. Further, I have run memTest86 for 30 hours on one of the servers and it showed nothing (although, as Chuck pointed out, the test may not have handled the cores correctly or may not have changed the temperature sufficiently to cause the problem we are seeing.) I have not found a mem test specifically for 64 bit processors. Right. After rescanning your posts (and feel free to correct any discrepancies), here is a summary : 1) you never saw this issue under a previous JVM 1.5 and Tomcat version 5.5.x 2) the problem happens on two separate servers, which seems to rule out a common server hardware issue 3) it happens under different versions of Linux, which seems to rule out a problem with one particular Linux distribution 4) it seems to be a SegFault in the JVM, leaving a core dump but no traces in the logs. (which SegFaults in my experience happen usually when trying to execute something which is not valid executable code for the platform at hand) Anyway, it does not seem to be due to running out of some resource, nor to a hidden call to system.exit(). 5) not quite sure of this anymore, but it seems to happen also on different JVMs, which would tend to rule out a problem with a particular JVM port. 6) it does not happen immediately, not in any obvious way related to what is being processsed, except that it seems to happen more readily under load 7) it is obviously not a common problem with either JVM or Tomcat, or we would have had laments from others by now 8) I don't know how a Java/Tomcat webapp application could trigger a SegFault on its own, other than by having the JVM participate in it. And apparently your apps are working fine up to the moment of the sudden death, so for once they do not appear as being among the usual suspects. 9) This, in one of your earlier posts, triggered my curiosity : quote This Tomcat is straight out of the box except for some modifications to JAVA_OPTS in tomcat/bin/catalina.sh (NDLR: canonically, a better place would be setenv.sh) and opening up ports and turning on SSL in tomcat/conf/server.xml. unquote So, maybe two suggestions, taking into account that I am just making wild guesses here (but that's pretty much what everyone by now is doing too, so I don't feel too bad) : - have you tried running Tomcat from the command-line, with STDOUT/STDERR to the console ? Maybe something shows up there which doesn't show up anywhere else ? - what about this SSL ? that just seems to me a likely candidate for something that is maybe not used all the time, probably calls stuff which should be native code, and is usually provided separately from Tomcat. Can you turn it off and still be operational ? Also, if it is provided separately, it should probably be relatively "grouped" in some directory, making it easier to check if everything is as it should be. Note also that apart from a direct hardware similarity between the servers on which it happens, another common element seems to be the place at which it happens, namely the server room. This is a long shot, but a power supply issue may also provoke hardware failures. Or if your server room is on top of a mountain, or near a particle accelerator ? (re relativistic gamma rays, dark energy and all that stuff). ;-) - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org --
Re: Re:[OT] Tomcat dies suddenly
Chuck, The cases and even power supplies are very different. The T105 is destined to be a backup server and the T110 is supposed to be the front line guy. Thanks, Carl - Original Message - From: "Caldarale, Charles R" To: "Tomcat Users List" Sent: Saturday, February 13, 2010 4:38 PM Subject: RE: Re:[OT] Tomcat dies suddenly From: André Warnier [mailto:a...@ice-sa.com] Subject: Re:[OT] Tomcat dies suddenly Read relativistic protons instead. Now you're talking about something that can do real damage. (Unlike a WIMP, which seems to be too shy to even show up at the party.) BTW, I was thinking that since the T105 and T110 were both from the same vendor and use the same case, there might be some common design factor causing these mysterious segfaults - but they're radically different on the inside (e.g., AMD vs. Intel, memory speed and type, slots). Not much in common, except perhaps the power supply. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Chuck, I am using the mysql-connector/j version 3.1.12. Interesting because the latest driver is version 5.1.11. Is this worth a shot or is it likely to just miuddy the waters? We typically have less than 20 open connections. Thanks, Carl - Original Message - From: "Caldarale, Charles R" To: "Tomcat Users List" Sent: Saturday, February 13, 2010 3:51 PM Subject: RE: Tomcat dies suddenly From: Carl [mailto:c...@etrak-plus.com] Subject: Re: Tomcat dies suddenly We use jdbc with the commons pooling process. Is your JDBC driver type 4, or does it utilize some native code? - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Chuck, I started with the default (except for Xms, Xmx and the PermSize settings) and only added the others after the failures started piling up. They are easy to remove and are not likely to be helping or hurting but may be muddying the waters. Thanks, Carl - Original Message - From: "Caldarale, Charles R" To: "Tomcat Users List" Sent: Saturday, February 13, 2010 5:39 PM Subject: RE: Tomcat dies suddenly From: Carl [mailto:c...@etrak-plus.com] Subject: Re: Tomcat dies suddenly > This Tomcat is straight out of the box except for some modifications > to JAVA_OPTS in tomcat/bin/catalina.sh Have you tried using the default GC mechanism, rather than CMS? (Sorry if I'm repeating something you've already done.) - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [OT] Tomcat dies suddenly
Jorge, If the problem was easy, there wouldn't be the number of threads. I am running the standard JVM from Sun, in fact, have tried two versions. The boxes are not the problem because the problem appears on two very different boxes. Thanks, Carl - Original Message - From: "Jorge Medina" To: "Tomcat Users List" Sent: Sunday, February 14, 2010 12:44 AM Subject: Re: [OT] Tomcat dies suddenly There have been 144 messages on this thread...and you have spent already months trying to solve the problem...I think it will be more cost effective to replace the boxes, run a standard JVM from Sun..and close this thread! On Sat, Feb 13, 2010 at 6:11 PM, Caldarale, Charles R < chuck.caldar...@unisys.com> wrote: > From: André Warnier [mailto:a...@ice-sa.com] > Subject: Re: [OT] Tomcat dies suddenly > > Maybe we should also investigate if the SegFaults are simultaneous with > anyone specific entering the room where the servers are. Ah yes, the old nylon underwear problem... Or the pizza with plutonium toppings. - Chuck THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY MATERIAL and is thus for use only by the intended recipient. If you received this in error, please contact the sender and delete the e-mail and its attachments from all computers. - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [OT] Tomcat dies suddenly
Anthony, Very certain it is not the hardware, am running Sun JVM, could be the OS. May be my next step (CentOS.) Thanks, Carl - Original Message - From: To: "Tomcat Users List" Sent: Sunday, February 14, 2010 1:01 AM Subject: Re: [OT] Tomcat dies suddenly CentOS, Sun JVM, IBM Hardware = 100% Uptime On Sat, Feb 13, 2010 at 9:44 PM, Jorge Medina wrote: There have been 144 messages on this thread...and you have spent already months trying to solve the problem...I think it will be more cost effective to replace the boxes, run a standard JVM from Sun..and close this thread! On Sat, Feb 13, 2010 at 6:11 PM, Caldarale, Charles R < chuck.caldar...@unisys.com> wrote: > > From: André Warnier [mailto:a...@ice-sa.com] > > Subject: Re: [OT] Tomcat dies suddenly > > > > Maybe we should also investigate if the SegFaults are simultaneous > > with > > anyone specific entering the room where the servers are. > > Ah yes, the old nylon underwear problem... > > Or the pizza with plutonium toppings. > > - Chuck > > > THIS COMMUNICATION MAY CONTAIN CONFIDENTIAL AND/OR OTHERWISE PROPRIETARY > MATERIAL and is thus for use only by the intended recipient. If you received > this in error, please contact the sender and delete the e-mail and its > attachments from all computers. > > - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Peter, That's what I thought. Thanks, Carl - Original Message - From: "Peter Crowther" To: "Tomcat Users List" Sent: Sunday, February 14, 2010 5:29 AM Subject: Re: Tomcat dies suddenly On 13 February 2010 15:29, Caldarale, Charles R wrote: Does memtest86+ fire up enough threads to heat up all the cores? No. It's a pure memory tester, and it's single-threaded so that it always knows what's adjacent to each bit. - Peter - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Mark, Thanks for the advice. I started down the 32 bit path but had difficulty with missing libraries on 64 bit Slackware. Brought up Slackware 32 bit but realized that was going back to the restrictive memory problems so stopped that path. I had thought about using a different JVM but wasn't certain which ones were truely different and not just a fork of Sun's JVM. You have clarified that for me. Thanks, Carl - Original Message - From: "Mark Thomas" To: "Tomcat Users List" Sent: Sunday, February 14, 2010 6:58 AM Subject: Re: Tomcat dies suddenly On 13/02/2010 22:23, Carl wrote: 5) not quite sure of this anymore, but it seems to happen also on different JVMs, which would tend to rule out a problem with a particular JVM port. No, I have only used Sun's 64 bit. Started with 1.6.0_17 and am now using 1.6.0_18. That is enough of a common factor in all of the failures that it worth looking to see if changing it makes a difference. I'd see if you get the same results with other JVMs. The ones I'd try are: - Sun 32-bit 1.6.0_18 (see if 32/64 makes a difference) - IBM 64-bit 1.6 SR7 (see if IBM/Sun makes a difference) - Sun 64-bit 1.6.0_07 (see if an older JVM from before the large(ish) changes in 1.6.0_10 makes a difference) Mark - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [OT] Tomcat dies suddenly
George, Been there. In Brookfield (CT), under certain circumstances, when someone would use the copier (not on the same circuit), the Novell server would go down. Here, we have all new wiring directly from the fuse box with really good UPS's in between. Thanks, Carl - Original Message - From: "George Sexton" To: "'Tomcat Users List'" Sent: Sunday, February 14, 2010 11:58 AM Subject: RE: [OT] Tomcat dies suddenly -Original Message- From: André Warnier [mailto:a...@ice-sa.com] Sent: Saturday, February 13, 2010 3:28 PM To: Tomcat Users List Subject: Re: [OT] Tomcat dies suddenly Caldarale, Charles R wrote: Since we must have by now exhausted all the normal causes of such errors, maybe we should recommend a) a visual inspection of the systems, to see if there are any pinsize holes, or paint flaking off or so b) the installation of a surveilance camera, to check if the SegFaults are synchronous with any visible phenomenon (sparks, Cerenkow radiation, etc.) c) moving the systems to the basement ? There's a story in a book I once read where a computer system crashed every morning around the same time. No one could figure it out. Finally, the head of IS goes down to the computer room at the expected time. In walks a maintenance man who comes in, opens the cabinet for the computer, and plugs a floor polisher into a spare outlet in the cabinet. When the maintenance man activates the polisher, boom, the system crashed. When asked by his boss what the problem was, he told him it was a buffer problem. http://www.amazon.com/Devouring-Fungus-Jennings-Karla/dp/0393307328 I used to have a Novell server that would mysteriously reboot every few days. In my case, the server was on the same circuit as a laser printer, and both were plugged into a Haworth cubicle outlet. Periodically, load was too much and it would causes a server reboot. We brought in another circuit NOT on the inadequate cubicle wiring system, and the problem went away. George Sexton MH Software, Inc. http://www.mhsoftware.com/ Voice: 303 438 9585 - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Chris, I upgraded to 5.1.11 but can go back to 3.1.12 in about 15 minutes. Have been running all day on 5.1.11... no burps so far but I haven't been pushing it. Tomoroow, I will see if I can break it. Thanks, Carl - Original Message - From: "Christopher Schultz" To: "Tomcat Users List" Sent: Sunday, February 14, 2010 4:15 PM Subject: Re: Tomcat dies suddenly -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Carl, On 2/13/2010 5:35 PM, Carl wrote: I am using the mysql-connector/j version 3.1.12. Interesting because the latest driver is version 5.1.11. Is this worth a shot or is it likely to just miuddy the waters? We typically have less than 20 open connections. To my knowledge, MySQL's Connector/J has never had a native driver: they've all been Type 4 drivers, so native code still isn't a problem. I'd upgrade to a new version of the driver eventually, but probably not until you get everything else stable. - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkt4Z+UACgkQ9CaO5/Lv0PCaCgCaA5OMR96UMLX6BuIxQ+nGvpDV edIAniumoLaxFYzQe/LUflUFX7jDCH3D =Dt6R -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: Tomcat dies suddenly
Chris, I agree it will be several crashes (maybe never) but strace is a low risk, low investment so I am running with it. I am also using your suggestion to trap the exit code although I believe we already know what it will be. The production server that has held up all day is the T110 which I did a fresh install and elimin ated the packaged jre prior to installing the operating system. My hypothesis is that the package removal process may have left something hanging around. Thanks for the suggestions. Carl - Original Message - From: "Christopher Schultz" To: "Tomcat Users List" Sent: Sunday, February 14, 2010 4:19 PM Subject: Re: Tomcat dies suddenly -BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Carl, On 2/13/2010 4:04 PM, Carl wrote: I will start the newly rebuilt server with strace tomorrow morning before anyone comes on. Hopefully, strace will yield some useful information. You'll likely have to run your server repeatedly under strace in order to gather any meaningful data. Honestly, it would be a miracle if every crash appeared to be for the same function call. Good luck, - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkt4aM8ACgkQ9CaO5/Lv0PC7NwCeJz4rJgZ9vvgaIyNWJ39K5Zr2 XN4An3qqqONIrJNlnrUoZ00rr+uiAQ93 =GlP1 -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
Re: [OT] Tomcat dies suddenly
Chris, You are evil. Thanks, Carl - Original Message - From: "Christopher Schultz" To: "Tomcat Users List" Sent: Sunday, February 14, 2010 4:21 PM Subject: Re: [OT] Tomcat dies suddenly -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Anthony, On 2/14/2010 1:01 AM, anthonyvie...@gmail.com wrote: CentOS, Sun JVM, IBM Hardware = 100% Uptime 100%, eh? Care to make a wager? - -chris -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (MingW32) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkt4aXQACgkQ9CaO5/Lv0PDdqgCdFW5C4rWO4XwCI9/EK9APlk+K CXIAn3akAol+pBE12guiAz+krXytzV1T =w3u5 -END PGP SIGNATURE- - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org - To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org For additional commands, e-mail: users-h...@tomcat.apache.org
