No problem. CS is at least running on tomcat 6.0.33 so that's good news. :)
> Subject: RE: Interesting 4.2.1. Issue... > From: a...@opencloud.net.au > To: dev@cloudstack.apache.org > Date: Fri, 4 Apr 2014 11:10:28 +0800 > > Thanks and thanks for sharing the steps > > Kind Regards > Amin > > -----Original Message----- > From: Michael Phillips [mailto:mphilli7...@hotmail.com] > Sent: Friday, 4 April 2014 11:02 AM > To: dev@cloudstack.apache.org > Subject: RE: Interesting 4.2.1. Issue... > > So I manually downloaded tomcat 6.0.33 > herehttps://cwiki.apache.org/confluence/display/CLOUDSTACK/Setting+up+CloudStack+Development+Environment+on+Linux > Then did the following1. extracted 6.0.33 to /usr/share/tomcat6.0.33. 2. > Changed symlink of /usr/share/cloudstack-managemet/bin to > /usr/share/tomcat6.0.33/bin3. Changed symlink of > /usr/share/cloudstack-management/lib to /usr/share/tomcat6.0.33/lib4. > Verified cloudstack was running tomcat 6.0.33 by creating a > tomcat_version.jsp file in /usr/share/cloudstack-management/webapps/client > code for tomcat_version.jsp can be found > herehttp://stackoverflow.com/questions/14925073/how-to-find-out-running-tomcat-version > I'll definitely let you know how it goes... > > > Subject: RE: Interesting 4.2.1. Issue... > > From: a...@opencloud.net.au > > To: dev@cloudstack.apache.org > > Date: Fri, 4 Apr 2014 10:43:35 +0800 > > > > I tried but I failed to do so, each time cloudstack attempts to install to > > go fetches the 6.0.35 from the repo, maybe you have installed it after > > installing the cloudstack, if you managed to have a running cloudstack > > version above the 6.0.33 feedback with the results. > > > > Kind Regards > > Amin > > > > -----Original Message----- > > From: Michael Phillips [mailto:mphilli7...@hotmail.com] > > Sent: Friday, 4 April 2014 10:41 AM > > To: dev@cloudstack.apache.org > > Subject: RE: Interesting 4.2.1. Issue... > > > > So did you try changing your version of tomcat? > > > > > Subject: RE: Interesting 4.2.1. Issue... > > > From: a...@opencloud.net.au > > > To: dev@cloudstack.apache.org > > > Date: Fri, 4 Apr 2014 10:35:42 +0800 > > > > > > cd /usr/share/tomcat6/bin/ > > > ./version.sh > > > > > > The output should be 6.0.33 instead of 6.0.35 > > > > > > Using CATALINA_BASE: /usr/share/tomcat6 > > > Using CATALINA_HOME: /usr/share/tomcat6 > > > Using CATALINA_TMPDIR: /usr/share/tomcat6/temp > > > Using JRE_HOME: /usr > > > Using CLASSPATH: /usr/share/tomcat6/bin/bootstrap.jar > > > Server version: Apache Tomcat/6.0.35 > > > Server built: > > > Server number: 6.0.35.0 > > > OS Name: Linux > > > OS Version: 3.11.0-18-generic > > > Architecture: amd64 > > > JVM Version: 1.6.0_30-b30 > > > JVM Vendor: Sun Microsystems Inc. > > > > > > > > > Kind Regards > > > Amin > > > > > > > > > > > > -----Original Message----- > > > From: Michael Phillips [mailto:mphilli7...@hotmail.com] > > > Sent: Friday, 4 April 2014 10:31 AM > > > To: dev@cloudstack.apache.org > > > Subject: RE: Interesting 4.2.1. Issue... > > > > > > I've put tomcat 6.0.33 on our mgmt servers. I'm going to monitor it for > > > the next few days to see if we get the error again. > > > Do you know any way way to verify the version of tomcat that's running? > > > > > > > Subject: RE: Interesting 4.2.1. Issue... > > > > From: a...@opencloud.net.au > > > > To: dev@cloudstack.apache.org > > > > Date: Thu, 3 Apr 2014 10:35:29 +0800 > > > > > > > > No we didn't, it wouldn't matter because the memory would still > > > > fill up, the problem is it opens a thread and it fails to close it > > > > so whatever you will increase soon or later the memory will fill > > > > up (if I understand right) > > > > > > > > The error in catalina is as follows: > > > > > > > > SEVERE: The web application [/client] created a ThreadLocal with key of > > > > type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@1bd66d2d]) > > > > and a value of type [com.cloud.api.SerializationContext] (value > > > > [com.cloud.api.SerializationContext@2f6baed9]) but failed to remove it > > > > when the web application was stopped. This is very likely to create a > > > > memory leak. > > > > > > > > If someone could help with this error generated in the catalina log, > > > > that would be much appreicated. > > > > > > > > Kind Regards > > > > Amin > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > From: Michael Phillips [mailto:mphilli7...@hotmail.com] > > > > Sent: Thursday, 3 April 2014 9:34 AM > > > > To: dev@cloudstack.apache.org > > > > Subject: RE: Interesting 4.2.1. Issue... > > > > > > > > A few other articles also mention setting the initial heap size "-Xms" > > > > to the same value as the heap size, to go ahead and fully commit the > > > > heap. Have you tried that? > > > > One other thing I am curious of is have you fiddled with the Perm Size > > > > "XX:Permsize" setting? > > > > > > > > > Subject: RE: Interesting 4.2.1. Issue... > > > > > From: a...@opencloud.net.au > > > > > To: dev@cloudstack.apache.org > > > > > Date: Thu, 3 Apr 2014 09:26:31 +0800 > > > > > > > > > > Believe or not!! We have tried setting them in both formats and > > > > > still the catalina log produces java heap space error > > > > > > > > > > Kind Regards > > > > > Amin > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > From: Michael Phillips [mailto:mphilli7...@hotmail.com] > > > > > Sent: Thursday, 3 April 2014 9:24 AM > > > > > To: dev@cloudstack.apache.org > > > > > Subject: RE: Interesting 4.2.1. Issue... > > > > > > > > > > This may be a silly question, but in all of the docs I am reading > > > > > online in regards to increasing the java heap size, they are > > > > > specifying it as -Xmx"size""MB" example -Xmx2048m vs -Xmx2g Is it > > > > > possible that it's not reading the 2g variable as 2GB? > > > > > > > > > > > Subject: RE: Interesting 4.2.1. Issue... > > > > > > From: a...@opencloud.net.au > > > > > > To: dev@cloudstack.apache.org > > > > > > Date: Thu, 3 Apr 2014 09:06:17 +0800 > > > > > > > > > > > > It is 6.0.35 and it still produces this error, even after > > > > > > increasing the Xmx to 4g, we have installed tomcat 6.0.33 and each > > > > > > time we install the cloudstack it does not sense the installed > > > > > > 6.0.33 and attempts to install 6.0.35 as it is dependent on it. > > > > > > Silly solution is that we scheduled a daily restart @ 2PM through a > > > > > > cron job!!!! But first you have to "killall jvsc" then restart the > > > > > > management server. > > > > > > > > > > > > What we are considering is to migrate the management server to > > > > > > CentOS 6.5 it comes with tomcat 6.0.24 and mysql 5.1, we attempted > > > > > > to restore the dump on a pilot environment and it worked. > > > > > > > > > > > > If someone else has a better solution than this would you please > > > > > > share it? > > > > > > > > > > > > Kind Regards > > > > > > Amin > > > > > > > > > > > > -----Original Message----- > > > > > > From: Michael Phillips [mailto:mphilli7...@hotmail.com] > > > > > > Sent: Thursday, 3 April 2014 5:31 AM > > > > > > To: dev@cloudstack.apache.org > > > > > > Subject: RE: Interesting 4.2.1. Issue... > > > > > > > > > > > > So I just checked my tomcat version and we are running 6.0.35 which > > > > > > must be the default that comes with Ubuntu 12.04 out of the box. > > > > > > Our memory settings are as follows: > > > > > > JAVA_OPTS="-Djava.awt.headless=true > > > > > > -Dcom.sun.management.jmxremote.port=45219 > > > > > > -Dcom.sun.management.jmxremote.authenticate=false > > > > > > -Dcom.sun.management.jmxremote.ssl=false -Xmx4g > > > > > > -XX:+HeapDumpOnOutOfMemoryError > > > > > > -XX:HeapDumpPath=/var/log/cloudstack/management/ -XX:PermSize=512M > > > > > > -XX:MaxPermSize=800m" > > > > > > So what version of tomcat are you running 6.0.35 or 6.0.33? > > > > > > > > > > > > > Subject: RE: Interesting 4.2.1. Issue... > > > > > > > From: a...@opencloud.net.au > > > > > > > To: dev@cloudstack.apache.org > > > > > > > Date: Tue, 1 Apr 2014 11:23:57 +0800 > > > > > > > > > > > > > > We have the same issue, after an upgrade from 2.2.14 to > > > > > > > 4.2.1, and during this upgrade we had upgrade from Ubuntu 10 > > > > > > > LTS to Ubuntu 12 LTS, it seems it related to tomcat 6.0.35, > > > > > > > because it is recommended to have tomcat 6.0.33 which > > > > > > > doesn't come by default with Ubuntu > > > > > > > 12.04.4 > > > > > > > > > > > > > > Kind Regards > > > > > > > Amin > > > > > > > > > > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > > From: Marcus [mailto:shadow...@gmail.com] > > > > > > > Sent: Tuesday, 1 April 2014 6:22 AM > > > > > > > To: dev@cloudstack.apache.org > > > > > > > Subject: Re: Interesting 4.2.1. Issue... > > > > > > > > > > > > > > I'm running 3 mgmt servers on 4.2.1, haven't seen any issues > > > > > > > like that. You can send along your memory settings... here's > > > > > > > what I'm > > > > > > > running: > > > > > > > > > > > > > > JAVA_OPTS="-Djava.awt.headless=true > > > > > > > -Dcom.sun.management.jmxremote.port=45219 > > > > > > > -Dcom.sun.management.jmxremote.authenticate=false > > > > > > > -Dcom.sun.management.jmxremote.ssl=false -Xmx2g > > > > > > > -XX:+HeapDumpOnOutOfMemoryError > > > > > > > -XX:HeapDumpPath=/var/log/cloudstack/management/ > > > > > > > -XX:PermSize=512M -XX:MaxPermSize=800m > > > > > > > > > > > > > > On Mon, Mar 31, 2014 at 9:33 AM, Michael Phillips > > > > > > > <mphilli7...@hotmail.com> wrote: > > > > > > > > So I have a redundant pair of management servers running on > > > > > > > > 4.2.1. At least once a day one of the management servers > > > > > > > > crashes and the log gets filled with the following messages: > > > > > > > > java.lang.OutOfMemoryError: Java heap > > > > > > > > spac0java.lang.ArrayIndexOutOfBoundsExceptioneCaused by: > > > > > > > > java.io.EOFException: SSL peer shut down incorrectlyCaused by: > > > > > > > > javax.net.ssl.SSLHandshakeException: Remote host closed > > > > > > > > connection during handshake and there are a few others. > > > > > > > > When the one management server tanks, although the other > > > > > > > > management server is up and active, it won't actually > > > > > > > > process any UI commands until the crashed server is > > > > > > > > restarted. Example of won't process any UI command is > > > > > > > > create a new instance, create a new account, etc. After > > > > > > > > doing some searching I have found that others have noticed > > > > > > > > java heap errors in 4.2.1, and the suggested fix is to > > > > > > > > increase the heap size. I am planning on increasing it > > > > > > > > from 2g to 4g, however if the problem is something like a > > > > > > > > memory leak, then increasing the heap size will just delay > > > > > > > > the inevitable. Has anyone else fixed this issue by > > > > > > > > increasing the heap size? Or what is the recommended value? > > > > > > > > **updated...as expected I increased the heap size from 2g > > > > > > > > to 4g, and it just too > > > > > > > k longer for the problem to reoccur... > > > > > > > > In my honest opinion of bigger concern is the fact that when > > > > > > > > one management server crashes the other stops functioning as > > > > > > > > well. So this begs the question of why even bother with a > > > > > > > > redundant pair of servers..Anybody else experience this issue? > > > > > > > > I would love to hear any dev guys opinion on this.... > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
