Thanks Rajesh. I have created a jira ticket for this https://issues.apache.org/jira/browse/CLOUDSTACK-6258. Will send in the fix for review in a couple of days.
Thanks Saurav On Wed, Mar 19, 2014 at 8:03 PM, Rajesh Battala <rajesh.batt...@citrix.com>wrote: > Can you please file a bug and send your fix for review. > > Thanks > Rajesh Battala > > -----Original Message----- > From: Saurav Lahiri [mailto:saurav.lah...@sungard.com] > Sent: Wednesday, March 19, 2014 7:20 PM > To: dev@cloudstack.apache.org > Subject: Re: system vm disk space issue in ACS 4.3 > > The problem appears to be the start function in the /etc/init.d/cloud > service for console proxy. > More specifically the following line also writes to /var/log/cloud.out > > > ---------------------------------------------------------------------------------------------------------------------------------- > (cd $CLOUD_COM_HOME/systemvm; nohup ./run.sh > /var/log/cloud/cloud.out > 2>&1 & ) > > ---------------------------------------------------------------------------------------------------------------------------------- > > since run.sh calls _run.sh and both has "set -x" enabled, in certain > situations they can keep logging messages to cloud.out without being aware > of the settings in log4j-cloud.xml > > > One way to fix that could be that run.sh and _run.sh would log to > cloud.out only if a debug flag was set to true, otherwise only the java > process would write to cloud.out and log4j would respect the settings in > log4j-cloud.xml > > > Thanks > Saurav > > > > On Mon, Mar 17, 2014 at 8:47 PM, Saurav Lahiri <saurav.lah...@sungard.com > >wrote: > > > Could it have something to do with the RollingFileAppender that is > > being used. > > The following > > rollingfileappender<http://apache-logging.6191.n7.nabble.com/RollingFi > > leAppender-not-working-consistently-td8582.html> link appears to be a > bit outdated but they more or less describe a similar problem that we are > seeing? > > > > > > On our environment that is what we have seeing for sometime on console > > proxy. The root filesystem goes full with the cloud.out.* occupying > > all the space. This happens pretty frequently and we have to regularly > > recycle the console proxy to resolve this issue. > > > > > > As seen below, cloud.out.2 should not have exceeded 10MB but it stands > > at 217MB now. > > > > drwxr-xr-x 2 root root 4.0K Mar 17 14:57 . > > drwxr-xr-x 8 root root 4.0K Mar 17 15:01 .. > > -rw-r--r-- 1 root root 0 Mar 12 18:18 api-server.log > > -rw-r--r-- 1 root root 357K Mar 17 15:06 cloud.out > > -rw-r--r-- 1 root root 2.1M Mar 17 14:56 cloud.out.1 > > -rw-r--r-- 1 root root 217M Mar 17 15:06 cloud.out.2 > > > > root@v-zzzz-VM:/var/log/cloud# lsof | grep cloud.out > > sleep 649 root 1w REG 202,1 226122291 181737 > > /var/log/cloud/cloud.out.2 > > sleep 649 root 2w REG 202,1 226122291 181737 > > /var/log/cloud/cloud.out.2 > > bash 2312 root 1w REG 202,1 226122291 181737 > > /var/log/cloud/cloud.out.2 > > bash 2312 root 2w REG 202,1 226122291 181737 > > /var/log/cloud/cloud.out.2 > > bash 2339 root 1w REG 202,1 226122291 181737 > > /var/log/cloud/cloud.out.2 > > bash 2339 root 2w REG 202,1 226122291 181737 > > /var/log/cloud/cloud.out.2 > > bash 2786 root 1w REG 202,1 226122291 181737 > > /var/log/cloud/cloud.out.2 > > bash 2786 root 2w REG 202,1 226122291 181737 > > /var/log/cloud/cloud.out.2 > > java 2805 root 1w REG 202,1 226122291 181737 > > /var/log/cloud/cloud.out.2 > > java 2805 root 2w REG 202,1 226122291 181737 > > /var/log/cloud/cloud.out.2 > > java 2805 root 116w REG 202,1 319382 181769 > > /var/log/cloud/cloud.out > > root@v-zzzz-VM:/var/log/cloud# ls -alh > > > > Thanks > > Saurav > > > > > > On Tue, Mar 11, 2014 at 7:58 AM, Chiradeep Vittal < > > chiradeep.vit...@citrix.com> wrote: > > > >> Yes, it was deliberate. I can¹t find the discussion, but it revolved > >> around a security best practice of having separate partitions for /, > >> /swap, home directories > >> > >> > >> On 3/10/14, 11:35 AM, "Marcus" <shadow...@gmail.com> wrote: > >> > >> >There have been several raised, actually regarding /var/log. As for > >> >the system vm partitioning, it was explicitly changed from single to > >> >multiple partitions last year. I have no idea why, but I generally > >> >don't file bugs without community discussion on things that seem > >> >deliberate. > >> > > >> >On Sat, Mar 8, 2014 at 11:32 AM, Marcus <shadow...@gmail.com> wrote: > >> >> Yeah, I've just seen on busy systems where even with log rotation > >> >>working properly the little space left in var after OS files is > >> >>barely enough, for example the conntrackd log on a busy VPC. We > >> >>actually ended up rolling our own system vm, the existing image > >> >>has plenty of space, its just locked up in other partitions. > >> >> > >> >> On Mar 8, 2014 8:58 AM, "Rajesh Battala" > >> >><rajesh.batt...@citrix.com> > >> >>wrote: > >> >>> > >> >>> Yes, only 435MB is available for /var . we can increase the space > >> also. > >> >>> But we need to find out the root cause which services are causing > >> >>>the /var to fill up. > >> >>> Can you please find out and post which log files are taking up > >> >>>more space in /var > >> >>> > >> >>> Thanks > >> >>> Rajesh Battala > >> >>> > >> >>> -----Original Message----- > >> >>> From: Marcus [mailto:shadow...@gmail.com] > >> >>> Sent: Saturday, March 8, 2014 8:19 PM > >> >>> To: dev@cloudstack.apache.org > >> >>> Subject: RE: system vm disk space issue in ACS 4.3 > >> >>> > >> >>> Perhaps there's a new service. I know in the past we've seen > >> >>>issues with this , specifically the conntrackd log. I think the > >> >>>cloud logs weren't getting rolled either, but I thought it was > >> >>>all fixed. > >> >>> > >> >>> There's also simply not a ton of space on /var, I wish we would > >> >>>go back to just having one partition because it orphans lots of > >> >>>free space in other filesystems. > >> >>> On Mar 8, 2014 12:37 AM, "Rajesh Battala" > >> >>><rajesh.batt...@citrix.com> > >> >>> wrote: > >> >>> > >> >>> > AFAIK, log roation is enabled in the systemvm. > >> >>> > Can you check whether the logs are getting zipped .? > >> >>> > > >> >>> > -----Original Message----- > >> >>> > From: Anirban Chakraborty [mailto:abc...@juniper.net] > >> >>> > Sent: Saturday, March 8, 2014 12:46 PM > >> >>> > To: dev@cloudstack.apache.org > >> >>> > Subject: system vm disk space issue in ACS 4.3 > >> >>> > > >> >>> > Hi All, > >> >>> > > >> >>> > I am seeing system vm disk has no space left after running for > >> >>> > few > >> >>>days. > >> >>> > Cloudstack UI shows the agent in v-2-VM in alert state, while > >> >>> > agent state of s-1-VM shows blank (hyphen in the UI). > >> >>> > Both the system vms are running and ssh-able from the host. The > >> >>> > log > >> >>>in > >> >>> > s-1-Vm shows following errors: > >> >>> > > >> >>> > root@s-1-VM:~# grep 'Exception' /var/log/cloud/*.* > >> >>> > /var/log/cloud/cloud.out.2:java.io.IOException: No space left > >> >>> > on device > >> >>> > /var/log/cloud/cloud.out.2:java.io.IOException: No space left > >> >>> > on device > >> >>> > > >> >>> > whereas logs in v-1-VM shows > >> >>> > /var/log/cloud/cloud.out.3:java.io.IOException: No space left > >> >>> > on device > >> >>> > /var/log/cloud/cloud.out.3:java.io.IOException: No space left > >> >>> > on device > >> >>> > /var/log/cloud/cloud.out.3:07:18:00,547 INFO > >> CSExceptionErrorCode:87 > >> >>> > - Could not find exception: > >> >>> > com.cloud.exception.AgentControlChannelException > >> >>> > in error code list for exceptions > >> >>> > > >> >>> > > >> > >> >>>/var/log/cloud/cloud.out.3:com.cloud.exception.AgentControlChannel > >> >>>Except > >> >>>ion: > >> >>> > Unable to post agent control request as link is not available > >> >>> > > >> >>> > Looks like cloud agent is filling up the log, which is leading > >> >>> > to > >> the > >> >>> > disk full state. > >> >>> > > >> >>> > Is this a known issue? Thanks. > >> >>> > > >> >>> > Anirban > >> >>> > > >> > >> > >> > > > >
