we can maybe switch to an agent properties for the system vms that use a size-based roll for logging.
On Mon, Mar 10, 2014 at 5:30 PM, Anirban Chakraborty <abc...@juniper.net> wrote: > Thanks for all the responses. I do not see cloud.log and cloud.out logs are > zipped in /var/log and /var/log/cloud respectively. Only file that was zipped > was cron.log. The two largest files are: > cloud.out.2 with following: > -- > + keyvalues=' root=LABEL console=tty0 xencons=ttyS0,115200 console=hvc0 > console=hvc0 template=domP type=secstorage host=10.84.58.252 port=8250 > name=s-1-VM zone=1 pod=1 guid=s-1-VM > resource=org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource > instance=SecStorage sslcopy=true role=templateProcessor mtu=1500' > + for i in '$CMDLINE' > ++ cut -s -d= -f1 > ++ echo eth2ip=10.84.59.176 > + KEY=eth2ip > ++ cut -s -d= -f2 > ++ echo eth2ip=10.84.59.176 > + VALUE=10.84.59.176 > + '[' eth2ip == '' ']' > + case $KEY in > + keyvalues=' root=LABEL console=tty0 xencons=ttyS0,115200 console=hvc0 > console=hvc0 template=domP type=secstorage host=10.84.58.252 port=8250 > name=s-1-VM zone=1 pod=1 guid=s-1-VM resource=org.apache.cloudstack.sJava > HotSpot(TM) Client VM warning: Insufficient space for shared memory file: > /tmp/hsperfdata_root/8004 > Try using the -Djava.io.tmpdir= option to select an alternate temp location. > > log4j:WARN No appenders could be found for logger > (com.cloud.agent.AgentShell). > log4j:WARN Please initialize the log4j system properly. > log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more > info. > log4j:WARN No such property [maxFileSize] in > org.apache.log4j.rolling.RollingFileAppender. > log4j:WARN No such property [maxBackupIndex] in > org.apache.log4j.rolling.RollingFileAppender. > log4j:WARN Please set a rolling policy for the RollingFileAppender named > 'FILE3' > 08:31:55,320 INFO AgentShell:318 - Agent started > log4j:ERROR Failed to flush writer, > java.io.IOException: No space left on device > -- > and, repetition of following in /var/log/cloud.log.1 > -- > 2014-03-01 07:05:08,607 DEBUG [cloud.utils.ProcessUtil] (main:null) PID TTY > TIME CMD > 3938 ? 00:00:36 java > > 2014-03-01 07:05:08,607 ERROR [cloud.agent.AgentShell] (main:null) Unable to > start agent: Java process is being started twice. If this is not true, > remove /var/run/agent.SecStorage.pid > 2014-03-01 07:05:19,028 INFO [cloud.agent.AgentShell] (main:null) Agent > started > 2014-03-01 07:05:19,030 INFO [cloud.agent.AgentShell] (main:null) > Implementation Version is 4.3.0-SNAPSHOT > 2014-03-01 07:05:19,030 INFO [cloud.agent.AgentShell] (main:null) > agent.properties found at /usr/local/cloud/systemvm/conf/agent.properties > 2014-03-01 07:05:19,038 DEBUG [cloud.agent.AgentShell] (main:null) Found > property: instance > 2014-03-01 07:05:19,038 DEBUG [cloud.agent.AgentShell] (main:null) Found > property: resource > 2014-03-01 07:05:19,038 INFO [cloud.agent.AgentShell] (main:null) Defaulting > to using properties file for storage > 2014-03-01 07:05:19,039 INFO [cloud.agent.AgentShell] (main:null) Defaulting > to the constant time backoff algorithm > 2014-03-01 07:05:19,048 INFO [cloud.utils.LogUtils] (main:null) log4j > configuration found at /usr/local/cloud/systemvm/conf/log4j-cloud.xml > 2014-03-01 07:05:19,062 DEBUG [cloud.agent.AgentShell] (main:null) Checking > to see if agent.SecStorage.pid exists. > 2014-03-01 07:05:19,064 DEBUG [cloud.utils.ProcessUtil] (main:null) > environment.properties could not be opened > 2014-03-01 07:05:19,071 DEBUG [cloud.utils.ProcessUtil] (main:null) > Executing: bash -c ps -p 3938 > 2014-03-01 07:05:19,077 DEBUG [cloud.utils.ProcessUtil] (main:null) Execution > is successful. > > eventually the log gets full with, > 2014-03-10 09:54:09,420 INFO [utils.nio.NioClient] (Agent-Selector:null) > Connecting to 10.84.58.252:8250 > 2014-03-10 09:54:36,916 WARN [utils.nio.NioConnection] (Agent-Selector:null) > Unable to connect to remote: is there a server running on port 8250 > -- > Looks like the agent on s-1-vm could not connect to the management server at > some point of time after the system vm startup and the log gets filled with > above 'Unable to connect' messages. > > Anirban > > On Mar 7, 2014, at 11:37 PM, Rajesh Battala <rajesh.batt...@citrix.com> wrote: > >> AFAIK, log roation is enabled in the systemvm. >> Can you check whether the logs are getting zipped .? >> >> -----Original Message----- >> From: Anirban Chakraborty [mailto:abc...@juniper.net] >> Sent: Saturday, March 8, 2014 12:46 PM >> To: dev@cloudstack.apache.org >> Subject: system vm disk space issue in ACS 4.3 >> >> Hi All, >> >> I am seeing system vm disk has no space left after running for few days. >> Cloudstack UI shows the agent in v-2-VM in alert state, while agent state of >> s-1-VM shows blank (hyphen in the UI). >> Both the system vms are running and ssh-able from the host. The log in >> s-1-Vm shows following errors: >> >> root@s-1-VM:~# grep 'Exception' /var/log/cloud/*.* >> /var/log/cloud/cloud.out.2:java.io.IOException: No space left on device >> /var/log/cloud/cloud.out.2:java.io.IOException: No space left on device >> >> whereas logs in v-1-VM shows >> /var/log/cloud/cloud.out.3:java.io.IOException: No space left on device >> /var/log/cloud/cloud.out.3:java.io.IOException: No space left on device >> /var/log/cloud/cloud.out.3:07:18:00,547 INFO CSExceptionErrorCode:87 - >> Could not find exception: com.cloud.exception.AgentControlChannelException >> in error code list for exceptions >> /var/log/cloud/cloud.out.3:com.cloud.exception.AgentControlChannelException: >> Unable to post agent control request as link is not available >> >> Looks like cloud agent is filling up the log, which is leading to the disk >> full state. >> >> Is this a known issue? Thanks. >> >> Anirban >> >> > >
