Hi, Mārtiņš. Unfortunately CloudStack doesn't support this yet. Best regards, Kirk
On 08/26/2014 03:17 AM, Mārtiņš Jakubovičs wrote: > Hello, > > I test schaduled volume snapshots and when I create in same time rules > for few machines (in my case I got 5 VM's) they all fails except one. > > http://pastebin.com/E60emjuJ > > How I can solve this? > > INFO [c.c.h.HighAvailabilityManagerImpl] (HA-4:ctx-125a21bc) checking > health of usage server > INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-49:ctx-1bf25e45 > job-9783) Add job-9783 into job monitoring > INFO [o.a.c.a.c.u.s.CreateSnapshotCmd] > (API-Job-Executor-49:ctx-1bf25e45 job-9783 ctx-9d7e935b) VOLSS: > createSnapshotCmd starts:1409047550501 > WARN [c.c.u.d.Merovingian2] (API-Job-Executor-49:ctx-1bf25e45 job-9783 > ctx-9d7e935b) Was unable to find lock for the key vm_instance1173 and > thread id 158740505 > INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-50:ctx-bb4641db > job-9785) Add job-9785 into job monitoring > INFO [o.a.c.a.c.u.s.CreateSnapshotCmd] > (API-Job-Executor-50:ctx-bb4641db job-9785 ctx-26da1a0f) VOLSS: > createSnapshotCmd starts:1409047550587 > WARN [c.c.u.d.Merovingian2] (API-Job-Executor-50:ctx-bb4641db job-9785 > ctx-26da1a0f) Was unable to find lock for the key vm_instance1172 and > thread id 736777923 > INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-51:ctx-1f378859 > job-9787) Add job-9787 into job monitoring > INFO [o.a.c.a.c.u.s.CreateSnapshotCmd] > (API-Job-Executor-51:ctx-1f378859 job-9787 ctx-ae2be6e3) VOLSS: > createSnapshotCmd starts:1409047550660 > WARN [c.c.u.d.Merovingian2] (API-Job-Executor-51:ctx-1f378859 job-9787 > ctx-ae2be6e3) Was unable to find lock for the key vm_instance1171 and > thread id 142175984 > INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-52:ctx-197114de > job-9788) Add job-9788 into job monitoring > INFO [o.a.c.a.c.u.s.CreateSnapshotCmd] > (API-Job-Executor-52:ctx-197114de job-9788 ctx-f29734ce) VOLSS: > createSnapshotCmd starts:1409047550750 > INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-53:ctx-1e698b15 > job-9790) Add job-9790 into job monitoring > WARN [c.c.u.d.Merovingian2] (API-Job-Executor-52:ctx-197114de job-9788 > ctx-f29734ce) Was unable to find lock for the key vm_instance1170 and > thread id 1643232589 > INFO [o.a.c.a.c.u.s.CreateSnapshotCmd] > (API-Job-Executor-53:ctx-1e698b15 job-9790 ctx-97cf6519) VOLSS: > createSnapshotCmd starts:1409047550795 > WARN [c.c.u.d.Merovingian2] (API-Job-Executor-53:ctx-1e698b15 job-9790 > ctx-97cf6519) Was unable to find lock for the key vm_instance1168 and > thread id 1037433024 > INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-68:ctx-6e51f025 > job-9783/job-9784) Add job-9784 into job monitoring > INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-69:ctx-cd90b5b2 > job-9785/job-9786) Add job-9786 into job monitoring > INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-70:ctx-d44cfd70 > job-9787/job-9789) Add job-9789 into job monitoring > INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-71:ctx-cf2e5ec2 > job-9788/job-9791) Add job-9791 into job monitoring > INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-72:ctx-48cef8a6 > job-9790/job-9792) Add job-9792 into job monitoring > WARN [o.a.c.h.x.XenServerResourceNewBase] > (DirectAgent-303:ctx-c4ae6319) No event for task > OpaqueRef:d7e9133b-f9bb-59e6-720c-ab4e79e0feff > WARN [o.a.c.h.x.XenServerResourceNewBase] > (DirectAgent-119:ctx-cac68540) No event for task > OpaqueRef:04cc04cf-f676-d187-25f7-801572973d7a > WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-119:ctx-cac68540) Host > 10.10.10.66 OpaqueRef:18b050f6-a1db-1ba5-f290-bcd58811caad: Unable to > remove SR > WARN [c.c.h.x.r.XenServerStorageProcessor] > (DirectAgent-119:ctx-cac68540) BackupSnapshot Failed due to No event for > task OpaqueRef:04cc04cf-f676-d187-25f7-801572973d7a > java.util.concurrent.TimeoutException: No event for task > OpaqueRef:04cc04cf-f676-d187-25f7-801572973d7a > at > org.apache.cloudstack.hypervisor.xenserver.XenServerResourceNewBase.waitForTask(XenServerResourceNewBase.java:124) > at > com.cloud.hypervisor.xen.resource.Xenserver625StorageProcessor.backupSnapshot(Xenserver625StorageProcessor.java:430) > at > com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:94) > at > com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:52) > at > com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:546) > at > com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:61) > at > com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:102) > at > com.cloud.hypervisor.xen.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65) > at > com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:216) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-70:ctx-d44cfd70 > job-9787/job-9789) Remove job-9789 from job monitoring > INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-51:ctx-1f378859 > job-9787) Remove job-9787 from job monitoring > WARN [c.c.h.x.r.XenServerStorageProcessor] > (DirectAgent-303:ctx-c4ae6319) BackupSnapshot Failed due to No event for > task OpaqueRef:d7e9133b-f9bb-59e6-720c-ab4e79e0feff > java.util.concurrent.TimeoutException: No event for task > OpaqueRef:d7e9133b-f9bb-59e6-720c-ab4e79e0feff > at > org.apache.cloudstack.hypervisor.xenserver.XenServerResourceNewBase.waitForTask(XenServerResourceNewBase.java:124) > at > com.cloud.hypervisor.xen.resource.Xenserver625StorageProcessor.backupSnapshot(Xenserver625StorageProcessor.java:430) > at > com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:94) > at > com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:52) > at > com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:546) > at > com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:61) > at > com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:102) > at > com.cloud.hypervisor.xen.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65) > at > com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:216) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-68:ctx-6e51f025 > job-9783/job-9784) Remove job-9784 from job monitoring > INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-49:ctx-1bf25e45 > job-9783) Remove job-9783 from job monitoring > WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-176:ctx-830c05b9) > callHostPlugin failed for cmd: mountNfsSecondaryStorage with args > remoteDir: 10.10.10.137:/vol/cs_xs8_sec_vol_04082014_173817, localDir: > /var/cloud_mount/543e60ad-5a48-3644-a8c5-5b3ebd4e0ae3, due to You gave > an invalid session reference. It may have been invalidated by a server > restart, or timed out. You should get a new session handle, using one > of the session.login_ calls. This error does not invalidate the current > connection. The handle parameter echoes the bad value given. > You gave an invalid session reference. It may have been invalidated by > a server restart, or timed out. You should get a new session handle, > using one of the session.login_ calls. This error does not invalidate > the current connection. The handle parameter echoes the bad value given. > at com.xensource.xenapi.Types.checkResponse(Types.java:218) > at com.xensource.xenapi.Connection.dispatch(Connection.java:350) > at > com.cloud.hypervisor.xen.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool.java:456) > at com.xensource.xenapi.Event.properFrom(Event.java:310) > at > org.apache.cloudstack.hypervisor.xenserver.XenServerResourceNewBase.waitForTask(XenServerResourceNewBase.java:116) > at > com.cloud.hypervisor.xen.resource.CitrixResourceBase.callHostPluginAsync(CitrixResourceBase.java:3506) > at > com.cloud.hypervisor.xen.resource.Xenserver625StorageProcessor.mountNfs(Xenserver625StorageProcessor.java:68) > at > com.cloud.hypervisor.xen.resource.Xenserver625StorageProcessor.backupSnapshot(Xenserver625StorageProcessor.java:418) > at > com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:94) > at > com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:52) > at > com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:546) > at > com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:61) > at > com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:102) > at > com.cloud.hypervisor.xen.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65) > at > com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:216) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > WARN [c.c.h.x.r.XenServerStorageProcessor] > (DirectAgent-176:ctx-830c05b9) Could not mount secondary storage > 10.10.10.137:/vol/cs_xs8_sec_vol_04082014_173817 on host > /var/cloud_mount/543e60ad-5a48-3644-a8c5-5b3ebd4e0ae3 > WARN [c.c.h.x.r.XenServerStorageProcessor] > (DirectAgent-176:ctx-830c05b9) BackupSnapshot Failed due to Could not > mount secondary storage 10.10.10.137:/vol/cs_xs8_sec_vol_04082014_173817 > on host /var/cloud_mount/543e60ad-5a48-3644-a8c5-5b3ebd4e0ae3 > com.cloud.utils.exception.CloudRuntimeException: Could not mount > secondary storage 10.10.10.137:/vol/cs_xs8_sec_vol_04082014_173817 on > host /var/cloud_mount/543e60ad-5a48-3644-a8c5-5b3ebd4e0ae3 > at > com.cloud.hypervisor.xen.resource.Xenserver625StorageProcessor.mountNfs(Xenserver625StorageProcessor.java:73) > at > com.cloud.hypervisor.xen.resource.Xenserver625StorageProcessor.backupSnapshot(Xenserver625StorageProcessor.java:418) > at > com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:94) > at > com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:52) > at > com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:546) > at > com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:61) > at > com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:102) > at > com.cloud.hypervisor.xen.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65) > at > com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:216) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-288:ctx-0efa2c48) Task > failed! Task record: uuid: > cd20da12-7cbc-18cf-9629-c74dd1e3fa53 > nameLabel: Async.VDI.copy > nameDescription: > allowedOperations: [cancel] > currentOperations: {} > created: Tue Aug 26 13:07:15 EEST 2014 > finished: Thu Jan 01 03:00:00 EET 1970 > status: pending > residentOn: com.xensource.xenapi.Host@40960732 > progress: 0.0 > type: <none/> > result: > errorInfo: [] > otherConfig: {} > subtaskOf: com.xensource.xenapi.Task@aaf13f6f > subtasks: [com.xensource.xenapi.Task@1e6e981e] > > WARN [c.c.h.x.r.XenServerStorageProcessor] > (DirectAgent-288:ctx-0efa2c48) unable to destroy > task(OpaqueRef:be099f81-ab1b-0808-6652-af015577a575) due to You gave an > invalid object reference. The object may have recently been deleted. > The class parameter gives the type of reference given, and the handle > parameter echoes the bad value given. > INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-72:ctx-48cef8a6 > job-9790/job-9792) Remove job-9792 from job monitoring > INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-53:ctx-1e698b15 > job-9790) Remove job-9790 from job monitoring > WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-288:ctx-0efa2c48) Host > 10.10.10.66 OpaqueRef:bf007bb1-b83c-9e14-75f7-6790b52de61b: Unable to > remove SR > WARN [c.c.h.x.r.XenServerStorageProcessor] > (DirectAgent-288:ctx-0efa2c48) BackupSnapshot Failed due to Task failed! > Task record: uuid: cd20da12-7cbc-18cf-9629-c74dd1e3fa53 > nameLabel: Async.VDI.copy > nameDescription: > allowedOperations: [cancel] > currentOperations: {} > created: Tue Aug 26 13:07:15 EEST 2014 > finished: Thu Jan 01 03:00:00 EET 1970 > status: pending > residentOn: com.xensource.xenapi.Host@40960732 > progress: 0.0 > type: <none/> > result: > errorInfo: [] > otherConfig: {} > subtaskOf: com.xensource.xenapi.Task@aaf13f6f > subtasks: [com.xensource.xenapi.Task@1e6e981e] > > Task failed! Task record: uuid: > cd20da12-7cbc-18cf-9629-c74dd1e3fa53 > nameLabel: Async.VDI.copy > nameDescription: > allowedOperations: [cancel] > currentOperations: {} > created: Tue Aug 26 13:07:15 EEST 2014 > finished: Thu Jan 01 03:00:00 EET 1970 > status: pending > residentOn: com.xensource.xenapi.Host@40960732 > progress: 0.0 > type: <none/> > result: > errorInfo: [] > otherConfig: {} > subtaskOf: com.xensource.xenapi.Task@aaf13f6f > subtasks: [com.xensource.xenapi.Task@1e6e981e] > > at > com.cloud.hypervisor.xen.resource.CitrixResourceBase.checkForSuccess(CitrixResourceBase.java:3293) > at > com.cloud.hypervisor.xen.resource.Xenserver625StorageProcessor.backupSnapshot(Xenserver625StorageProcessor.java:431) > at > com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:94) > at > com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:52) > at > com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:546) > at > com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:61) > at > com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:102) > at > com.cloud.hypervisor.xen.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65) > at > com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:216) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) > at > org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) > at > org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-71:ctx-cf2e5ec2 > job-9788/job-9791) Remove job-9791 from job monitoring > INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-52:ctx-197114de > job-9788) Remove job-9788 from job monitoring > WARN [o.a.c.alerts] (CapacityChecker:ctx-3116e57f) alertType:: 3 // > dataCenterId:: 5 // podId:: 5 // clusterId:: null // message:: System > Alert: Remaining unallocated Storage is low in cluster DEMO Cluster-1 > pod Demo-Rack-1 of availability zone Demo > WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-1:ctx-2ca2a925) Task > (job-9785) has been pending for 168 seconds > WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-1:ctx-2ca2a925) Task > (job-9786) has been pending for 166 seconds > WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-1:ctx-bd10b11d) Task > (job-9785) has been pending for 168 seconds > WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-1:ctx-bd10b11d) Task > (job-9786) has been pending for 166 seconds > WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-1:ctx-423be8e1) Task > (job-9785) has been pending for 222 seconds > WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-1:ctx-423be8e1) Task > (job-9786) has been pending for 221 seconds > INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-69:ctx-cd90b5b2 > job-9785/job-9786) Remove job-9786 from job monitoring > INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-50:ctx-bb4641db > job-9785) Remove job-9785 from job monitoring >
