Hello, I test schaduled volume snapshots and when I create in same time rules for few machines (in my case I got 5 VM's) they all fails except one.
http://pastebin.com/E60emjuJ How I can solve this? INFO [c.c.h.HighAvailabilityManagerImpl] (HA-4:ctx-125a21bc) checking health of usage server INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-49:ctx-1bf25e45 job-9783) Add job-9783 into job monitoring INFO [o.a.c.a.c.u.s.CreateSnapshotCmd] (API-Job-Executor-49:ctx-1bf25e45 job-9783 ctx-9d7e935b) VOLSS: createSnapshotCmd starts:1409047550501 WARN [c.c.u.d.Merovingian2] (API-Job-Executor-49:ctx-1bf25e45 job-9783 ctx-9d7e935b) Was unable to find lock for the key vm_instance1173 and thread id 158740505 INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-50:ctx-bb4641db job-9785) Add job-9785 into job monitoring INFO [o.a.c.a.c.u.s.CreateSnapshotCmd] (API-Job-Executor-50:ctx-bb4641db job-9785 ctx-26da1a0f) VOLSS: createSnapshotCmd starts:1409047550587 WARN [c.c.u.d.Merovingian2] (API-Job-Executor-50:ctx-bb4641db job-9785 ctx-26da1a0f) Was unable to find lock for the key vm_instance1172 and thread id 736777923 INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-51:ctx-1f378859 job-9787) Add job-9787 into job monitoring INFO [o.a.c.a.c.u.s.CreateSnapshotCmd] (API-Job-Executor-51:ctx-1f378859 job-9787 ctx-ae2be6e3) VOLSS: createSnapshotCmd starts:1409047550660 WARN [c.c.u.d.Merovingian2] (API-Job-Executor-51:ctx-1f378859 job-9787 ctx-ae2be6e3) Was unable to find lock for the key vm_instance1171 and thread id 142175984 INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-52:ctx-197114de job-9788) Add job-9788 into job monitoring INFO [o.a.c.a.c.u.s.CreateSnapshotCmd] (API-Job-Executor-52:ctx-197114de job-9788 ctx-f29734ce) VOLSS: createSnapshotCmd starts:1409047550750 INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-53:ctx-1e698b15 job-9790) Add job-9790 into job monitoring WARN [c.c.u.d.Merovingian2] (API-Job-Executor-52:ctx-197114de job-9788 ctx-f29734ce) Was unable to find lock for the key vm_instance1170 and thread id 1643232589 INFO [o.a.c.a.c.u.s.CreateSnapshotCmd] (API-Job-Executor-53:ctx-1e698b15 job-9790 ctx-97cf6519) VOLSS: createSnapshotCmd starts:1409047550795 WARN [c.c.u.d.Merovingian2] (API-Job-Executor-53:ctx-1e698b15 job-9790 ctx-97cf6519) Was unable to find lock for the key vm_instance1168 and thread id 1037433024 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-68:ctx-6e51f025 job-9783/job-9784) Add job-9784 into job monitoring INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-69:ctx-cd90b5b2 job-9785/job-9786) Add job-9786 into job monitoring INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-70:ctx-d44cfd70 job-9787/job-9789) Add job-9789 into job monitoring INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-71:ctx-cf2e5ec2 job-9788/job-9791) Add job-9791 into job monitoring INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-72:ctx-48cef8a6 job-9790/job-9792) Add job-9792 into job monitoring WARN [o.a.c.h.x.XenServerResourceNewBase] (DirectAgent-303:ctx-c4ae6319) No event for task OpaqueRef:d7e9133b-f9bb-59e6-720c-ab4e79e0feff WARN [o.a.c.h.x.XenServerResourceNewBase] (DirectAgent-119:ctx-cac68540) No event for task OpaqueRef:04cc04cf-f676-d187-25f7-801572973d7a WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-119:ctx-cac68540) Host 10.10.10.66 OpaqueRef:18b050f6-a1db-1ba5-f290-bcd58811caad: Unable to remove SR WARN [c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-119:ctx-cac68540) BackupSnapshot Failed due to No event for task OpaqueRef:04cc04cf-f676-d187-25f7-801572973d7a java.util.concurrent.TimeoutException: No event for task OpaqueRef:04cc04cf-f676-d187-25f7-801572973d7a at org.apache.cloudstack.hypervisor.xenserver.XenServerResourceNewBase.waitForTask(XenServerResourceNewBase.java:124) at com.cloud.hypervisor.xen.resource.Xenserver625StorageProcessor.backupSnapshot(Xenserver625StorageProcessor.java:430) at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:94) at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:52) at com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:546) at com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:61) at com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:102) at com.cloud.hypervisor.xen.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65) at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:216) at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-70:ctx-d44cfd70 job-9787/job-9789) Remove job-9789 from job monitoring INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-51:ctx-1f378859 job-9787) Remove job-9787 from job monitoring WARN [c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-303:ctx-c4ae6319) BackupSnapshot Failed due to No event for task OpaqueRef:d7e9133b-f9bb-59e6-720c-ab4e79e0feff java.util.concurrent.TimeoutException: No event for task OpaqueRef:d7e9133b-f9bb-59e6-720c-ab4e79e0feff at org.apache.cloudstack.hypervisor.xenserver.XenServerResourceNewBase.waitForTask(XenServerResourceNewBase.java:124) at com.cloud.hypervisor.xen.resource.Xenserver625StorageProcessor.backupSnapshot(Xenserver625StorageProcessor.java:430) at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:94) at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:52) at com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:546) at com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:61) at com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:102) at com.cloud.hypervisor.xen.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65) at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:216) at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-68:ctx-6e51f025 job-9783/job-9784) Remove job-9784 from job monitoring INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-49:ctx-1bf25e45 job-9783) Remove job-9783 from job monitoring WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-176:ctx-830c05b9) callHostPlugin failed for cmd: mountNfsSecondaryStorage with args remoteDir: 10.10.10.137:/vol/cs_xs8_sec_vol_04082014_173817, localDir: /var/cloud_mount/543e60ad-5a48-3644-a8c5-5b3ebd4e0ae3, due to You gave an invalid session reference. It may have been invalidated by a server restart, or timed out. You should get a new session handle, using one of the session.login_ calls. This error does not invalidate the current connection. The handle parameter echoes the bad value given. You gave an invalid session reference. It may have been invalidated by a server restart, or timed out. You should get a new session handle, using one of the session.login_ calls. This error does not invalidate the current connection. The handle parameter echoes the bad value given. at com.xensource.xenapi.Types.checkResponse(Types.java:218) at com.xensource.xenapi.Connection.dispatch(Connection.java:350) at com.cloud.hypervisor.xen.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool.java:456) at com.xensource.xenapi.Event.properFrom(Event.java:310) at org.apache.cloudstack.hypervisor.xenserver.XenServerResourceNewBase.waitForTask(XenServerResourceNewBase.java:116) at com.cloud.hypervisor.xen.resource.CitrixResourceBase.callHostPluginAsync(CitrixResourceBase.java:3506) at com.cloud.hypervisor.xen.resource.Xenserver625StorageProcessor.mountNfs(Xenserver625StorageProcessor.java:68) at com.cloud.hypervisor.xen.resource.Xenserver625StorageProcessor.backupSnapshot(Xenserver625StorageProcessor.java:418) at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:94) at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:52) at com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:546) at com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:61) at com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:102) at com.cloud.hypervisor.xen.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65) at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:216) at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) WARN [c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-176:ctx-830c05b9) Could not mount secondary storage 10.10.10.137:/vol/cs_xs8_sec_vol_04082014_173817 on host /var/cloud_mount/543e60ad-5a48-3644-a8c5-5b3ebd4e0ae3 WARN [c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-176:ctx-830c05b9) BackupSnapshot Failed due to Could not mount secondary storage 10.10.10.137:/vol/cs_xs8_sec_vol_04082014_173817 on host /var/cloud_mount/543e60ad-5a48-3644-a8c5-5b3ebd4e0ae3 com.cloud.utils.exception.CloudRuntimeException: Could not mount secondary storage 10.10.10.137:/vol/cs_xs8_sec_vol_04082014_173817 on host /var/cloud_mount/543e60ad-5a48-3644-a8c5-5b3ebd4e0ae3 at com.cloud.hypervisor.xen.resource.Xenserver625StorageProcessor.mountNfs(Xenserver625StorageProcessor.java:73) at com.cloud.hypervisor.xen.resource.Xenserver625StorageProcessor.backupSnapshot(Xenserver625StorageProcessor.java:418) at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:94) at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:52) at com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:546) at com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:61) at com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:102) at com.cloud.hypervisor.xen.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65) at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:216) at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-288:ctx-0efa2c48) Task failed! Task record: uuid: cd20da12-7cbc-18cf-9629-c74dd1e3fa53 nameLabel: Async.VDI.copy nameDescription: allowedOperations: [cancel] currentOperations: {} created: Tue Aug 26 13:07:15 EEST 2014 finished: Thu Jan 01 03:00:00 EET 1970 status: pending residentOn: com.xensource.xenapi.Host@40960732 progress: 0.0 type: <none/> result: errorInfo: [] otherConfig: {} subtaskOf: com.xensource.xenapi.Task@aaf13f6f subtasks: [com.xensource.xenapi.Task@1e6e981e] WARN [c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-288:ctx-0efa2c48) unable to destroy task(OpaqueRef:be099f81-ab1b-0808-6652-af015577a575) due to You gave an invalid object reference. The object may have recently been deleted. The class parameter gives the type of reference given, and the handle parameter echoes the bad value given. INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-72:ctx-48cef8a6 job-9790/job-9792) Remove job-9792 from job monitoring INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-53:ctx-1e698b15 job-9790) Remove job-9790 from job monitoring WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-288:ctx-0efa2c48) Host 10.10.10.66 OpaqueRef:bf007bb1-b83c-9e14-75f7-6790b52de61b: Unable to remove SR WARN [c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-288:ctx-0efa2c48) BackupSnapshot Failed due to Task failed! Task record: uuid: cd20da12-7cbc-18cf-9629-c74dd1e3fa53 nameLabel: Async.VDI.copy nameDescription: allowedOperations: [cancel] currentOperations: {} created: Tue Aug 26 13:07:15 EEST 2014 finished: Thu Jan 01 03:00:00 EET 1970 status: pending residentOn: com.xensource.xenapi.Host@40960732 progress: 0.0 type: <none/> result: errorInfo: [] otherConfig: {} subtaskOf: com.xensource.xenapi.Task@aaf13f6f subtasks: [com.xensource.xenapi.Task@1e6e981e] Task failed! Task record: uuid: cd20da12-7cbc-18cf-9629-c74dd1e3fa53 nameLabel: Async.VDI.copy nameDescription: allowedOperations: [cancel] currentOperations: {} created: Tue Aug 26 13:07:15 EEST 2014 finished: Thu Jan 01 03:00:00 EET 1970 status: pending residentOn: com.xensource.xenapi.Host@40960732 progress: 0.0 type: <none/> result: errorInfo: [] otherConfig: {} subtaskOf: com.xensource.xenapi.Task@aaf13f6f subtasks: [com.xensource.xenapi.Task@1e6e981e] at com.cloud.hypervisor.xen.resource.CitrixResourceBase.checkForSuccess(CitrixResourceBase.java:3293) at com.cloud.hypervisor.xen.resource.Xenserver625StorageProcessor.backupSnapshot(Xenserver625StorageProcessor.java:431) at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:94) at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:52) at com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:546) at com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:61) at com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:102) at com.cloud.hypervisor.xen.resource.XenServer620SP1Resource.executeRequest(XenServer620SP1Resource.java:65) at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:216) at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-71:ctx-cf2e5ec2 job-9788/job-9791) Remove job-9791 from job monitoring INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-52:ctx-197114de job-9788) Remove job-9788 from job monitoring WARN [o.a.c.alerts] (CapacityChecker:ctx-3116e57f) alertType:: 3 // dataCenterId:: 5 // podId:: 5 // clusterId:: null // message:: System Alert: Remaining unallocated Storage is low in cluster DEMO Cluster-1 pod Demo-Rack-1 of availability zone Demo WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-1:ctx-2ca2a925) Task (job-9785) has been pending for 168 seconds WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-1:ctx-2ca2a925) Task (job-9786) has been pending for 166 seconds WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-1:ctx-bd10b11d) Task (job-9785) has been pending for 168 seconds WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-1:ctx-bd10b11d) Task (job-9786) has been pending for 166 seconds WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-1:ctx-423be8e1) Task (job-9785) has been pending for 222 seconds WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-1:ctx-423be8e1) Task (job-9786) has been pending for 221 seconds INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-69:ctx-cd90b5b2 job-9785/job-9786) Remove job-9786 from job monitoring INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-50:ctx-bb4641db job-9785) Remove job-9785 from job monitoring
