This is a product bug and needs to be fixed. There is some serialization issue
that needs fixing.
In the below code volumeToFiler is null as Map is not a serializable type. It
needs to be replaced with getVolumeToFilerAsList() and then the data needs to
be read appropriately.
final Map<VolumeTO, StorageFilerTO> volumeToFiler =
command.getVolumeToFiler();
-Koushik
On 24/05/16, 3:23 PM, "Ivan Derbenev" <[email protected]> wrote:
>Hello!
>After the migration onto XS 6.5 (and installing all the updates) and CS 4.8 I
>can't migrate VMs with Local Storage
>
>The issue is similar to this:
>https://issues.apache.org/jira/browse/CLOUDSTACK-9353, but it doesn't have any
>solutions
>Any ideas?
>
>Errors:
>
>2016-05-24 12:10:16,440 DEBUG [c.c.v.VirtualMachinePowerStateSyncImpl]
>(DirectAgentCronJob-26:ctx-ab4250ff) (logid:4fa92c13) Done with process of VM
>state report. host: 136592
>2016-05-24 12:10:16,450 DEBUG [c.c.h.x.r.CitrixResourceBase]
>(DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Looking for network named
>cloud-main
>2016-05-24 12:10:16,466 DEBUG [c.c.h.x.r.XsLocalNetwork]
>(DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Found a network called
>cloud-main on host=10.7.130.103;
>Network=9632020b-4d98-1bed-9ae8-477e0993cab0;
>pif=3c7d1121-2bde-26cf-9478-18e51ffca75f
>2016-05-24 12:10:16,477 DEBUG [c.c.h.x.r.CitrixResourceBase]
>(DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Creating VLAN 1007 on host
>10.7.130.103 on device eth0
>2016-05-24 12:10:17,163 DEBUG [c.c.h.x.r.CitrixResourceBase]
>(DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) VLAN is created for 1007. The
>uuid is ec053fee-0426-9665-150a-77399562196b
>2016-05-24 12:10:17,170 WARN
>[c.c.h.x.r.w.x.XenServer610MigrateWithStorageCommandWrapper]
>(DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Catch Exception
>java.lang.NullPointerException. Storage motion failed due to
>java.lang.NullPointerException
>java.lang.NullPointerException
> at
> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86)
> at
> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54)
> at
> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
> at
> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
> at
> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
>2016-05-24 12:10:17,171 DEBUG [c.c.a.m.DirectAgentAttache]
>(DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Seq 136589-479633360314957887:
>Response Received:
>2016-05-24 12:10:17,171 DEBUG [c.c.a.t.Request] (DirectAgent-132:ctx-a97d60fa)
>(logid:8748bffe) Seq 136589-479633360314957887: Processing: { Ans: , MgmtId:
>128437433941265, via: 136589(cloudstack-computing03-sar01.tech-corps.com),
>Ver: v1, Flags: 110,
>[{"com.cloud.agent.api.MigrateWithStorageAnswer":{"result":false,"details":"Exception:
> java.lang.NullPointerException\nMessage: null\nStack:
>java.lang.NullPointerException\n\tat
>com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86)\n\tat
>
>com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54)\n\tat
>
>com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)\n\tat
>
>com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)\n\tat
>
>com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)\n\tat
>
>org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)\n\tat
>
>org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)\n\tat
>
>org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)\n\tat
>
>org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)\n\tat
>
>org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)\n\tat
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)\n\tat
>java.util.concurrent.FutureTask.run(FutureTask.java:262)\n\tat
>java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)\n\tat
>
>java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)\n\tat
>
>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)\n\tat
>
>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\n\tat
> java.lang.Thread.run(Thread.java:745)\n","wait":0}}] }
>2016-05-24 12:10:17,171 DEBUG [c.c.a.m.AgentAttache]
>(DirectAgent-132:ctx-a97d60fa) (logid:8748bffe) Seq 136589-479633360314957887:
>No more commands found
>2016-05-24 12:10:17,171 DEBUG [c.c.a.t.Request]
>(Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>(logid:8748bffe) Seq 136589-479633360314957887: Received: { Ans: , MgmtId:
>128437433941265, via: 136589(cloudstack-computing03-sar01.tech-corps.com),
>Ver: v1, Flags: 110, { MigrateWithStorageAnswer } }
>2016-05-24 12:10:17,171 ERROR [o.a.c.s.m.XenServerStorageMotionStrategy]
>(Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>(logid:8748bffe) Migration with storage of vm VM[User|i-3-271-VM] failed.
>Details: Exception: java.lang.NullPointerException
>Message: null
>Stack: java.lang.NullPointerException
> at
> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86)
> at
> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54)
> at
> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
> at
> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
> at
> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
>
>2016-05-24 12:10:17,171 ERROR [o.a.c.s.m.XenServerStorageMotionStrategy]
>(Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>(logid:8748bffe) copy failed
>com.cloud.utils.exception.CloudRuntimeException: Error while migrating the vm
>VM[User|i-3-271-VM] to host Host[-136589-Routing]. Exception:
>java.lang.NullPointerException
>Message: null
>Stack: java.lang.NullPointerException
> at
> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86)
> at
> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54)
> at
> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
> at
> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
> at
> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
>
> at
> org.apache.cloudstack.storage.motion.XenServerStorageMotionStrategy.migrateVmWithVolumesWithinCluster(XenServerStorageMotionStrategy.java:212)
> at
> org.apache.cloudstack.storage.motion.XenServerStorageMotionStrategy.copyAsync(XenServerStorageMotionStrategy.java:116)
> at
> org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:90)
> at
> org.apache.cloudstack.storage.volume.VolumeServiceImpl.migrateVolumes(VolumeServiceImpl.java:1160)
> at
> org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1002)
> at
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:2274)
> at
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:4638)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> at
> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4741)
> at
> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
>2016-05-24 12:10:17,183 WARN [o.a.c.s.d.ObjectInDataStoreManagerImpl]
>(Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>(logid:8748bffe) Unsupported data object (VOLUME,
>org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@142a92a2), no
>need to delete from object in store ref table
>2016-05-24 12:10:17,183 DEBUG [o.a.c.e.o.VolumeOrchestrator]
>(Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>(logid:8748bffe) Failed to migrated vm VM[User|i-3-271-VM] along with its
>volumes. com.cloud.utils.exception.CloudRuntimeException: Error while
>migrating the vm VM[User|i-3-271-VM] to host Host[-136589-Routing]. Exception:
>java.lang.NullPointerException
>Message: null
>Stack: java.lang.NullPointerException
> at
> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:86)
> at
> com.cloud.hypervisor.xenserver.resource.wrapper.xen610.XenServer610MigrateWithStorageCommandWrapper.execute(XenServer610MigrateWithStorageCommandWrapper.java:54)
> at
> com.cloud.hypervisor.xenserver.resource.wrapper.xenbase.CitrixRequestWrapper.execute(CitrixRequestWrapper.java:122)
> at
> com.cloud.hypervisor.xenserver.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:1677)
> at
> com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:315)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
>
>2016-05-24 12:10:17,183 INFO [c.c.v.VirtualMachineManagerImpl]
>(Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>(logid:8748bffe) Migration was unsuccessful. Cleaning up: VM[User|i-3-271-VM]
>2016-05-24 12:10:17,184 WARN [o.a.c.alerts] (Work-Job-Executor-3:ctx-601b1307
>job-4874/job-4875 ctx-619c9d4e) (logid:8748bffe) alertType:: 17 //
>dataCenterId:: 1 // podId:: 1 // clusterId:: null // message:: Unable to
>migrate vm i-3-271-VM from host cloudstack-computing01-sar01.tech-corps.com in
>zone zone1.sar and pod zone1.sar
>2016-05-24 12:10:17,185 DEBUG [c.c.a.AlertManagerImpl]
>(Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>(logid:8748bffe) Have already sent: 1 emails for alert type '17' -- skipping
>send email
>2016-05-24 12:10:17,185 DEBUG [c.c.a.m.AgentAttache]
>(Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>(logid:8748bffe) Seq 136589-479633360314957888: Routed from 128437433941265
>2016-05-24 12:10:17,185 DEBUG [c.c.a.t.Request]
>(Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>(logid:8748bffe) Seq 136589-479633360314957888: Sending { Cmd , MgmtId:
>128437433941265, via: 136589(cloudstack-computing03-sar01.tech-corps.com),
>Ver: v1, Flags: 100011,
>[{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"checkBeforeCleanup":false,"vmName":"i-3-271-VM","wait":0}}]
> }
>2016-05-24 12:10:17,185 DEBUG [c.c.a.t.Request]
>(Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>(logid:8748bffe) Seq 136589-479633360314957888: Executing: { Cmd , MgmtId:
>128437433941265, via: 136589(cloudstack-computing03-sar01.tech-corps.com),
>Ver: v1, Flags: 100011,
>[{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"checkBeforeCleanup":false,"vmName":"i-3-271-VM","wait":0}}]
> }
>2016-05-24 12:10:17,185 DEBUG [c.c.a.m.DirectAgentAttache]
>(DirectAgent-34:ctx-65ef585a) (logid:924b5b07) Seq 136589-479633360314957888:
>Executing request
>2016-05-24 12:10:17,197 DEBUG [c.c.c.CapacityManagerImpl]
>(Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>(logid:8748bffe) VM state transitted from :Migrating to Running with event:
>OperationFailedvm's original host id: 136592 new host id: 136592 host id
>before state transition: 136589
>2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl]
>(Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>(logid:8748bffe) Hosts's actual total CPU: 15960 and CPU after applying
>overprovisioning: 39900
>2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl]
>(Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>(logid:8748bffe) Hosts's actual total RAM: 11549697280 and RAM after applying
>overprovisioning: 11549697024
>2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl]
>(Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>(logid:8748bffe) release cpu from host: 136589, old used: 3990,reserved: 0,
>actual total: 15960, total with overprovisioning: 39900; new used:
>1995,reserved:0; movedfromreserved: false,moveToReserveredfalse
>2016-05-24 12:10:17,200 DEBUG [c.c.c.CapacityManagerImpl]
>(Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>(logid:8748bffe) release mem from host: 136589, old used: 5368709120,reserved:
>0, total: 11549697024; new used: 4294967296,reserved:0; movedfromreserved:
>false,moveToReserveredfalse
>2016-05-24 12:10:17,201 DEBUG [c.c.a.m.DirectAgentAttache]
>(DirectAgent-34:ctx-65ef585a) (logid:8748bffe) Seq 136589-479633360314957888:
>Response Received:
>2016-05-24 12:10:17,202 DEBUG [c.c.a.t.Request] (DirectAgent-34:ctx-65ef585a)
>(logid:8748bffe) Seq 136589-479633360314957888: Processing: { Ans: , MgmtId:
>128437433941265, via: 136589(cloudstack-computing03-sar01.tech-corps.com),
>Ver: v1, Flags: 10,
>[{"com.cloud.agent.api.StopAnswer":{"result":true,"details":"VM does not
>exist","wait":0}}] }
>2016-05-24 12:10:17,202 DEBUG [c.c.a.m.AgentAttache]
>(DirectAgent-34:ctx-65ef585a) (logid:8748bffe) Seq 136589-479633360314957888:
>Unable to find listener.
>2016-05-24 12:10:17,218 ERROR [c.c.v.VmWorkJobHandlerProxy]
>(Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>(logid:8748bffe) Invocation exception, caused by:
>com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm
>VM[User|i-3-271-VM] along with its volumes.
>2016-05-24 12:10:17,218 INFO [c.c.v.VmWorkJobHandlerProxy]
>(Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875 ctx-619c9d4e)
>(logid:8748bffe) Rethrow exception
>com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm
>VM[User|i-3-271-VM] along with its volumes.
>2016-05-24 12:10:17,218 DEBUG [c.c.v.VmWorkJobDispatcher]
>(Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe) Done
>with run of VM work job: com.cloud.vm.VmWorkMigrateWithStorage for VM 271, job
>origin: 4874
>2016-05-24 12:10:17,218 ERROR [c.c.v.VmWorkJobDispatcher]
>(Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe) Unable
>to complete AsyncJobVO {id:4875, userId: 17, accountId: 3, instanceType: null,
>instanceId: null, cmd: com.cloud.vm.VmWorkMigrateWithStorage, cmdInfo:
>rO0ABXNyACVjb20uY2xvdWQudm0uVm1Xb3JrTWlncmF0ZVdpdGhTdG9yYWdlsew9z6UxtXMCAANKAApkZXN0SG9zdElkSgAJc3JjSG9zdElkTAAMdm9sdW1lVG9Qb29sdAAPTGphdmEvdXRpbC9NYXA7eHIAE2NvbS5jbG91ZC52bS5WbVdvcmufmbZW8CVnawIABEoACWFjY291bnRJZEoABnVzZXJJZEoABHZtSWRMAAtoYW5kbGVyTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO3hwAAAAAAAAAAMAAAAAAAAAEQAAAAAAAAEPdAAZVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbAAAAAAAAhWNAAAAAAACFZBzcgARamF2YS51dGlsLkhhc2hNYXAFB9rBwxZg0QMAAkYACmxvYWRGYWN0b3JJAAl0aHJlc2hvbGR4cD9AAAAAAAAQdwgAAAAQAAAAAHg,
> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result:
>null, initMsid: 128437433941265, completeMsid: null, lastUpdated: null,
>lastPolled: null, created: Tue May 24 12:10:16 MSK 2016}, job origin:4874
>com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm
>VM[User|i-3-271-VM] along with its volumes.
> at
> org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1007)
> at
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:2274)
> at
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:4638)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> at
> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4741)
> at
> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
>2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>(Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe) Complete
>async job-4875, jobStatus: FAILED, resultCode: 0, result:
>rO0ABXNyAC9jb20uY2xvdWQudXRpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGlvbgAAAABWTU9yAwABSQALY3NFcnJvckNvZGV4cgAaamF2YS5sYW5nLlJ1bnRpbWVFeGNlcHRpb26eXwZHCjSD5QIAAHhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATamF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Rocm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3RhY2tUcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRFeGNlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwcQB-AAh0AEJGYWlsZWQgdG8gbWlncmF0ZWQgdm0gVk1bVXNlcnxpLTMtMjcxLVZNXSBhbG9uZyB3aXRoIGl0cyB2b2x1bWVzLiB1cgAeW0xqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnQ7AkYqPDz9IjkCAAB4cAAAABZzcgAbamF2YS5sYW5nLlN0YWNrVHJhY2VFbGVtZW50YQnFmiY23YUCAARJAApsaW5lTnVtYmVyTAAOZGVjbGFyaW5nQ2xhc3NxAH4ABUwACGZpbGVOYW1lcQB-AAVMAAptZXRob2ROYW1lcQB-AAV4cAAAA-90AD1vcmcuYXBhY2hlLmNsb3Vkc3RhY2suZW5naW5lLm9yY2hlc3RyYXRpb24uVm9sdW1lT3JjaGVzdHJhdG9ydAAXVm9sdW1lT3JjaGVzdHJhdG9yLmphdmF0AA5taWdyYXRlVm9sdW1lc3NxAH4ADAAACOJ0ACZjb20uY2xvdWQudm0uVmlydHVhbE1hY2hpbmVNYW5hZ2VySW1wbHQAHlZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwuamF2YXQAHW9yY2hlc3RyYXRlTWlncmF0ZVdpdGhTdG9yYWdlc3EAfgAMAAASHnEAfgAScQB-ABNxAH4AFHNxAH4ADP____50ACRzdW4ucmVmbGVjdC5OYXRpdmVNZXRob2RBY2Nlc3NvckltcGx0AB1OYXRpdmVNZXRob2RBY2Nlc3NvckltcGwuamF2YXQAB2ludm9rZTBzcQB-AAwAAAA5cQB-ABdxAH4AGHQABmludm9rZXNxAH4ADAAAACt0AChzdW4ucmVmbGVjdC5EZWxlZ2F0aW5nTWV0aG9kQWNjZXNzb3JJbXBsdAAhRGVsZWdhdGluZ01ldGhvZEFjY2Vzc29ySW1wbC5qYXZhcQB-ABtzcQB-AAwAAAJedAAYamF2YS5sYW5nLnJlZmxlY3QuTWV0aG9kdAALTWV0aG9kLmphdmFxAH4AG3NxAH4ADAAAAGt0ACJjb20uY2xvdWQudm0uVm1Xb3JrSm9iSGFuZGxlclByb3h5dAAaVm1Xb3JrSm9iSGFuZGxlclByb3h5LmphdmF0AA9oYW5kbGVWbVdvcmtKb2JzcQB-AAwAABKFcQB-ABJxAH4AE3EAfgAlc3EAfgAMAAAAZnQAIGNvbS5jbG91ZC52bS5WbVdvcmtKb2JEaXNwYXRjaGVydAAYVm1Xb3JrSm9iRGlzcGF0Y2hlci5qYXZhdAAGcnVuSm9ic3EAfgAMAAACKnQAP29yZy5hcGFjaGUuY2xvdWRzdGFjay5mcmFtZXdvcmsuam9icy5pbXBsLkFzeW5jSm9iTWFuYWdlckltcGwkNXQAGEFzeW5jSm9iTWFuYWdlckltcGwuamF2YXQADHJ1bkluQ29udGV4dHNxAH4ADAAAADF0AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUuamF2YXQAA3J1bnNxAH4ADAAAADh0AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0LmltcGwuRGVmYXVsdE1hbmFnZWRDb250ZXh0JDF0ABpEZWZhdWx0TWFuYWdlZENvbnRleHQuamF2YXQABGNhbGxzcQB-AAwAAABndABAb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5pbXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dHEAfgA1dAAPY2FsbFdpdGhDb250ZXh0c3EAfgAMAAAANXEAfgA4cQB-ADV0AA5ydW5XaXRoQ29udGV4dHNxAH4ADAAAAC50ADxvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVxAH4AMXEAfgAyc3EAfgAMAAAB9nEAfgAscQB-AC1xAH4AMnNxAH4ADAAAAdd0AC5qYXZhLnV0aWwuY29uY3VycmVudC5FeGVjdXRvcnMkUnVubmFibGVBZGFwdGVydAAORXhlY3V0b3JzLmphdmFxAH4ANnNxAH4ADAAAAQZ0AB9qYXZhLnV0aWwuY29uY3VycmVudC5GdXR1cmVUYXNrdAAPRnV0dXJlVGFzay5qYXZhcQB-ADJzcQB-AAwAAAR5dAAnamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9ydAAXVGhyZWFkUG9vbEV4ZWN1dG9yLmphdmF0AAlydW5Xb3JrZXJzcQB-AAwAAAJndAAuamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9yJFdvcmtlcnEAfgBHcQB-ADJzcQB-AAwAAALpdAAQamF2YS5sYW5nLlRocmVhZHQAC1RocmVhZC5qYXZhcQB-ADJzcgAmamF2YS51dGlsLkNvbGxlY3Rpb25zJFVubW9kaWZpYWJsZUxpc3T8DyUxteyOEAIAAUwABGxpc3RxAH4AB3hyACxqYXZhLnV0aWwuQ29sbGVjdGlvbnMkVW5tb2RpZmlhYmxlQ29sbGVjdGlvbhlCAIDLXvceAgABTAABY3QAFkxqYXZhL3V0aWwvQ29sbGVjdGlvbjt4cHNyABNqYXZhLnV0aWwuQXJyYXlMaXN0eIHSHZnHYZ0DAAFJAARzaXpleHAAAAAAdwQAAAAAeHEAfgBTeAAAEJp3CAAAAAAAAAAAeA
>2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>(Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe) Publish
>async job-4875 complete on message bus
>2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>(Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe) Wake up
>jobs related to job-4875
>2016-05-24 12:10:17,220 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>(Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe) Update
>db status for job-4875
>2016-05-24 12:10:17,221 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>(Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe) Wake up
>jobs joined with job-4875 and disjoin all subjobs created from job- 4875
>2016-05-24 12:10:17,230 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>(Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe) Done
>executing com.cloud.vm.VmWorkMigrateWithStorage for job-4875
>2016-05-24 12:10:17,231 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>(Work-Job-Executor-3:ctx-601b1307 job-4874/job-4875) (logid:8748bffe) Remove
>job-4875 from job monitoring
>2016-05-24 12:10:17,252 ERROR [c.c.a.ApiAsyncJobDispatcher]
>(API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Unexpected
>exception while executing
>org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVolumeCmd
>com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm
>VM[User|i-3-271-VM] along with its volumes.
> at
> org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.migrateVolumes(VolumeOrchestrator.java:1007)
> at
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:2274)
> at
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrateWithStorage(VirtualMachineManagerImpl.java:4638)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> at
> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4741)
> at
> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
>2016-05-24 12:10:17,253 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>(API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Complete async
>job-4874, jobStatus: FAILED, resultCode: 530, result:
>org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed
> to migrated vm VM[User|i-3-271-VM] along with its volumes. "}
>2016-05-24 12:10:17,254 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>(API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Publish async
>job-4874 complete on message bus
>2016-05-24 12:10:17,254 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>(API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Wake up jobs
>related to job-4874
>2016-05-24 12:10:17,254 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>(API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Update db status
>for job-4874
>2016-05-24 12:10:17,255 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>(API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Wake up jobs
>joined with job-4874 and disjoin all subjobs created from job- 4874
>2016-05-24 12:10:17,263 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>(API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Done executing
>org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVolumeCmd
>for job-4874
>2016-05-24 12:10:17,263 INFO [o.a.c.f.j.i.AsyncJobMonitor]
>(API-Job-Executor-3:ctx-6ceed1a1 job-4874) (logid:8748bffe) Remove job-4874
>from job monitoring
>^C
>
>
>Regards,
>IT engineer
>Farheap, Russia
>Ivan Derbenev
>
DISCLAIMER
==========
This e-mail may contain privileged and confidential information which is the
property of Accelerite, a Persistent Systems business. It is intended only for
the use of the individual or entity to which it is addressed. If you are not
the intended recipient, you are not authorized to read, retain, copy, print,
distribute or use this message. If you have received this communication in
error, please notify the sender and delete all copies of this message.
Accelerite, a Persistent Systems business does not accept any liability for
virus infected mails.
