Problem when use Ceph 0.56.4 as primary storage in CloudStack 3.0.2, ssvm
and cpvm can not start up in normal. If i should migartion cs to 4.0? as i
know cs 4.0 support ceph RBD.
*
*
*Environment:*
Server 1: Cloudstack 3.0.2 in Redhat RHEL 6.2 Kernel 2.6.32
Server 2: KVM in Redhat RHEL 6.2 Kernel 2.6.32, with Ceph 0.56.4
*Ceph Mount in Server2:*
[root@RDR02S01 ceph]# df -k
Filesystem 1K-blocks Used Available Use% Mounted on
ceph-fuse 143233024 3371008 139862016 3% /cloudfs
ps
root 29885 1 0 Apr10 ? 00:01:06 ceph-fuse -m
10.0.0.41:6789/cloudfs/
[root@RDR02S01 ceph]# ll /cloudfs/zone_kvm_bas/
total 256
*Case:*
Create basic zone with kvm, use ceph for primary storage.
Mount in CS:
Type: SharedMountPoint
Path: /cloudfs/zone_kvm_bas
*Problem:*
Basic Zone created success, but ssvm and cpvm can not start up, always down
and restart.
exception in log as below:
2013-04-11 00:03:51,052 DEBUG [agent.transport.Request] (secstorage-1:null)
Seq 27-1926955018: Sending { Cmd , MgmtId: 133846799640, via: 27, Ver: v1,
Flags: 100111,
[{"storage.PrimaryStorageDownloadCommand":{"localPath":"/mnt/9fa1f231-3fda-3a0f-adeb-d8d53871243d","poolUuid":"4c5fb9ed-669f-41d7-9eba-b4213e605e29","poolId":243,"secondaryStorageUrl":"nfs://
10.0.0.2/backup/secondary_kvm
","primaryStorageUrl":"nfs://localhost/cloudfs/zone_kvm_bas","url":"nfs://
10.0.0.2/backup/secondary_kvm/template/tmpl/1/3/","format":"QCOW2","accountId":1,"name":"routing-3","wait":10800}}]
}
2013-04-11 00:03:51,057 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(consoleproxy-1:null) Cleaning up resources for the vm
VM[ConsoleProxy|v-3657-VM] in Starting state
2013-04-11 00:03:51,060 DEBUG [agent.transport.Request]
(consoleproxy-1:null) Seq 27-1926955019: Waiting for Seq 1926955018
Scheduling: { Cmd , MgmtId: 133846799640, via: 27, Ver: v1, Flags: 100111,
[{"StopCommand":{"isProxy":false,"vmName":"v-3657-VM","wait":0}}] }
2013-04-11 00:03:51,113 DEBUG [agent.transport.Request]
(AgentManager-Handler-10:null) Seq 27-1926955018: Processing: { Ans: ,
MgmtId: 133846799640, via: 27, Ver: v1, Flags: 110,
[{"Answer":{"result":false,"details":"java.lang.NullPointerException\n\tat
com.cloud.agent.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:504)\n\tat
com.cloud.agent.storage.LibvirtStorageAdaptor.getStoragePoolByUri(LibvirtStorageAdaptor.java:717)\n\tat
com.cloud.agent.storage.LibvirtStorageAdaptor.getStoragePoolByUri(LibvirtStorageAdaptor.java:655)\n\tat
com.cloud.agent.storage.KVMStoragePoolManager.getStoragePoolByURI(KVMStoragePoolManager.java:48)\n\tat
com.cloud.agent.resource.computing.LibvirtComputingResource.execute(LibvirtComputingResource.java:1692)\n\tat
com.cloud.agent.resource.computing.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:959)\n\tat
com.cloud.agent.Agent.processRequest(Agent.java:517)\n\tat
com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:827)\n\tat
com.cloud.utils.nio.Task.run(Task.java:79)\n\tat
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)\n\tat
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)\n\tat
java.lang.Thread.run(Thread.java:679)\n","wait":0}}] }
2013-04-11 00:03:51,113 WARN [cloud.vm.VirtualMachineManagerImpl]
(AgentManager-Handler-10:null) Cleanup failed due to
java.lang.NullPointerException
at
com.cloud.agent.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:504)
at
com.cloud.agent.storage.LibvirtStorageAdaptor.getStoragePoolByUri(LibvirtStorageAdaptor.java:717)
at
com.cloud.agent.storage.LibvirtStorageAdaptor.getStoragePoolByUri(LibvirtStorageAdaptor.java:655)
at
com.cloud.agent.storage.KVMStoragePoolManager.getStoragePoolByURI(KVMStoragePoolManager.java:48)
at
com.cloud.agent.resource.computing.LibvirtComputingResource.execute(LibvirtComputingResource.java:1692)
at
com.cloud.agent.resource.computing.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:959)
at com.cloud.agent.Agent.processRequest(Agent.java:517)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:827)
at com.cloud.utils.nio.Task.run(Task.java:79)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2013-04-11 00:03:51,113 DEBUG [agent.transport.Request] (secstorage-1:null)
Seq 27-1926955018: Received: { Ans: , MgmtId: 133846799640, via: 27, Ver:
v1, Flags: 110, { Answer } }
2013-04-11 00:03:51,114 DEBUG [agent.manager.AgentAttache]
(AgentManager-Handler-10:null) Seq 27-1926955019: Sending now. is current
sequence.
2013-04-11 00:03:51,114 WARN [cloud.vm.VirtualMachineManagerImpl]
(secstorage-1:null) Cleanup failed due to java.lang.NullPointerException
at
com.cloud.agent.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:504)
at
com.cloud.agent.storage.LibvirtStorageAdaptor.getStoragePoolByUri(LibvirtStorageAdaptor.java:717)
at
com.cloud.agent.storage.LibvirtStorageAdaptor.getStoragePoolByUri(LibvirtStorageAdaptor.java:655)
at
com.cloud.agent.storage.KVMStoragePoolManager.getStoragePoolByURI(KVMStoragePoolManager.java:48)
at
com.cloud.agent.resource.computing.LibvirtComputingResource.execute(LibvirtComputingResource.java:1692)
at
com.cloud.agent.resource.computing.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:959)
at com.cloud.agent.Agent.processRequest(Agent.java:517)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:827)
at com.cloud.utils.nio.Task.run(Task.java:79)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2013-04-11 00:03:51,114 DEBUG [agent.manager.AgentManagerImpl]
(secstorage-1:null) Details from executing class
com.cloud.agent.api.storage.PrimaryStorageDownloadCommand:
java.lang.NullPointerException
at
com.cloud.agent.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:504)
at
com.cloud.agent.storage.LibvirtStorageAdaptor.getStoragePoolByUri(LibvirtStorageAdaptor.java:717)
at
com.cloud.agent.storage.LibvirtStorageAdaptor.getStoragePoolByUri(LibvirtStorageAdaptor.java:655)
at
com.cloud.agent.storage.KVMStoragePoolManager.getStoragePoolByURI(KVMStoragePoolManager.java:48)
at
com.cloud.agent.resource.computing.LibvirtComputingResource.execute(LibvirtComputingResource.java:1692)
at
com.cloud.agent.resource.computing.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:959)
at com.cloud.agent.Agent.processRequest(Agent.java:517)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:827)
at com.cloud.utils.nio.Task.run(Task.java:79)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2013-04-11 00:03:51,115 ERROR [cloud.vm.VirtualMachineManagerImpl]
(secstorage-1:null) Failed to start instance
VM[SecondaryStorageVm|s-3656-VM]
java.lang.ClassCastException: com.cloud.agent.api.Answer cannot be cast to
com.cloud.agent.api.storage.PrimaryStorageDownloadAnswer
at
com.cloud.template.TemplateManagerImpl.prepareTemplateForCreate(TemplateManagerImpl.java:659)
at
com.cloud.utils.db.DatabaseCallback.intercept(DatabaseCallback.java:30)
at
com.cloud.storage.StorageManagerImpl.createVolume(StorageManagerImpl.java:3065)
at
com.cloud.storage.StorageManagerImpl.prepare(StorageManagerImpl.java:2981)
at
com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:743)
at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:461)
at
com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:454)
at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:243)
at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:670)
at
com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1265)
at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:115)
at
com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:46)
at
com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:102)
at
com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:30)
at
com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:79)
at
com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:69)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at
java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2013-04-11 00:03:51,160 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(secstorage-1:null) Cleaning up resources for the vm
VM[SecondaryStorageVm|s-3656-VM] in Starting state
2013-04-11 00:03:51,162 DEBUG [agent.transport.Request] (secstorage-1:null)
Seq 27-1926955020: Waiting for Seq 1926955019 Scheduling: { Cmd , MgmtId:
133846799640, via: 27, Ver: v1, Flags: 100111,
[{"StopCommand":{"isProxy":false,"vmName":"s-3656-VM","wait":0}}] }
2013-04-11 00:03:51,395 DEBUG [agent.transport.Request]
(AgentManager-Handler-11:null) Seq 27-1926955019: Processing: { Ans: ,
MgmtId: 133846799640, via: 27, Ver: v1, Flags: 110,
[{"StopAnswer":{"vncPort":0,"bytesSent":0,"bytesReceived":0,"result":true,"wait":0}}]
}
2013-04-11 00:03:51,395 DEBUG [cloud.vm.VirtualMachineManagerImpl]
(AgentManager-Handler-11:null) Cleanup succeeded. Details null
2013-04-11 00:03:51,395 DEBUG [agent.transport.Request]
(consoleproxy-1:null) Seq 27-1926955019: Received: { Ans: , MgmtId:
133846799640, via: 27, Ver: v1, Flags: 110, { StopAnswer } }
--
Guangjian
