Hi Nux,
no, because qemu is on exclude list in yum (I'm using intank patched
version of Centos 6.4 with RBD support) and manually compiled libvirt
1.2.3, compiled 9m ago...
Image is created, and then... just error thrown.
I also enabled debug libvirt logging and no info there eather...
Now I'm trying to downgrade packages slowly on one node...
Really crazy - VPC VR are created just fine from same template, but Guest
VR does not work...
I'm going mad slowly...
On 15 March 2015 at 19:37, Nux! <n...@li.nux.ro> wrote:
> Hi,
>
> Has the yum update touch any of your qemu/libvirt packages?
>
> Lucian
>
> --
> Sent from the Delta quadrant using Borg technology!
>
> Nux!
> www.nux.ro
>
> ----- Original Message -----
> > From: "Andrija Panic" <andrija.pa...@gmail.com>
> > To: us...@cloudstack.apache.org, dev@cloudstack.apache.org
> > Sent: Sunday, 15 March, 2015 16:19:07
> > Subject: [URGENT - HELP NEEDED]
>
> > Hi guys,
> >
> > we have updated the cloudstack from 4.3.0 to 4.3.2 (OS updated right
> before
> > that, from CentOS 6.5 to CentOS 6.6)
> >
> > And now I can not start SYSTEM VR - that is used for SHARED GUEST network
> > anymore.
> > And some VMs are down - and cant be started because they depend on this
> > VR...
> >
> > VPC VRs are created fine, so new VR for VPC are created fine, but this
> one
> > fro Guest network fails to start:
> >
> > Here you can see, after agent copies template from secondary storage, to
> > primary local storage, it created base image, and backing file - so
> storage
> > setup seems completed.
> >
> > Than all out of sudden we have errors:
> >
> > can't find connection: KVM, for vm: r-2281-VM, continue
> > 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
> > (agentRequest-Handler-3:null) can't find connection: LXC, for vm:
> > r-2281-VM, continue
> > 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
> > (agentRequest-Handler-3:null) can't find which hypervisor the vm used ,
> > then use the default hypervisor
> > 2015-03-15 16:52:45,678 DEBUG [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-3:null) Failed to get dom xml:
> > org.libvirt.LibvirtException: Domain not found: no domain with matching
> > name 'r-2281-VM'
> >
> > - and BTW - absolutely NOTHING in the libvirt log !
> >
> >
> > Here is the full log, with all the steps, on the host that doesnt host
> any
> > other VMs (except one - so log is easy to read) - you can notice after
> 1st
> > fail, agent copies new image/backing file, and tries again with new
> image...
> >
> > And BTW, same template is of course used for both System VR and VPC
> VRs...
> >
> > 2015-03-15 16:52:30,405 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-4:null) Request:Seq 1-1823670284: { Cmd , MgmtId:
> > 161344838950, via: 1, Ver: v1, Flags: 100011,
> >
> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"template/tmpl/2/615/7b06d463-eb70-3e6f-9b12-a5f19ac76b20.qcow2","origUrl":"
> > http://www.admintweets.com/systemvm64template-4.3-kvm.qcow2.bz2
> >
> ","uuid":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":615,"format":"QCOW2","accountId":2,"checksum":"70c17810202eb89e0a7e998c38acdf0c","hvm":true,"displayText":"systemvm-kvm-4.3-ghost","imageDataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://
> > 10.44.253.13/var/lib/libvirt/SECONDARY
> >
> ","_role":"Image"}},"name":"615-2-a26a3970-a877-37c4-b74a-16e63206ba60","hypervisorType":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"origUrl":"
> > http://www.admintweets.com/systemvm64template-4.3-kvm.qcow2.bz2
> >
> ","uuid":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":615,"format":"QCOW2","accountId":2,"checksum":"70c17810202eb89e0a7e998c38acdf0c","hvm":true,"displayText":"systemvm-kvm-4.3-ghost","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617-445e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem://
> >
> 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9e-5617-445e-983f-0885a744cab7
> "}},"name":"615-2-a26a3970-a877-37c4-b74a-16e63206ba60","hypervisorType":"KVM"}},"executeInSequence":false,"options":{},"wait":10800}}]
> > }
> > 2015-03-15 16:52:30,405 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-4:null) Processing command:
> > org.apache.cloudstack.storage.command.CopyCommand
> > 2015-03-15 16:52:30,406 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > (agentRequest-Handler-4:null) createStoragePool didn't find existing
> > running pool: org.libvirt.LibvirtException: Storage pool not found: no
> > storage pool with matching uuid 'M2��N0�e���,�, need to create it
> > 2015-03-15 16:52:30,407 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > (agentRequest-Handler-4:null) Didn't find an existing storage pool
> > 274d3297-d74e-30ce-81d7-65a3ffe52cfc by UUID, checking for pools with
> > duplicate paths
> > 2015-03-15 16:52:30,407 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > (agentRequest-Handler-4:null) Checking path of existing pool
> > 1b469c9e-5617-445e-983f-0885a744cab7 against pool we want to create
> > 2015-03-15 16:52:30,409 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > (agentRequest-Handler-4:null) Checking path of existing pool
> > 93655746-a9ef-394d-95e9-6e62471dd39f against pool we want to create
> > 2015-03-15 16:52:30,410 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > (agentRequest-Handler-4:null) Checking path of existing pool
> > 5b93422e-1a66-353d-88a8-2203f79b1dc6 against pool we want to create
> > 2015-03-15 16:52:30,412 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > (agentRequest-Handler-4:null) Attempting to create storage pool
> > 274d3297-d74e-30ce-81d7-65a3ffe52cfc
> > 2015-03-15 16:52:30,427 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > (agentRequest-Handler-4:null) <pool type='netfs'>
> > <name>274d3297-d74e-30ce-81d7-65a3ffe52cfc</name>
> > <uuid>274d3297-d74e-30ce-81d7-65a3ffe52cfc</uuid>
> > <source>
> > <host name='10.44.253.13'/>
> > <dir path='/var/lib/libvirt/SECONDARY/template/tmpl/2/615'/>
> > </source>
> > <target>
> > <path>/mnt/274d3297-d74e-30ce-81d7-65a3ffe52cfc</path>
> > </target>
> > </pool>
> >
> > 2015-03-15 16:52:30,473 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > (agentRequest-Handler-4:null) <volume>
> > <name>66db63ef-7e8d-4d49-9b8a-365babd88732</name>
> > <capacity>2621440000</capacity>
> > <target>
> > <format type='qcow2'/>
> > <permissions><mode>0744</mode></permissions></target>
> > </volume>
> >
> > 2015-03-15 16:52:30,510 DEBUG [utils.script.Script]
> > (agentRequest-Handler-4:null) Executing: qemu-img info
> >
> /mnt/274d3297-d74e-30ce-81d7-65a3ffe52cfc/7b06d463-eb70-3e6f-9b12-a5f19ac76b20.qcow2
> > 2015-03-15 16:52:30,520 DEBUG [utils.script.Script]
> > (agentRequest-Handler-4:null) Execution is successful.
> > 2015-03-15 16:52:30,522 DEBUG [utils.script.Script]
> > (agentRequest-Handler-4:null) Executing: /bin/bash -c cp -f
> >
> /mnt/274d3297-d74e-30ce-81d7-65a3ffe52cfc/7b06d463-eb70-3e6f-9b12-a5f19ac76b20.qcow2
> > /var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732
> > 2015-03-15 16:52:38,339 DEBUG [utils.script.Script]
> > (agentRequest-Handler-4:null) Execution is successful.
> > 2015-03-15 16:52:38,347 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-4:null) Seq 1-1823670284: { Ans: , MgmtId:
> > 161344838950, via: 1, Ver: v1, Flags: 10,
> >
> [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":0,"format":"QCOW2","accountId":0,"hvm":false}},"result":true,"wait":0}}]
> > }
> > 2015-03-15 16:52:45,310 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-2:null) Request:Seq 1-1823670285: { Cmd , MgmtId:
> > 161344838950, via: 1, Ver: v1, Flags: 100011,
> >
> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"66db63ef-7e8d-4d49-9b8a-365babd88732","origUrl":"
> > http://www.admintweets.com/systemvm64template-4.3-kvm.qcow2.bz2
> >
> ","uuid":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":615,"format":"QCOW2","accountId":2,"checksum":"70c17810202eb89e0a7e998c38acdf0c","hvm":true,"displayText":"systemvm-kvm-4.3-ghost","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617-445e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem://
> >
> 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9e-5617-445e-983f-0885a744cab7
> >
> "}},"name":"615-2-a26a3970-a877-37c4-b74a-16e63206ba60","hypervisorType":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"72ec282b-f53d-478f-955b-3dfa275da19b","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617-445e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem://
> >
> 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9e-5617-445e-983f-0885a744cab7
> "}},"name":"ROOT-2281","size":2621440000,"volumeId":3870,"vmName":"r-2281-VM","accountId":1,"format":"QCOW2","id":3870,"deviceId":0,"hypervisorType":"KVM"}},"executeInSequence":false,"options":{},"wait":0}}]
> > }
> > 2015-03-15 16:52:45,310 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-2:null) Processing command:
> > org.apache.cloudstack.storage.command.CopyCommand
> > 2015-03-15 16:52:45,323 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > (agentRequest-Handler-2:null) <volume>
> > <name>72ec282b-f53d-478f-955b-3dfa275da19b</name>
> > <capacity>2621440000</capacity>
> > <target>
> > <format type='qcow2'/>
> > <permissions><mode>0744</mode></permissions></target>
> > </volume>
> >
> > 2015-03-15 16:52:45,362 DEBUG [utils.script.Script]
> > (agentRequest-Handler-2:null) Executing: qemu-img create -f qcow2 -b
> > /var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732
> > /var/lib/libvirt/images/72ec282b-f53d-478f-955b-3dfa275da19b
> > 2015-03-15 16:52:45,371 DEBUG [utils.script.Script]
> > (agentRequest-Handler-2:null) Execution is successful.
> > 2015-03-15 16:52:45,372 DEBUG [utils.script.Script]
> > (agentRequest-Handler-2:null) libust[28625/28625]: Warning: HOME
> > environment variable not set. Disabling LTTng-UST per-user tracing. (in
> > setup_local_apps() at lttng-ust-comm.c:305)
> > Formatting
> '/var/lib/libvirt/images/72ec282b-f53d-478f-955b-3dfa275da19b',
> > fmt=qcow2 size=2621440000
> >
> backing_file='/var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732'
> > encryption=off cluster_size=65536
> >
> > 2015-03-15 16:52:45,373 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-2:null) Seq 1-1823670285: { Ans: , MgmtId:
> > 161344838950, via: 1, Ver: v1, Flags: 10,
> >
> [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"size":2621440000,"path":"72ec282b-f53d-478f-955b-3dfa275da19b","accountId":0,"format":"QCOW2","id":0}},"result":true,"wait":0}}]
> > }
> > 2015-03-15 16:52:45,675 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-3:null) Request:Seq 1-1823670286: { Cmd , MgmtId:
> > 161344838950, via: 1, Ver: v1, Flags: 100011,
> >
> [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"r-2281-VM","wait":0}}]
> > }
> > 2015-03-15 16:52:45,676 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-3:null) Processing command:
> > com.cloud.agent.api.StopCommand
> > 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
> > (agentRequest-Handler-3:null) can't find connection: KVM, for vm:
> > r-2281-VM, continue
> > 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
> > (agentRequest-Handler-3:null) can't find connection: LXC, for vm:
> > r-2281-VM, continue
> > 2015-03-15 16:52:45,677 DEBUG [kvm.resource.LibvirtConnection]
> > (agentRequest-Handler-3:null) can't find which hypervisor the vm used ,
> > then use the default hypervisor
> > 2015-03-15 16:52:45,678 DEBUG [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-3:null) Failed to get dom xml:
> > org.libvirt.LibvirtException: Domain not found: no domain with matching
> > name 'r-2281-VM'
> > 2015-03-15 16:52:45,678 DEBUG [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-3:null) Failed to get dom xml:
> > org.libvirt.LibvirtException: Domain not found: no domain with matching
> > name 'r-2281-VM'
> > 2015-03-15 16:52:45,679 DEBUG [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-3:null) Try to stop the vm at first
> > 2015-03-15 16:52:45,679 DEBUG [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-3:null) VM r-2281-VM doesn't exist, no need to
> stop it
> > 2015-03-15 16:52:45,679 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-3:null) Seq 1-1823670286: { Ans: , MgmtId:
> > 161344838950, via: 1, Ver: v1, Flags: 10,
> > [{"com.cloud.agent.api.StopAnswer":{"result":true,"wait":0}}] }
> > 2015-03-15 16:52:46,333 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-1:null) Request:Seq 1-1823670287: { Cmd , MgmtId:
> > 161344838950, via: 1, Ver: v1, Flags: 100011,
> >
> [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"path":"66db63ef-7e8d-4d49-9b8a-365babd88732","origUrl":"
> > http://www.admintweets.com/systemvm64template-4.3-kvm.qcow2.bz2
> >
> ","uuid":"66db63ef-7e8d-4d49-9b8a-365babd88732","id":615,"format":"QCOW2","accountId":2,"checksum":"70c17810202eb89e0a7e998c38acdf0c","hvm":true,"displayText":"systemvm-kvm-4.3-ghost","imageDataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617-445e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem://
> >
> 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9e-5617-445e-983f-0885a744cab7
> >
> "}},"name":"615-2-a26a3970-a877-37c4-b74a-16e63206ba60","hypervisorType":"KVM"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"6a49f9e6-6f86-4412-ad45-a4e30b435e46","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"1b469c9e-5617-445e-983f-0885a744cab7","id":200,"poolType":"Filesystem","host":"10.44.253.10","path":"/var/lib/libvirt/images","port":0,"url":"Filesystem://
> >
> 10.44.253.10//var/lib/libvirt/images/?ROLE=Primary&STOREUUID=1b469c9e-5617-445e-983f-0885a744cab7
> "}},"name":"ROOT-2281","size":2621440000,"volumeId":3871,"vmName":"r-2281-VM","accountId":1,"format":"QCOW2","id":3871,"deviceId":0,"hypervisorType":"KVM"}},"executeInSequence":false,"options":{},"wait":0}}]
> > }
> > 2015-03-15 16:52:46,333 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-1:null) Processing command:
> > org.apache.cloudstack.storage.command.CopyCommand
> > 2015-03-15 16:52:46,343 DEBUG [kvm.storage.LibvirtStorageAdaptor]
> > (agentRequest-Handler-1:null) <volume>
> > <name>6a49f9e6-6f86-4412-ad45-a4e30b435e46</name>
> > <capacity>2621440000</capacity>
> > <target>
> > <format type='qcow2'/>
> > <permissions><mode>0744</mode></permissions></target>
> > </volume>
> >
> > 2015-03-15 16:52:46,376 DEBUG [utils.script.Script]
> > (agentRequest-Handler-1:null) Executing: qemu-img create -f qcow2 -b
> > /var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732
> > /var/lib/libvirt/images/6a49f9e6-6f86-4412-ad45-a4e30b435e46
> > 2015-03-15 16:52:46,385 DEBUG [utils.script.Script]
> > (agentRequest-Handler-1:null) Execution is successful.
> > 2015-03-15 16:52:46,386 DEBUG [utils.script.Script]
> > (agentRequest-Handler-1:null) libust[28634/28634]: Warning: HOME
> > environment variable not set. Disabling LTTng-UST per-user tracing. (in
> > setup_local_apps() at lttng-ust-comm.c:305)
> > Formatting
> '/var/lib/libvirt/images/6a49f9e6-6f86-4412-ad45-a4e30b435e46',
> > fmt=qcow2 size=2621440000
> >
> backing_file='/var/lib/libvirt/images/66db63ef-7e8d-4d49-9b8a-365babd88732'
> > encryption=off cluster_size=65536
> >
> > 2015-03-15 16:52:46,387 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-1:null) Seq 1-1823670287: { Ans: , MgmtId:
> > 161344838950, via: 1, Ver: v1, Flags: 10,
> >
> [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"newData":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"size":2621440000,"path":"6a49f9e6-6f86-4412-ad45-a4e30b435e46","accountId":0,"format":"QCOW2","id":0}},"result":true,"wait":0}}]
> > }
> > 2015-03-15 16:52:46,691 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-5:null) Request:Seq 1-1823670288: { Cmd , MgmtId:
> > 161344838950, via: 1, Ver: v1, Flags: 100011,
> >
> [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"r-2281-VM","wait":0}}]
> > }
> > 2015-03-15 16:52:46,691 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-5:null) Processing command:
> > com.cloud.agent.api.StopCommand
> > 2015-03-15 16:52:46,692 DEBUG [kvm.resource.LibvirtConnection]
> > (agentRequest-Handler-5:null) can't find connection: KVM, for vm:
> > r-2281-VM, continue
> > 2015-03-15 16:52:46,693 DEBUG [kvm.resource.LibvirtConnection]
> > (agentRequest-Handler-5:null) can't find connection: LXC, for vm:
> > r-2281-VM, continue
> > 2015-03-15 16:52:46,693 DEBUG [kvm.resource.LibvirtConnection]
> > (agentRequest-Handler-5:null) can't find which hypervisor the vm used ,
> > then use the default hypervisor
> > 2015-03-15 16:52:46,694 DEBUG [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-5:null) Failed to get dom xml:
> > org.libvirt.LibvirtException: Domain not found: no domain with matching
> > name 'r-2281-VM'
> > 2015-03-15 16:52:46,694 DEBUG [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-5:null) Failed to get dom xml:
> > org.libvirt.LibvirtException: Domain not found: no domain with matching
> > name 'r-2281-VM'
> > 2015-03-15 16:52:46,694 DEBUG [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-5:null) Try to stop the vm at first
> > 2015-03-15 16:52:46,695 DEBUG [kvm.resource.LibvirtComputingResource]
> > (agentRequest-Handler-5:null) VM r-2281-VM doesn't exist, no need to
> stop it
> > 2015-03-15 16:52:46,695 DEBUG [cloud.agent.Agent]
> > (agentRequest-Handler-5:null) Seq 1-1823670288: { Ans: , MgmtId:
> > 161344838950, via: 1, Ver: v1, Flags: 10,
> > [{"com.cloud.agent.api.StopAnswer":{"result":true,"wait":0}}] }
> >
> >
> >
> >
> >
> > --
> >
> > Andrija Panić
>
--
Andrija Panić
