Hi, You right, it looking like bug in libvirt 0.9.4-23.el6_2.4 I get error from libvirt when call the "pool-create"
Workaround before upgrade libvirt, is unmount mounted "sec. storage" and re-create it again. Then, libvirt can create the new pool's -- Serg On Jun 6, 2012, at 1:14 AM, Edison Su wrote: > Sounds like your libvirt can’t create a storage pool. Can you try the > following command: > 1. > cat >> storage.xml << EOF > <pool type='netfs'> > <name>304424d9-db34-40d9-b0a7-3bd689b0b862</name> > <uuid>304424d9-db34-40d9-b0a7-3bd689b0b862</uuid> > <source> > <host name='NFS.DOMAIN.COM'/> > <dir path='/opt/secondary'/> > </source> > <target> > <path>/mnt/304424d9-db34-40d9-b0a7-3bd689b0b862</path> > </target> > </pool> > > 2. virsh pool-create storage.xml > 3. Any error in the above command? If you get the error like “cannot > open volume '/mnt/304424d9-db34-40d9-b0a7-3bd689b0b862/template”, can you > upgrade libvirt to > Installed Packages > Name : libvirt > Arch : x86_64 > Version : 0.9.4 > Release : 23.el6_2.8 > Size : 4.0 M > > From: Serg [mailto:kernc...@gmail.com] > Sent: Tuesday, June 05, 2012 11:48 AM > To: Edison Su > Cc: cloudstack-dev@incubator.apache.org > Subject: Re: 3.0.2 - KVM Agent problem? Please help. > > Hi , > > Thanks for your patch. It's solved the critical problem with Re-Start the > instances ! =) > ( In my setup only HA instances have shared storage ) > > But problem with snapshot's on "Running Instances" still exist > > 2012-06-05 19:15:52,853 DEBUG [cloud.agent.Agent] > (agentRequest-Handler-2:null) Request:Seq 34-77010656: { Cmd , MgmtId: > 207382744522, via: 34, Ver: v1, Flags: 100011, > [{"BackupSnapshotCommand":{"prevSnapshotUuid":"/var/lib/libvirt/images/4341f531-bef6-47b6-ae75-2b8d6de63038/i-8-403-VM_ROOT-403_20120605164038","prevBackupUuid":"/snapshots/5/8/442/i-8-403-VM_ROOT-403_20120605164038","isVolumeInactive":false,"vmName":"i-8-403-VM","snapshotId":1746,"pool":{"id":208,"uuid":"c6f1f032-5aad-4983-bc83-0ec1a6b5e32c","host":"111.111.111.111","path":"/var/lib/libvirt/images/","port":0,"type":"Filesystem"},"primaryStoragePoolNameLabel":"c6f1f032-5aad-4983-bc83-0ec1a6b5e32c","snapshotUuid":"/var/lib/libvirt/images/4341f531-bef6-47b6-ae75-2b8d6de63038/i-8-403-VM_ROOT-403_20120605170902","snapshotName":"i-8-403-VM_ROOT-403_20120605170902","secondaryStorageUrl":"nfs://NFS-URL/opt/secondary","dcId":5,"accountId":8,"volumeId":442,"volumePath":"4341f531-bef6-47b6-ae75-2b8d6de63038","wait":10800}}] > } > > Then "Job" create new directory under /mnt/ for access to template ??? > > 2012-06-05 19:15:53,558 DEBUG [agent.storage.LibvirtStorageAdaptor] > (agentRequest-Handler-2:null)org.libvirt.LibvirtException: cannot open volume > '/mnt/304424d9-db34-40d9-b0a7-3bd689b0b862/template':Is a directory > > > Full log of job : > http://pastebin.com/fE1M1BfS > > UI : Error Popup > " Created snapshot: > com.cloud.storage.SnapshotVO$$EnhancerByCGLIB$$4d81b9c7@78ab83a on primary > but failed to backup on secondary" > > Can I provide some additional logs ? > > > > -- > Serg > > > > On Jun 5, 2012, at 12:20 AM, Edison Su wrote: > > > It's a bug in kvm agent code, I fixed the upgrade issue from 2.2.x to 3.0.x, > but it only worked for shared storage. > You can fix it by update db with the following sql: > > update volumes set path=SUBSTRING_INDEX(`path`, '/', -1); > > , then restart mgt server. > > > -----Original Message----- > From: Serg [mailto:kernc...@gmail.com] > Sent: Monday, June 04, 2012 1:25 PM > To: Edison Su > Cc: cloudstack-dev@incubator.apache.org > Subject: Re: 3.0.2 - KVM Agent problem? Please help. > > Hi, > > Thanks for your response, > > I turned on debug logging but nothing new has been discovered. > after upgrading from 2.2.14 to 3.0.2 all KVM server with Local Storage > behave identically > > - Unable to re-start the instance ( instance can't be started anymore ) > - Make a snapshot for volume. > - Can't download the volume > > For now, back to online such instances I'm can by the following way > - import volume of stopped instance as template and the start new > instance from > > Log : > libvir: Storage error : Storage volume not found: no storage vol with > matching name > > [root@ /]# virsh vol-list c6f1f032-5aad-4983-bc83-0ec1a6b5e32c | grep > 4341f531-bef6-47b6-ae75-2b8d6de63038 > 4341f531-bef6-47b6-ae75-2b8d6de63038 /var/lib/libvirt/images/4341f531- > bef6-47b6-ae75-2b8d6de63038 > > [root@ /]# qemu-img check /var/lib/libvirt/images/4341f531-bef6-47b6- > ae75-2b8d6de63038 > No errors were found on the image. > > Full log output : > > Log output : http://pastebin.com/dd0Wz22a > > > -- > Serg > > > > > On Jun 4, 2012, at 8:33 PM, Edison Su wrote: > > Please turn log level to debug(/etc/cloud/agent/log4j-cloud.xml) on > your kvm host, then restart cloud-agent, take a look at agent log again. > > -----Original Message----- > From: Serg [mailto:kernc...@gmail.com] > Sent: Sunday, June 03, 2012 11:48 AM > To: cloudstack-dev@incubator.apache.org > Subject: 3.0.2 - KVM Agent problem? Please help. > > Hi, > > > How to debug such problem ? > All volumes exist and instances running, but can't to > > CentOS release 6.2 (Final) > libvirt-0.9.4-23.el6_2.4.x86_64 > libvirt-client-0.9.4-23.el6_2.4.x86_64 > > > [root@kvmhost2 snapshot]# tail -10 /var/log/cloud/agent/agent.log > com.cloud.utils.exception.CloudRuntimeException: > org.libvirt.LibvirtException: Storage volume not found: no storage > vol > with matching name '/var/lib/libvirt/images/67033fb9-a432-4162-8b22- > d8a7b2aeb100' > at > > com.cloud.agent.storage.LibvirtStorageAdaptor.getVolume(LibvirtStorageA > daptor.java:81) > ...cut... > libvir: Storage error : Storage volume not found: no storage vol > with > matching name '/opt/primary/f5b97a37-f8dd-485c-945c-da31d20b9e23' > ...cut... > libvir: Storage error : Storage volume not found: no storage vol > with > matching name '/var/lib/libvirt/images/58ba9eed-c864-44f3-a987- > 9d1eaab182e1' > > [root@kvmhost2 snapshot]# tail -10 /var/log/cloud/agent/agent.log | > awk > '{print $15}' | xargs file > /opt/primary/e371ebb7-8d9e-4c64-83e9-effea15a00e7: Qemu > Image, Format: Qcow , Version: 2 > /opt/primary/e371ebb7-8d9e-4c64-83e9-effea15a00e7: Qemu > Image, Format: Qcow , Version: 2 > /opt/primary/e55358fa-8dc5-4070-b3b6-3f4bc01b17ae: Qemu > Image, Format: Qcow , Version: 2 > /opt/primary/e55358fa-8dc5-4070-b3b6-3f4bc01b17ae: Qemu > Image, Format: Qcow , Version: 2 > /var/lib/libvirt/images/d49b318c-8bb4-4bb2-a897-1c40ef234d30: Qemu > Image, Format: Qcow , Version: 2 > /var/lib/libvirt/images/d49b318c-8bb4-4bb2-a897-1c40ef234d30: Qemu > Image, Format: Qcow , Version: 2 > /var/lib/libvirt/images/193c378c-80ba-4b82-affe-45c9da0ee839: Qemu > Image, Format: Qcow , Version: 2 > /var/lib/libvirt/images/193c378c-80ba-4b82-affe-45c9da0ee839: Qemu > Image, Format: Qcow , Version: 2 > /var/lib/libvirt/images/58ba9eed-c864-44f3-a987-9d1eaab182e1: Qemu > Image, Format: Qcow , Version: 2 > /var/lib/libvirt/images/58ba9eed-c864-44f3-a987-9d1eaab182e1: Qemu > Image, Format: Qcow , Version: 2 > > When I'm try to start instance through UI I'm get the following > error : > UI Error pop-up : Resource [Host:34] is unreachable: Host 34: Unable > to > start instance due to Unable to get answer that is of class > com.cloud.agent.api.StartAnswer > > Agent LOG : > > 2012-06-03 20:47:13,397 WARN > [resource.computing.LibvirtComputingResource] (agentRequest-Handler- > 2:null) Exception > com.cloud.utils.exception.CloudRuntimeException: > org.libvirt.LibvirtException: Storage volume not found: no storage > vol > with matching name '/var/lib/libvirt/images/19d086dd-8229-48c2-aaa7- > ea3d218dc5dc' > at > > com.cloud.agent.storage.LibvirtStorageAdaptor.getVolume(LibvirtStorageA > daptor.java:81) > at > > com.cloud.agent.storage.LibvirtStorageAdaptor.getPhysicalDisk(LibvirtSt > orageAdaptor.java:450) > at > > com.cloud.agent.storage.LibvirtStoragePool.getPhysicalDisk(LibvirtStora > gePool.java:104) > at > > com.cloud.agent.resource.computing.LibvirtComputingResource.createVbd(L > ibvirtComputingResource.java:2593) > at > > com.cloud.agent.resource.computing.LibvirtComputingResource.execute(Lib > virtComputingResource.java:2510) > at > > com.cloud.agent.resource.computing.LibvirtComputingResource.executeRequ > est(LibvirtComputingResource.java:985) > at com.cloud.agent.Agent.processRequest(Agent.java:517) > at > com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:827) > at com.cloud.utils.nio.Task.run(Task.java:79) > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.ja > va:1110) > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.j > ava:603) > at java.lang.Thread.run(Thread.java:679) > 2012-06-03 20:47:13,399{GMT} WARN [cloud.agent.Agent] > (agentRequest- > Handler-2:) Caught: > java.lang.NullPointerException > at > > com.cloud.agent.resource.computing.LibvirtComputingResource.cleanupVMNe > tworks(LibvirtComputingResource.java:3664) > at > > com.cloud.agent.resource.computing.LibvirtComputingResource.handleVmSta > rtFailure(LibvirtComputingResource.java:2416) > at > > com.cloud.agent.resource.computing.LibvirtComputingResource.execute(Lib > virtComputingResource.java:2539) > at > > com.cloud.agent.resource.computing.LibvirtComputingResource.executeRequ > est(LibvirtComputingResource.java:985) > at com.cloud.agent.Agent.processRequest(Agent.java:517) > at > com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:827) > at com.cloud.utils.nio.Task.run(Task.java:79) > at > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.ja > va:1110) > at > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.j > ava:603) > at java.lang.Thread.run(Thread.java:679) > > > > -- > Serg > >
