> -----Original Message----- > From: de Jong, Mark-Jan [mailto:mark-jan.dej...@teoco.com] > Sent: Thursday, June 07, 2012 11:04 AM > To: Edison Su; 'cloudstack-dev@incubator.apache.org' > Subject: RE: 3.0.2 - KVM Agent problem? Please help. > > > > > -----Original Message----- > > From: Edison Su [mailto:edison...@citrix.com] > > Sent: Thursday, June 07, 2012 1:54 PM > > To: 'cloudstack-dev@incubator.apache.org'; de Jong, Mark-Jan > > Subject: RE: 3.0.2 - KVM Agent problem? Please help. > > > > Thanks for your report, I'll fix it in the future release(not in > 3.0.3 > > though). > > And a bug is created(http://bugs.cloudstack.org/browse/CS-15259). > > > I'm assuming that "upgrade from 2.2.x to 3.0.2 failed on KVM, if local > storage is used" encompasses shared storage as well. We're using GFS.
When you create the storage pool, are you using "sharedmount" or NFS? > > > > -----Original Message----- > > > From: de Jong, Mark-Jan [mailto:mark-jan.dej...@teoco.com] > > > Sent: Wednesday, June 06, 2012 6:45 PM > > > To: 'cloudstack-dev@incubator.apache.org' > > > Subject: RE: 3.0.2 - KVM Agent problem? Please help. > > > > > > > > > > > > > -----Original Message----- > > > > From: Edison Su [mailto:edison...@citrix.com] > > > > Sent: Monday, June 04, 2012 5:20 PM > > > > To: de Jong, Mark-Jan > > > > Subject: RE: 3.0.2 - KVM Agent problem? Please help. > > > > > > > > It's a bug in kvm agent code, I fixed the upgrade issue from > 2.2.x > > > > to 3.0.x, but it only worked for shared storage. > > > > You can fix it by update db with the following sql: > > > > > > > > update volumes set path=SUBSTRING_INDEX(`path`, '/', -1); > > > > > > > > , then restart mgt server. > > > > > > I ran into the same issue and found that, along with the DB update > > > mentioned above, I had to run the following to fix template VM > > builds... > > > > > > update template_spool_ref set > > local_path=SUBSTRING_INDEX(`local_path`, > > > '/', -1); > > > update template_spool_ref set > > > install_path=SUBSTRING_INDEX(`install_path`, '/', -1) > > > > > > > > > > > > > > > -----Original Message----- > > > > > From: Serg [mailto:kernc...@gmail.com] > > > > > Sent: Monday, June 04, 2012 1:25 PM > > > > > To: Edison Su > > > > > Cc: cloudstack-dev@incubator.apache.org > > > > > Subject: Re: 3.0.2 - KVM Agent problem? Please help. > > > > > > > > > > Hi, > > > > > > > > > > Thanks for your response, > > > > > > > > > > I turned on debug logging but nothing new has been discovered. > > > > > after upgrading from 2.2.14 to 3.0.2 all KVM server with Local > > > > Storage > > > > > behave identically > > > > > > > > > > - Unable to re-start the instance ( instance can't be started > > > > anymore > > > > > ) > > > > > - Make a snapshot for volume. > > > > > - Can't download the volume > > > > > > > > > > For now, back to online such instances I'm can by the following > > > > > way > > > > > - import volume of stopped instance as template and the start > new > > > > > instance from > > > > > > > > > > Log : > > > > > libvir: Storage error : Storage volume not found: no storage > vol > > > with > > > > > matching name > > > > > > > > > > [root@ /]# virsh vol-list c6f1f032-5aad-4983-bc83-0ec1a6b5e32c > | > > > grep > > > > > 4341f531-bef6-47b6-ae75-2b8d6de63038 > > > > > 4341f531-bef6-47b6-ae75-2b8d6de63038 > > > > /var/lib/libvirt/images/4341f531- > > > > > bef6-47b6-ae75-2b8d6de63038 > > > > > > > > > > [root@ /]# qemu-img check /var/lib/libvirt/images/4341f531- > bef6- > > > 47b6- > > > > > ae75-2b8d6de63038 > > > > > No errors were found on the image. > > > > > > > > > > Full log output : > > > > > > > > > > Log output : http://pastebin.com/dd0Wz22a > > > > > > > > > > > > > > > -- > > > > > Serg > > > > > > > > > > > > > > > > > > > > > > > > > On Jun 4, 2012, at 8:33 PM, Edison Su wrote: > > > > > > > > > > > Please turn log level to debug(/etc/cloud/agent/log4j- > > cloud.xml) > > > on > > > > > your kvm host, then restart cloud-agent, take a look at agent > log > > > > again. > > > > > > > > > > > >> -----Original Message----- > > > > > >> From: Serg [mailto:kernc...@gmail.com] > > > > > >> Sent: Sunday, June 03, 2012 11:48 AM > > > > > >> To: cloudstack-dev@incubator.apache.org > > > > > >> Subject: 3.0.2 - KVM Agent problem? Please help. > > > > > >> > > > > > >> Hi, > > > > > >> > > > > > >> > > > > > >> How to debug such problem ? > > > > > >> All volumes exist and instances running, but can't to > > > > > >> > > > > > >> CentOS release 6.2 (Final) > > > > > >> libvirt-0.9.4-23.el6_2.4.x86_64 > > > > > >> libvirt-client-0.9.4-23.el6_2.4.x86_64 > > > > > >> > > > > > >> > > > > > >> [root@kvmhost2 snapshot]# tail -10 > > > /var/log/cloud/agent/agent.log > > > > > >> com.cloud.utils.exception.CloudRuntimeException: > > > > > >> org.libvirt.LibvirtException: Storage volume not found: no > > > storage > > > > > vol > > > > > >> with matching name > > > > > >> '/var/lib/libvirt/images/67033fb9-a432-4162-8b22- > > > > > >> d8a7b2aeb100' > > > > > >> at > > > > > >> > > > > > > > > > > > > > > > com.cloud.agent.storage.LibvirtStorageAdaptor.getVolume(LibvirtStorage > > > > > A > > > > > >> daptor.java:81) > > > > > >> ...cut... > > > > > >> libvir: Storage error : Storage volume not found: no storage > > > > > >> vol > > > > > with > > > > > >> matching name '/opt/primary/f5b97a37-f8dd-485c-945c- > > > da31d20b9e23' > > > > > >> ...cut... > > > > > >> libvir: Storage error : Storage volume not found: no storage > > > > > >> vol > > > > > with > > > > > >> matching name '/var/lib/libvirt/images/58ba9eed-c864-44f3- > > a987- > > > > > >> 9d1eaab182e1' > > > > > >> > > > > > >> [root@kvmhost2 snapshot]# tail -10 > > > /var/log/cloud/agent/agent.log > > > > | > > > > > awk > > > > > >> '{print $15}' | xargs file > > > > > >> /opt/primary/e371ebb7-8d9e-4c64-83e9-effea15a00e7: > > > Qemu > > > > > >> Image, Format: Qcow , Version: 2 > > > > > >> /opt/primary/e371ebb7-8d9e-4c64-83e9-effea15a00e7: > > > Qemu > > > > > >> Image, Format: Qcow , Version: 2 > > > > > >> /opt/primary/e55358fa-8dc5-4070-b3b6-3f4bc01b17ae: > > > Qemu > > > > > >> Image, Format: Qcow , Version: 2 > > > > > >> /opt/primary/e55358fa-8dc5-4070-b3b6-3f4bc01b17ae: > > > Qemu > > > > > >> Image, Format: Qcow , Version: 2 > > > > > >> /var/lib/libvirt/images/d49b318c-8bb4-4bb2-a897-1c40ef234d30: > > > Qemu > > > > > >> Image, Format: Qcow , Version: 2 > > > > > >> /var/lib/libvirt/images/d49b318c-8bb4-4bb2-a897-1c40ef234d30: > > > Qemu > > > > > >> Image, Format: Qcow , Version: 2 > > > > > >> /var/lib/libvirt/images/193c378c-80ba-4b82-affe-45c9da0ee839: > > > Qemu > > > > > >> Image, Format: Qcow , Version: 2 > > > > > >> /var/lib/libvirt/images/193c378c-80ba-4b82-affe-45c9da0ee839: > > > Qemu > > > > > >> Image, Format: Qcow , Version: 2 > > > > > >> /var/lib/libvirt/images/58ba9eed-c864-44f3-a987-9d1eaab182e1: > > > Qemu > > > > > >> Image, Format: Qcow , Version: 2 > > > > > >> /var/lib/libvirt/images/58ba9eed-c864-44f3-a987-9d1eaab182e1: > > > Qemu > > > > > >> Image, Format: Qcow , Version: 2 > > > > > >> > > > > > >> When I'm try to start instance through UI I'm get the > > following > > > > > error : > > > > > >> UI Error pop-up : Resource [Host:34] is unreachable: Host 34: > > > > > >> Unable > > > > > to > > > > > >> start instance due to Unable to get answer that is of class > > > > > >> com.cloud.agent.api.StartAnswer > > > > > >> > > > > > >> Agent LOG : > > > > > >> > > > > > >> 2012-06-03 20:47:13,397 WARN > > > > > >> [resource.computing.LibvirtComputingResource] > > > > > >> (agentRequest-Handler- > > > > > >> 2:null) Exception > > > > > >> com.cloud.utils.exception.CloudRuntimeException: > > > > > >> org.libvirt.LibvirtException: Storage volume not found: no > > > storage > > > > > vol > > > > > >> with matching name > > > > > >> '/var/lib/libvirt/images/19d086dd-8229-48c2-aaa7- > > > > > >> ea3d218dc5dc' > > > > > >> at > > > > > >> > > > > > > > > > > > > > > > com.cloud.agent.storage.LibvirtStorageAdaptor.getVolume(LibvirtStorage > > > > > A > > > > > >> daptor.java:81) > > > > > >> at > > > > > >> > > > > > > > > > > > > > > > com.cloud.agent.storage.LibvirtStorageAdaptor.getPhysicalDisk(LibvirtS > > > > > t > > > > > >> orageAdaptor.java:450) > > > > > >> at > > > > > >> > > > > > > > > > > > > > > > com.cloud.agent.storage.LibvirtStoragePool.getPhysicalDisk(LibvirtStor > > > > > a > > > > > >> gePool.java:104) > > > > > >> at > > > > > >> > > > > > > > > > > > > > > > com.cloud.agent.resource.computing.LibvirtComputingResource.createVbd( > > > > > L > > > > > >> ibvirtComputingResource.java:2593) > > > > > >> at > > > > > >> > > > > > > > > > > > > > > > com.cloud.agent.resource.computing.LibvirtComputingResource.execute(Li > > > > > b > > > > > >> virtComputingResource.java:2510) at > > > > > >> > > > > > > > > > > > > > > > com.cloud.agent.resource.computing.LibvirtComputingResource.executeReq > > > > > u > > > > > >> est(LibvirtComputingResource.java:985) > > > > > >> at com.cloud.agent.Agent.processRequest(Agent.java:517) > > > > > >> at > > > > > >> > > com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:827 > > > > > >> ) at com.cloud.utils.nio.Task.run(Task.java:79) > > > > > >> at > > > > > >> > > > > > > > > > > > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.j > > > > > a > > > > > >> va:1110) > > > > > >> at > > > > > >> > > > > > > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor. > > > > > j > > > > > >> ava:603) > > > > > >> at java.lang.Thread.run(Thread.java:679) > > > > > >> 2012-06-03 20:47:13,399{GMT} WARN [cloud.agent.Agent] > > > > > (agentRequest- > > > > > >> Handler-2:) Caught: > > > > > >> java.lang.NullPointerException > > > > > >> at > > > > > >> > > > > > > > > > > > > > > > com.cloud.agent.resource.computing.LibvirtComputingResource.cleanupVMN > > > > > e > > > > > >> tworks(LibvirtComputingResource.java:3664) > > > > > >> at > > > > > >> > > > > > > > > > > > > > > > com.cloud.agent.resource.computing.LibvirtComputingResource.handleVmSt > > > > > a > > > > > >> rtFailure(LibvirtComputingResource.java:2416) > > > > > >> at > > > > > >> > > > > > > > > > > > > > > > com.cloud.agent.resource.computing.LibvirtComputingResource.execute(Li > > > > > b > > > > > >> virtComputingResource.java:2539) at > > > > > >> > > > > > > > > > > > > > > > com.cloud.agent.resource.computing.LibvirtComputingResource.executeReq > > > > > u > > > > > >> est(LibvirtComputingResource.java:985) > > > > > >> at com.cloud.agent.Agent.processRequest(Agent.java:517) > > > > > >> at > > > > > >> > > com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:827 > > > > > >> ) at com.cloud.utils.nio.Task.run(Task.java:79) > > > > > >> at > > > > > >> > > > > > > > > > > > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.j > > > > > a > > > > > >> va:1110) > > > > > >> at > > > > > >> > > > > > > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor. > > > > > j > > > > > >> ava:603) > > > > > >> at java.lang.Thread.run(Thread.java:679) > > > > > >> > > > > > >> > > > > > >> > > > > > >> -- > > > > > >> Serg > > > > > > > > > > > > > > > > > > > PRIVILEGED AND CONFIDENTIAL > > > PLEASE NOTE: The information contained in this message is > privileged > > > and confidential, and is intended only for the use of the > individual > > > to whom it is addressed and others who have been specifically > > > authorized to receive it. If you are not the intended recipient, > you > > > are hereby notified that any dissemination, distribution or copying > > of > > > this communication is strictly prohibited. If you have received > this > > > communication in error, or if any problems occur with transmission, > > > please contact sender. Thank you. > > > > > > Please consider the environment before printing this e-mail. > > > PRIVILEGED AND CONFIDENTIAL > PLEASE NOTE: The information contained in this message is privileged > and confidential, and is intended only for the use of the individual to > whom it is addressed and others who have been specifically authorized > to receive it. If you are not the intended recipient, you are hereby > notified that any dissemination, distribution or copying of this > communication is strictly prohibited. If you have received this > communication in error, or if any problems occur with transmission, > please contact sender. Thank you. > > Please consider the environment before printing this e-mail.
