> -----Original Message----- > From: Edison Su [mailto:edison...@citrix.com] > Sent: Thursday, June 07, 2012 2:15 PM > To: de Jong, Mark-Jan; 'cloudstack-dev@incubator.apache.org' > Subject: RE: 3.0.2 - KVM Agent problem? Please help. > > > > > -----Original Message----- > > From: de Jong, Mark-Jan [mailto:mark-jan.dej...@teoco.com] > > Sent: Thursday, June 07, 2012 11:04 AM > > To: Edison Su; 'cloudstack-dev@incubator.apache.org' > > Subject: RE: 3.0.2 - KVM Agent problem? Please help. > > > > > > > > > -----Original Message----- > > > From: Edison Su [mailto:edison...@citrix.com] > > > Sent: Thursday, June 07, 2012 1:54 PM > > > To: 'cloudstack-dev@incubator.apache.org'; de Jong, Mark-Jan > > > Subject: RE: 3.0.2 - KVM Agent problem? Please help. > > > > > > Thanks for your report, I'll fix it in the future release(not in > > 3.0.3 > > > though). > > > And a bug is created(http://bugs.cloudstack.org/browse/CS-15259). > > > > > I'm assuming that "upgrade from 2.2.x to 3.0.2 failed on KVM, if > > local storage is used" encompasses shared storage as well. We're > using GFS. > > When you create the storage pool, are you using "sharedmount" or NFS?
Primary storage is type "SharedMountPoint". > > > > > > > -----Original Message----- > > > > From: de Jong, Mark-Jan [mailto:mark-jan.dej...@teoco.com] > > > > Sent: Wednesday, June 06, 2012 6:45 PM > > > > To: 'cloudstack-dev@incubator.apache.org' > > > > Subject: RE: 3.0.2 - KVM Agent problem? Please help. > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > From: Edison Su [mailto:edison...@citrix.com] > > > > > Sent: Monday, June 04, 2012 5:20 PM > > > > > To: de Jong, Mark-Jan > > > > > Subject: RE: 3.0.2 - KVM Agent problem? Please help. > > > > > > > > > > It's a bug in kvm agent code, I fixed the upgrade issue from > > 2.2.x > > > > > to 3.0.x, but it only worked for shared storage. > > > > > You can fix it by update db with the following sql: > > > > > > > > > > update volumes set path=SUBSTRING_INDEX(`path`, '/', -1); > > > > > > > > > > , then restart mgt server. > > > > > > > > I ran into the same issue and found that, along with the DB > update > > > > mentioned above, I had to run the following to fix template VM > > > builds... > > > > > > > > update template_spool_ref set > > > local_path=SUBSTRING_INDEX(`local_path`, > > > > '/', -1); > > > > update template_spool_ref set > > > > install_path=SUBSTRING_INDEX(`install_path`, '/', -1) > > > > > > > > > > > > > > > > > > > -----Original Message----- > > > > > > From: Serg [mailto:kernc...@gmail.com] > > > > > > Sent: Monday, June 04, 2012 1:25 PM > > > > > > To: Edison Su > > > > > > Cc: cloudstack-dev@incubator.apache.org > > > > > > Subject: Re: 3.0.2 - KVM Agent problem? Please help. > > > > > > > > > > > > Hi, > > > > > > > > > > > > Thanks for your response, > > > > > > > > > > > > I turned on debug logging but nothing new has been > discovered. > > > > > > after upgrading from 2.2.14 to 3.0.2 all KVM server with > Local > > > > > Storage > > > > > > behave identically > > > > > > > > > > > > - Unable to re-start the instance ( instance can't be > started > > > > > anymore > > > > > > ) > > > > > > - Make a snapshot for volume. > > > > > > - Can't download the volume > > > > > > > > > > > > For now, back to online such instances I'm can by the > > > > > > following way > > > > > > - import volume of stopped instance as template and the start > > new > > > > > > instance from > > > > > > > > > > > > Log : > > > > > > libvir: Storage error : Storage volume not found: no storage > > vol > > > > with > > > > > > matching name > > > > > > > > > > > > [root@ /]# virsh vol-list c6f1f032-5aad-4983-bc83- > 0ec1a6b5e32c > > | > > > > grep > > > > > > 4341f531-bef6-47b6-ae75-2b8d6de63038 > > > > > > 4341f531-bef6-47b6-ae75-2b8d6de63038 > > > > > /var/lib/libvirt/images/4341f531- > > > > > > bef6-47b6-ae75-2b8d6de63038 > > > > > > > > > > > > [root@ /]# qemu-img check /var/lib/libvirt/images/4341f531- > > bef6- > > > > 47b6- > > > > > > ae75-2b8d6de63038 > > > > > > No errors were found on the image. > > > > > > > > > > > > Full log output : > > > > > > > > > > > > Log output : http://pastebin.com/dd0Wz22a > > > > > > > > > > > > > > > > > > -- > > > > > > Serg > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Jun 4, 2012, at 8:33 PM, Edison Su wrote: > > > > > > > > > > > > > Please turn log level to debug(/etc/cloud/agent/log4j- > > > cloud.xml) > > > > on > > > > > > your kvm host, then restart cloud-agent, take a look at agent > > log > > > > > again. > > > > > > > > > > > > > >> -----Original Message----- > > > > > > >> From: Serg [mailto:kernc...@gmail.com] > > > > > > >> Sent: Sunday, June 03, 2012 11:48 AM > > > > > > >> To: cloudstack-dev@incubator.apache.org > > > > > > >> Subject: 3.0.2 - KVM Agent problem? Please help. > > > > > > >> > > > > > > >> Hi, > > > > > > >> > > > > > > >> > > > > > > >> How to debug such problem ? > > > > > > >> All volumes exist and instances running, but can't to > > > > > > >> > > > > > > >> CentOS release 6.2 (Final) > > > > > > >> libvirt-0.9.4-23.el6_2.4.x86_64 > > > > > > >> libvirt-client-0.9.4-23.el6_2.4.x86_64 > > > > > > >> > > > > > > >> > > > > > > >> [root@kvmhost2 snapshot]# tail -10 > > > > /var/log/cloud/agent/agent.log > > > > > > >> com.cloud.utils.exception.CloudRuntimeException: > > > > > > >> org.libvirt.LibvirtException: Storage volume not found: no > > > > storage > > > > > > vol > > > > > > >> with matching name > > > > > > >> '/var/lib/libvirt/images/67033fb9-a432-4162-8b22- > > > > > > >> d8a7b2aeb100' > > > > > > >> at > > > > > > >> > > > > > > > > > > > > > > > > > > > > > com.cloud.agent.storage.LibvirtStorageAdaptor.getVolume(LibvirtStorage > > > > > > A > > > > > > >> daptor.java:81) > > > > > > >> ...cut... > > > > > > >> libvir: Storage error : Storage volume not found: no > > > > > > >> storage vol > > > > > > with > > > > > > >> matching name '/opt/primary/f5b97a37-f8dd-485c-945c- > > > > da31d20b9e23' > > > > > > >> ...cut... > > > > > > >> libvir: Storage error : Storage volume not found: no > > > > > > >> storage vol > > > > > > with > > > > > > >> matching name '/var/lib/libvirt/images/58ba9eed-c864-44f3- > > > a987- > > > > > > >> 9d1eaab182e1' > > > > > > >> > > > > > > >> [root@kvmhost2 snapshot]# tail -10 > > > > /var/log/cloud/agent/agent.log > > > > > | > > > > > > awk > > > > > > >> '{print $15}' | xargs file > > > > > > >> /opt/primary/e371ebb7-8d9e-4c64-83e9-effea15a00e7: > > > > Qemu > > > > > > >> Image, Format: Qcow , Version: 2 > > > > > > >> /opt/primary/e371ebb7-8d9e-4c64-83e9-effea15a00e7: > > > > Qemu > > > > > > >> Image, Format: Qcow , Version: 2 > > > > > > >> /opt/primary/e55358fa-8dc5-4070-b3b6-3f4bc01b17ae: > > > > Qemu > > > > > > >> Image, Format: Qcow , Version: 2 > > > > > > >> /opt/primary/e55358fa-8dc5-4070-b3b6-3f4bc01b17ae: > > > > Qemu > > > > > > >> Image, Format: Qcow , Version: 2 > > > > > > >> /var/lib/libvirt/images/d49b318c-8bb4-4bb2-a897- > 1c40ef234d30: > > > > Qemu > > > > > > >> Image, Format: Qcow , Version: 2 > > > > > > >> /var/lib/libvirt/images/d49b318c-8bb4-4bb2-a897- > 1c40ef234d30: > > > > Qemu > > > > > > >> Image, Format: Qcow , Version: 2 > > > > > > >> /var/lib/libvirt/images/193c378c-80ba-4b82-affe- > 45c9da0ee839: > > > > Qemu > > > > > > >> Image, Format: Qcow , Version: 2 > > > > > > >> /var/lib/libvirt/images/193c378c-80ba-4b82-affe- > 45c9da0ee839: > > > > Qemu > > > > > > >> Image, Format: Qcow , Version: 2 > > > > > > >> /var/lib/libvirt/images/58ba9eed-c864-44f3-a987- > 9d1eaab182e1: > > > > Qemu > > > > > > >> Image, Format: Qcow , Version: 2 > > > > > > >> /var/lib/libvirt/images/58ba9eed-c864-44f3-a987- > 9d1eaab182e1: > > > > Qemu > > > > > > >> Image, Format: Qcow , Version: 2 > > > > > > >> > > > > > > >> When I'm try to start instance through UI I'm get the > > > following > > > > > > error : > > > > > > >> UI Error pop-up : Resource [Host:34] is unreachable: Host > 34: > > > > > > >> Unable > > > > > > to > > > > > > >> start instance due to Unable to get answer that is of > class > > > > > > >> com.cloud.agent.api.StartAnswer > > > > > > >> > > > > > > >> Agent LOG : > > > > > > >> > > > > > > >> 2012-06-03 20:47:13,397 WARN > > > > > > >> [resource.computing.LibvirtComputingResource] > > > > > > >> (agentRequest-Handler- > > > > > > >> 2:null) Exception > > > > > > >> com.cloud.utils.exception.CloudRuntimeException: > > > > > > >> org.libvirt.LibvirtException: Storage volume not found: no > > > > storage > > > > > > vol > > > > > > >> with matching name > > > > > > >> '/var/lib/libvirt/images/19d086dd-8229-48c2-aaa7- > > > > > > >> ea3d218dc5dc' > > > > > > >> at > > > > > > >> > > > > > > > > > > > > > > > > > > > > > com.cloud.agent.storage.LibvirtStorageAdaptor.getVolume(LibvirtStorage > > > > > > A > > > > > > >> daptor.java:81) > > > > > > >> at > > > > > > >> > > > > > > > > > > > > > > > > > > > > > com.cloud.agent.storage.LibvirtStorageAdaptor.getPhysicalDisk(LibvirtS > > > > > > t > > > > > > >> orageAdaptor.java:450) > > > > > > >> at > > > > > > >> > > > > > > > > > > > > > > > > > > > > > com.cloud.agent.storage.LibvirtStoragePool.getPhysicalDisk(LibvirtStor > > > > > > a > > > > > > >> gePool.java:104) > > > > > > >> at > > > > > > >> > > > > > > > > > > > > > > > > > > > > > com.cloud.agent.resource.computing.LibvirtComputingResource.createVbd( > > > > > > L > > > > > > >> ibvirtComputingResource.java:2593) > > > > > > >> at > > > > > > >> > > > > > > > > > > > > > > > > > > > > > com.cloud.agent.resource.computing.LibvirtComputingResource.execute(Li > > > > > > b > > > > > > >> virtComputingResource.java:2510) at > > > > > > >> > > > > > > > > > > > > > > > > > > > > > com.cloud.agent.resource.computing.LibvirtComputingResource.executeReq > > > > > > u > > > > > > >> est(LibvirtComputingResource.java:985) > > > > > > >> at com.cloud.agent.Agent.processRequest(Agent.java:517) > > > > > > >> at > > > > > > >> > > > com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:827 > > > > > > >> ) at com.cloud.utils.nio.Task.run(Task.java:79) > > > > > > >> at > > > > > > >> > > > > > > > > > > > > > > > > > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.j > > > > > > a > > > > > > >> va:1110) > > > > > > >> at > > > > > > >> > > > > > > > > > > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor. > > > > > > j > > > > > > >> ava:603) > > > > > > >> at java.lang.Thread.run(Thread.java:679) > > > > > > >> 2012-06-03 20:47:13,399{GMT} WARN [cloud.agent.Agent] > > > > > > (agentRequest- > > > > > > >> Handler-2:) Caught: > > > > > > >> java.lang.NullPointerException at > > > > > > >> > > > > > > > > > > > > > > > > > > > > > com.cloud.agent.resource.computing.LibvirtComputingResource.cleanupVMN > > > > > > e > > > > > > >> tworks(LibvirtComputingResource.java:3664) > > > > > > >> at > > > > > > >> > > > > > > > > > > > > > > > > > > > > > com.cloud.agent.resource.computing.LibvirtComputingResource.handleVmSt > > > > > > a > > > > > > >> rtFailure(LibvirtComputingResource.java:2416) > > > > > > >> at > > > > > > >> > > > > > > > > > > > > > > > > > > > > > com.cloud.agent.resource.computing.LibvirtComputingResource.execute(Li > > > > > > b > > > > > > >> virtComputingResource.java:2539) at > > > > > > >> > > > > > > > > > > > > > > > > > > > > > com.cloud.agent.resource.computing.LibvirtComputingResource.executeReq > > > > > > u > > > > > > >> est(LibvirtComputingResource.java:985) > > > > > > >> at com.cloud.agent.Agent.processRequest(Agent.java:517) > > > > > > >> at > > > > > > >> > > > com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:827 > > > > > > >> ) at com.cloud.utils.nio.Task.run(Task.java:79) > > > > > > >> at > > > > > > >> > > > > > > > > > > > > > > > > > > > > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.j > > > > > > a > > > > > > >> va:1110) > > > > > > >> at > > > > > > >> > > > > > > > > > > > > > > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor. > > > > > > j > > > > > > >> ava:603) > > > > > > >> at java.lang.Thread.run(Thread.java:679) > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> -- > > > > > > >> Serg > > > > > > > > > > > > > > > > > > > > > > > > PRIVILEGED AND CONFIDENTIAL > > > > PLEASE NOTE: The information contained in this message is > > privileged > > > > and confidential, and is intended only for the use of the > > individual > > > > to whom it is addressed and others who have been specifically > > > > authorized to receive it. If you are not the intended recipient, > > you > > > > are hereby notified that any dissemination, distribution or > > > > copying > > > of > > > > this communication is strictly prohibited. If you have received > > this > > > > communication in error, or if any problems occur with > > > > transmission, please contact sender. Thank you. > > > > > > > > Please consider the environment before printing this e-mail. > > > > > > PRIVILEGED AND CONFIDENTIAL > > PLEASE NOTE: The information contained in this message is privileged > > and confidential, and is intended only for the use of the individual > > to whom it is addressed and others who have been specifically > > authorized to receive it. If you are not the intended recipient, you > > are hereby notified that any dissemination, distribution or copying > of > > this communication is strictly prohibited. If you have received this > > communication in error, or if any problems occur with transmission, > > please contact sender. Thank you. > > > > Please consider the environment before printing this e-mail. PRIVILEGED AND CONFIDENTIAL PLEASE NOTE: The information contained in this message is privileged and confidential, and is intended only for the use of the individual to whom it is addressed and others who have been specifically authorized to receive it. If you are not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, or if any problems occur with transmission, please contact sender. Thank you. Please consider the environment before printing this e-mail.
