On Thu, Sep 24, 2015 at 6:26 PM, Nir Soffer <[email protected]> wrote:
> On Thu, Sep 24, 2015 at 5:24 PM, Ian Fraser <[email protected]> wrote: > >> Hi Nir, >> >> >> >> https://gerrit.ovirt.org/46634/ has improved the situation, even tho the >> “Import Virtual Machine(s)” window still hangs. I queried all 3 vCenter >> Hosts then I run: >> >> >> >> grep -e v2v -e libvirtE -e ERROR /var/log/vdsm/vdsm.log >> >> >> >> I get the following much more helpful information: >> >> >> >> Thread-19193::ERROR::2015-09-24 >> 13:05:20,542::v2v::158::root::(get_external_vms) error getting domain xml >> for vm 'devsquid': internal error: Invalid or not yet handled value >> 'emptyBackingString' for VMX entry 'ide1:0.fileName' for device type >> 'cdrom-image' >> >> Thread-19193::ERROR::2015-09-24 >> 13:05:23,975::v2v::158::root::(get_external_vms) error getting domain xml >> for vm 'HD2012': internal error: Expecting VMX entry 'ethernet0.virtualDev' >> to be 'vlance' or 'vmxnet' or 'vmxnet3' or 'e1000' but found 'e1000e' >> >> Thread-19193::ERROR::2015-09-24 >> 13:05:41,102::v2v::158::root::(get_external_vms) error getting domain xml >> for vm 'QA11 2012 R2': internal error: Expecting VMX entry >> 'ethernet0.virtualDev' to be 'vlance' or 'vmxnet' or 'vmxnet3' or 'e1000' >> but found 'e1000e' >> >> Thread-19193::ERROR::2015-09-24 >> 13:05:43,643::v2v::158::root::(get_external_vms) error getting domain xml >> for vm 'HDWin8': internal error: Expecting VMX entry 'ethernet0.virtualDev' >> to be 'vlance' or 'vmxnet' or 'vmxnet3' or 'e1000' but found 'e1000e' >> >> Thread-19193::ERROR::2015-09-24 >> 13:05:45,861::v2v::158::root::(get_external_vms) error getting domain xml >> for vm 'HDwin10-2': internal error: Expecting VMX entry >> 'ethernet0.virtualDev' to be 'vlance' or 'vmxnet' or 'vmxnet3' or 'e1000' >> but found 'e1000e' >> >> Thread-19193::ERROR::2015-09-24 >> 13:05:51,421::v2v::158::root::(get_external_vms) error getting domain xml >> for vm 'HDWin10': internal error: Expecting VMX entry >> 'ethernet0.virtualDev' to be 'vlance' or 'vmxnet' or 'vmxnet3' or 'e1000' >> but found 'e1000e' >> >> Thread-19193::ERROR::2015-09-24 >> 13:06:05,973::v2v::158::root::(get_external_vms) error getting domain xml >> for vm 'wsus-ash-01': internal error: Expecting VMX entry >> 'ethernet0.virtualDev' to be 'vlance' or 'vmxnet' or 'vmxnet3' or 'e1000' >> but found 'e1000e' >> >> Thread-19193::ERROR::2015-09-24 >> 13:06:06,841::v2v::158::root::(get_external_vms) error getting domain xml >> for vm 'Hactar2': internal error: Expecting VMX entry 'numvcpus' to be an >> unsigned integer (1 or a multiple of 2) but found 3 >> >> Thread-19193::ERROR::2015-09-24 >> 13:06:21,972::v2v::158::root::(get_external_vms) error getting domain xml >> for vm 'squid3.2': internal error: Invalid or not yet handled value >> 'emptyBackingString' for VMX entry 'ide1:0.fileName' for device type >> 'cdrom-image' >> >> Thread-19382::ERROR::2015-09-24 >> 13:06:45,322::v2v::158::root::(get_external_vms) error getting domain xml >> for vm 'sequoia-demo-01': internal error: Expecting VMX entry >> 'ethernet0.virtualDev' to be 'vlance' or 'vmxnet' or 'vmxnet3' or 'e1000' >> but found 'e1000e' >> >> Thread-19382::ERROR::2015-09-24 >> 13:06:47,515::v2v::158::root::(get_external_vms) error getting domain xml >> for vm 'sales-demo-01': internal error: Expecting VMX entry >> 'ethernet0.virtualDev' to be 'vlance' or 'vmxnet' or 'vmxnet3' or 'e1000' >> but found 'e1000e' >> >> Thread-19382::ERROR::2015-09-24 >> 13:06:50,535::v2v::158::root::(get_external_vms) error getting domain xml >> for vm 'rdpgw-demo-01': internal error: Expecting VMX entry >> 'ethernet0.virtualDev' to be 'vlance' or 'vmxnet' or 'vmxnet3' or 'e1000' >> but found 'e1000e' >> >> Thread-19382::ERROR::2015-09-24 >> 13:06:54,202::v2v::688::root::(_add_disk_info) Error getting disk size >> >> File "/usr/share/vdsm/v2v.py", line 686, in _add_disk_info >> >> if ret is None: raise libvirtError ('virStorageVolGetInfo() failed', >> vol=self) >> >> libvirtError: internal error: Could not search in datastore >> 'MD3220i-LUN0': FileNotFound - File [MD3220i-LUN0] isostore was not found >> > > Thanks for testing the patch. This info should be very useful to libvirt > guys. > > >> Looks like it is continuing with some errors but still getting tripped on >> others. (vdsm.log attached) >> > > I will look at the log later - the fact that the engine front end still > hangs mean there is another bug hiding in the vdsm. > Looking in the log, we see: 1. Request started Thread-19382::DEBUG::2015-09-24 13:06:41,079::__init__::503::jsonrpc.JsonRpcServer::(_serveRequest) Calling 'Host.getExternalVMs' in bridge with {u'username': u'[email protected]', u'password': '********', u' uri': u'vpx:// ian_admin%[email protected]/Ashford_House/vh-ash-03.asm.org.uk?no_verify=1 '} 2. (Lot of errors...) 3. Request ends Thread-19382::DEBUG::2015-09-24 13:06:57,237::__init__::533::jsonrpc.JsonRpcServer::(_serveRequest) Return 'Host.getExternalVMs' in bridge with [{'status': 'Up', 'arch': 'i686', 'disks': [{'allocation': '16096690176', 'alias': '[CS215] remotesupport/remotesupport.vmdk', 'capacity': '16106127360', 'type': 'disk', 'dev': 'sda'}], 'vmId': '4204d4fc-60d9-ea0f-1130-48d05cc81abb', 'smp': 1, 'memSize': 256, 'vmName': 'remotesupport', 'networks': [{'bridge': 'ASM Internal Network', 'model': 'vmxnet3', 'macAddr': '00:50:56:84:32:29', 'type': 'bridge'}]}, {'status': 'Up', 'arch': 'i686', 'disks': [{'allocation': '80492888064', 'alias': "[CS215] sharon's Desktop/sharon's Desktop.vmdk", 'capacity': '80530636800', 'type': 'disk', 'dev': 'sda'}], 'vmId': '4204abe1-c816-ba3c-126d-6129941a2d6c', 'smp': 2, 'memSize': 4096, 'vmName': "sharon's Desktop", 'networks': [{'bridge': 'ASM Internal Network', 'model': 'vmxnet3', 'macAddr': '00:50:56:84:31:c6', 'type': 'bridge'}]}, {'status': 'Up', 'arch': 'x86_64', 'disks': [{'allocation': '64364740608', 'alias': '[CS215] QA8/QA8.vmdk', 'capacity': '64436843520', 'type': 'disk', 'dev': 'sda'}], 'vmId': '42043e44-82f2-7e5a-2cb3-59d2d52b0cda', 'smp': 2, 'memSize': 8192, 'vmName': 'QA8', 'networks': [{'bridge': 'ASM Internal Network', 'model': 'vmxnet3', 'macAddr': '00:50:56:84:00:14', 'type': 'bridge'}, {'bridge': 'ASM Internal Network', 'model': 'vmxnet3', 'macAddr': '00:50:56:84:00:15', 'type': 'bridge'}]}, {'status': 'Up', 'arch': 'x86_64', 'disks': [{'allocation': '27928821760', 'alias': '[CS215] Fenchurch/Fenchurch_3.vmdk', 'capacity': '37580963840', 'type': 'disk', 'dev': 'sda'}, {'allocation': '51758759936', 'alias': '[CS215] Fenchurch/Fenchurch_1.vmdk', 'capacity': '107389255680', 'type': 'disk', 'dev': 'sdb'}, {'allocation': '10485760', 'alias': '[CS215] Fenchurch/Fenchurch_2.vmdk', 'capacity': '10485760', 'type': 'disk', 'dev': 'sdq'}], 'vmId': '4204de6a-185e-33b2-de96-d7938a66a2af', 'smp': 1, 'memSize': 2048, 'vmName': 'Fenchurch', 'networks': [{'bridge': 'ASM Internal Network', 'model': 'e1000', 'macAddr': '00:50:56:84:00:12', 'type': 'bridge'}]}, {'status': 'Up', 'arch': 'i686', 'disks': [{'allocation': '32211206144', 'alias': '[CS215] QA7/QA7.vmdk', 'capacity': '32212254720', 'type': 'disk', 'dev': 'sda'}, {'allocation': '0', 'alias': '[CS215] QA7/QA7_1.vmdk', 'capacity': '21474836480', 'type': 'disk', 'dev': 'sdb'}], 'vmId': '42040c13-0291-ad9f-9a3e-53c3b897a0a8', 'smp': 1, 'memSize': 1024, 'vmName': 'QA7', 'networks': [{'bridge': 'ASM Internal Network', 'model': 'vmxnet3', 'macAddr': '00:50:56:84:50:89', 'type': 'bridge'}]}, {'status': 'Up', 'arch': 'x86_64', 'disks': [{'allocation': '8154775552', 'alias': '[CS215] QA9/QA9-000002.vmdk', 'capacity': '42949672960', 'type': 'disk', 'dev': 'sda'}, {'allocation': '588251136', 'alias': '[CS215] QA9/QA9_7-000001.vmdk', 'capacity': '42949672960', 'type': 'disk', 'dev': 'sdb'}, {'alias': '[MD3220i-LUN0] isostore/trinity-rescue-kit.3.4-build-372.iso', 'type': 'cdrom', 'dev': 'hdc'}], 'vmId': '42045216-fadd-96dc-84d3-586ac3a8c173', 'smp': 2, 'memSize': 2048, 'vmName': 'QA9', 'networks': [{'bridge': 'ASM Internal Network', 'model': 'vmxnet3', 'macAddr': '00:50:56:84:1d:01', 'type': 'bridge'}]}, {'status': 'Up', 'arch': 'x86_64', 'disks': [{'allocation': '19641925632', 'alias': '[CS215] QADB/QADB.vmdk', 'capacity': '26843545600', 'type': 'disk', 'dev': 'sda'}, {'allocation': '107373133824', 'alias': '[CS215] QADB/QADB_1.vmdk', 'capacity': '107374182400', 'type': 'disk', 'dev': 'sdb'}], 'vmId': '42042f12-6654-3e58-fae2-914f355747fa', 'smp': 2, 'memSize': 4096, 'vmName': 'QADB', 'networks': [{'bridge': 'ASM Internal Network', 'model': 'e1000', 'macAddr': '00:50:56:84:6f:bd', 'type': 'bridge'}]}, {'status': 'Up', 'arch': 'x86_64', 'disks': [{'allocation': '42601545728', 'alias': '[CS215] QA10/QA10.vmdk', 'capacity': '42949672960', 'type': 'disk', 'dev': 'sda'}], 'vmId': '4204307f-4c05-c8ef-351d-ec3885b3341f', 'smp': 2, 'memSize': 4096, 'vmName': 'QA10', 'networks': [{'bridge': 'ASM Internal Network', 'model': 'vmxnet3', 'macAddr': '00:50:56:84:50:80', 'type': 'bridge'}]}] So vdsm was successful, returning list of vms to engine - do you see the vm you want to import in this list? If engine is ui is stuck, there is probably be a bug in the engine side, or maybe vdsm and engine have different views on what should be returned. You can open an ovirt-engine bug with this log and engine log from the same timeframe. I guess you will find there big traceback which can explain why the ui get stuck. Arik, can you take a look at this? > >> Sadly none of these errors are showing in the events in the front end >> which would be more helpful. >> > > Please open a bug about that - the ui should display every vm found, with > error information for those cannot be imported. > > >> >> >> I have been playing divide and conquer with my 80 VMs and it seems there >> are still things that libvirt doesn’t like about some of those VMs. Is >> there a way I can run these commands from the command line so I can pin the >> problems down and alert the libvirt guys directly? >> > > Yes, use: > > vdsClient -s 0 getExternalVMs <uri> <username> <password> > > You should see the uri format sent by the engine in vdsm logs. > > >> If so where would I find what commands are being run? Should I still >> create BZ against ovirt for these (one for each different problem or one >> for them all)? >> > > Yes, please open a libvirt bug for these errors. > > I talked with some libvirty guys, and they suggested to post about this on > the libvirt users list. This is most likely to get the attention of > developers that have worked on the vmware driver in the past. > > Thanks again for your help >> >> >> >> Ian >> >> >> >> *From:* Nir Soffer [mailto:[email protected]] >> *Sent:* 23 September 2015 19:15 >> *To:* Ian Fraser <[email protected]>; users <[email protected]>; >> Shahar Havivi <[email protected]> >> >> *Subject:* Re: [ovirt-users] vmware import hangs after click load button >> on 3.5 rc5 >> >> >> >> On Wed, Sep 23, 2015 at 8:26 PM, Nir Soffer <[email protected]> wrote: >> >> On Wed, Sep 23, 2015 at 8:21 PM, Nir Soffer <[email protected]> wrote: >> >> On Wed, Sep 23, 2015 at 12:40 PM, Ian Fraser <[email protected]> >> wrote: >> >> Hi Nir, >> >> I have changed the v2v.py file as per your request, it has now changed >> the behaviour. The popup window still hangs and I get the following two >> events: >> >> Failed to retrieve VMs information from external server >> vpx://username%[email protected] >> /datacentre_name/hostname?no_verify=1 >> >> VDSM ovirt-host-02 command failed: internal error: Invalid or not yet >> handled value 'emptyBackingString' for VMX entry 'ide1:0.fileName' for >> device type 'cdrom-image' >> >> >> >> Fixing the first error, we see now the real error; libvirt cannot handle >> this vm configuration. We will ask one of the libvirt guys to look into >> this. >> >> >> >> In vdsm.log, we see that one vm was parsed successfully (it failed when >> geting disk info): >> >> >> >> Thread-1076::ERROR::2015-09-23 >> 10:18:51,778::v2v::682::root::(_add_disk_info) Error getting disk size >> >> Traceback (most recent call last): >> >> File "/usr/share/vdsm/v2v.py", line 679, in _add_disk_info >> >> vol = conn.storageVolLookupByPath(disk['alias']) >> >> File "/usr/lib64/python2.7/site-packages/libvirt.py", line 4308, in >> storageVolLookupByPath >> >> if ret is None:raise libvirtError('virStorageVolLookupByPath() >> failed', conn=self) >> >> libvirtError: An error occurred, but the cause is unknown >> >> >> >> Little later, libvirt failed to return the xml for another vm: >> >> >> >> Thread-1076::ERROR::2015-09-23 >> 10:18:52,604::__init__::526::jsonrpc.JsonRpcServer::(_serveRequest) >> Internal server error >> >> Traceback (most recent call last): >> >> File "/usr/lib/python2.7/site-packages/yajsonrpc/__init__.py", line >> 521, in _serveRequest >> >> res = method(**params) >> >> File "/usr/share/vdsm/rpc/Bridge.py", line 273, in _dynamicMethod >> >> result = fn(*methodArgs) >> >> File "/usr/share/vdsm/API.py", line 1436, in getExternalVMs >> >> return v2v.get_external_vms(uri, username, password) >> >> File "/usr/share/vdsm/v2v.py", line 152, in get_external_vms >> >> root = ET.fromstring(vm.XMLDesc(0)) >> >> File "/usr/lib64/python2.7/site-packages/libvirt.py", line 478, in >> XMLDesc >> >> if ret is None: raise libvirtError ('virDomainGetXMLDesc() failed', >> dom=self) >> >> libvirtError: internal error: Invalid or not yet handled value >> 'emptyBackingString' for VMX entry 'ide1:0.fileName' for device type >> 'cdrom-image' >> >> >> >> It is possible that the failing vm was the one you want to import, but it >> is also possible that the vm you want to import was fine. >> >> >> >> The following patch improve error handling, so we are not aborting the >> entire request if libvirt fail to return the domain xml for one of the vms. >> >> https://gerrit.ovirt.org/46634/ >> >> >> >> Please test the patch and report if it works for you. >> >> >> >> Nir >> >> >> >> >> >> You may try to disable the cdrom device on that vm, which is probably >> useless now. >> >> >> >> >> I have attached the vdsm.log to this email, should I also attach to the >> BZ I opened? >> >> >> >> Yes, please attach it. >> >> >> >> Hmm, this is a new issue, as the original issue is solved by the patch. >> >> >> >> Please open a new bug for the libvirt error: >> >> >> >> internal error: Invalid or not yet handled value 'emptyBackingString' for >> VMX entry 'ide1:0.fileName' for device type 'cdrom-image' >> >> >> >> Thanks, >> >> Nir >> >> >> >> >> >> (Adding back [email protected], since this thread may help others with >> same issue.) >> >> >> >> >> Many thanks >> >> Ian >> >> From: Nir Soffer [mailto:[email protected]] >> Sent: 23 September 2015 00:04 >> To: Ian Fraser <[email protected]>; Shahar Havivi <[email protected] >> > >> >> Cc: [email protected] >> Subject: Re: [ovirt-users] vmware import hangs after click load button on >> 3.5 rc5 >> >> Hi Ian, >> >> Your import failed because looking up some disk failed. Unfortunately, we >> don't have enough information in the log abut this failure. >> >> Because of incorrect error handling, this error failed the entire >> request, failing your import. >> >> Patch [1] fixes the second problem. If is possible that with this patch >> listing the external vms will work and you will be able to import the vm, >> but it is also possible that the first error was significant and will fail >> the import later. >> >> It would be useful if you test this patch and report if it works for you. >> >> Would you open an ovirt bug for this issue, attaching the vdsm log? >> >> [1] https://gerrit.ovirt.org/46540 >> >> Nir >> >> On Tue, Sep 22, 2015 at 9:09 AM, Ian Fraser <[email protected]> >> wrote: >> Thanks Nir, >> >> File attached. >> >> From: Nir Soffer [mailto:[email protected]] >> Sent: 21 September 2015 23:16 >> To: Ian Fraser <[email protected]> >> Cc: [email protected] >> Subject: Re: [ovirt-users] vmware import hangs after click load button on >> 3.5 rc5 >> >> On Tue, Sep 22, 2015 at 12:14 AM, Ian Fraser <[email protected]> >> wrote: >> I did get a “VDSM <hostname> command failed: local variable 'capacity' >> referenced before assignment” error in the events I have just noticed, does >> that shed any more light? >> >> This shed some light. Can you share the vdsm.log containing this error? >> >> Look in /var/log/vdsm/vdsm.log* >> >> ______________________________________________________________________ >> This email has been scanned by the Symantec Email Security.cloud service. >> For more information please visit http://www.symanteccloud.com >> ______________________________________________________________________ >> >> ________________________________________ >> >> >> The information in this message and any attachment is intended for the >> addressee and is confidential. If you are not that addressee, no action >> should be taken in reliance on the information and you should please reply >> to this message immediately to inform us of incorrect receipt and destroy >> this message and any attachments. >> >> For the purposes of internet level email security incoming and outgoing >> emails may be read by personnel other than the named recipient or sender. >> >> Whilst all reasonable efforts are made, ASM (UK) Ltd cannot guarantee >> that emails and attachments are virus free or compatible with your systems. >> You should make your own checks and ASM (UK) Ltd does not accept liability >> in respect of viruses or computer problems experienced. >> Registered address: Agency Sector Management (UK) Ltd. Ashford House, >> 41-45 Church Road, Ashford, Middlesex, TW15 2TQ >> Registered in England No.2053849 >> >> ______________________________________________________________________ >> This email has been scanned by the Symantec Email Security.cloud service. >> For more information please visit http://www.symanteccloud.com >> ______________________________________________________________________ >> >> >> ______________________________________________________________________ >> This email has been scanned by the Symantec Email Security.cloud service. >> For more information please visit http://www.symanteccloud.com >> ______________________________________________________________________ >> >> ________________________________ >> >> The information in this message and any attachment is intended for the >> addressee and is confidential. If you are not that addressee, no action >> should be taken in reliance on the information and you should please reply >> to this message immediately to inform us of incorrect receipt and destroy >> this message and any attachments. >> >> For the purposes of internet level email security incoming and outgoing >> emails may be read by personnel other than the named recipient or sender. >> >> Whilst all reasonable efforts are made, ASM (UK) Ltd cannot guarantee >> that emails and attachments are virus free or compatible with your systems. >> You should make your own checks and ASM (UK) Ltd does not accept liability >> in respect of viruses or computer problems experienced. >> Registered address: Agency Sector Management (UK) Ltd. Ashford House, >> 41-45 Church Road, Ashford, Middlesex, TW15 2TQ >> Registered in England No.2053849 >> >> ______________________________________________________________________ >> This email has been scanned by the Symantec Email Security.cloud service. >> For more information please visit http://www.symanteccloud.com >> ______________________________________________________________________ >> >> >> >> >> >> >> >> >> ______________________________________________________________________ >> This email has been scanned by the Symantec Email Security.cloud service. >> For more information please visit http://www.symanteccloud.com >> ______________________________________________________________________ >> >> ------------------------------ >> >> The information in this message and any attachment is intended for the >> addressee and is confidential. If you are not that addressee, no action >> should be taken in reliance on the information and you should please reply >> to this message immediately to inform us of incorrect receipt and destroy >> this message and any attachments. >> >> For the purposes of internet level email security incoming and outgoing >> emails may be read by personnel other than the named recipient or sender. >> >> Whilst all reasonable efforts are made, ASM (UK) Ltd cannot guarantee >> that emails and attachments are virus free or compatible with your systems. >> You should make your own checks and ASM (UK) Ltd does not accept liability >> in respect of viruses or computer problems experienced. >> Registered address: Agency Sector Management (UK) Ltd. Ashford House, >> 41-45 Church Road, Ashford, Middlesex, TW15 2TQ >> Registered in England No.2053849 >> >> ______________________________________________________________________ >> This email has been scanned by the Symantec Email Security.cloud service. >> For more information please visit http://www.symanteccloud.com >> ______________________________________________________________________ >> > >
_______________________________________________ Users mailing list [email protected] http://lists.ovirt.org/mailman/listinfo/users
