Thank you for your response Dag.
I read your article (even before you linked it :-).
After executing the SQL querry, I matched the results from it with all possible
UUIDs i could think of. There were no matches. :-(
BTW Dag, I guess you should correct the column naming in your SQL statement,
‘cause it’s voltype and not volpath (..cloud.volumes.volume_type as volpath,)
:-)
I read the article to the end and tried reading the VHD directly with vhd-util,
but noticed that it is is actually missing:
[root@x3 ~]# ls -la
/dev/VG_XenStorage-3b6e386d-8736-6b6b-7006-f3f4df9bd586/VHD-1a240d45-ee0a-4c30-809b-3114dfaf85ba
ls:
/dev/VG_XenStorage-3b6e386d-8736-6b6b-7006-f3f4df9bd586/VHD-1a240d45-ee0a-4c30-809b-3114dfaf85ba:
No such file or directory
[root@x3 ~]# ls -la
/dev/VG_XenStorage-3b6e386d-8736-6b6b-7006-f3f4df9bd586/VHD-1a240d45-ee0a-4c30-809b-3114dfaf85ba
ls:
/dev/VG_XenStorage-3b6e386d-8736-6b6b-7006-f3f4df9bd586/VHD-1a240d45-ee0a-4c30-809b-3114dfaf85ba:
No such file or directory
[root@x3 ~]# ls -la /dev/VG_XenStorage-3b6e386d-8736-6b6b-7006-f3f4df9bd586/
total 24
drwxr-xr-x 2 root root 180 Dec 8 15:12 .
drwxr-xr-x 22 root root 17200 Dec 8 15:11 ..
lrwxrwxrwx 1 root root 112 Jun 28 14:30
hb-f6d699aa-f58f-449e-9a51-6e965562f178 ->
/dev/mapper/VG_XenStorage--3b6e386d--8736--6b6b--7006--f3f4df9bd586-hb--f6d699aa--f58f--449e--9a51--6e965562f178
lrwxrwxrwx 1 root root 71 Jun 28 12:22 MGT ->
/dev/mapper/VG_XenStorage--3b6e386d--8736--6b6b--7006--f3f4df9bd586-MGT
lrwxrwxrwx 1 root root 113 Nov 12 03:26
VHD-10e0a90f-6f47-41f4-910c-53a1514a0f2c ->
/dev/mapper/VG_XenStorage--3b6e386d--8736--6b6b--7006--f3f4df9bd586-VHD--10e0a90f--6f47--41f4--910c--53a1514a0f2c
lrwxrwxrwx 1 root root 113 Dec 1 14:04
VHD-19fd1780-dacf-4fa9-8615-cd3204df6b14 ->
/dev/mapper/VG_XenStorage--3b6e386d--8736--6b6b--7006--f3f4df9bd586-VHD--19fd1780--dacf--4fa9--8615--cd3204df6b14
lrwxrwxrwx 1 root root 113 Nov 12 03:26
VHD-b51b4cb7-f2c5-4ba3-98d3-4a23ff599ad9 ->
/dev/mapper/VG_XenStorage--3b6e386d--8736--6b6b--7006--f3f4df9bd586-VHD--b51b4cb7--f2c5--4ba3--98d3--4a23ff599ad9
lrwxrwxrwx 1 root root 113 Dec 1 14:01
VHD-bb171b0f-e806-4620-8b2c-0ca46203a431 ->
/dev/mapper/VG_XenStorage--3b6e386d--8736--6b6b--7006--f3f4df9bd586-VHD--bb171b0f--e806--4620--8b2c--0ca46203a431
lrwxrwxrwx 1 root root 113 Nov 12 03:26
VHD-e04aa8fa-93ce-4133-8fcb-35c9ce57ed08 ->
/dev/mapper/VG_XenStorage--3b6e386d--8736--6b6b--7006--f3f4df9bd586-VHD--e04aa8fa--93ce--4133--8fcb--35c9ce57ed08
[root@x3 ~]# vhd-util read -p -n
/dev/VG_XenStorage-3b6e386d-8736-6b6b-7006-f3f4df9bd586/VHD-1a240d45-ee0a-4c30-809b-3114dfaf85ba
Failed to open
/dev/VG_XenStorage-3b6e386d-8736-6b6b-7006-f3f4df9bd586/VHD-1a240d45-ee0a-4c30-809b-3114dfaf85ba:
-2
/dev/VG_XenStorage-3b6e386d-8736-6b6b-7006-f3f4df9bd586/VHD-1a240d45-ee0a-4c30-809b-3114dfaf85ba
appears invalid; dumping headers
Then I decided to see to which chain it belongs to. I simply looked SMlog for
VHD *1a240d45[VHD](20.000G//136.000M|n) after a srscan. Result of “changed”
vdis:
(Uppon request I can provide the whole chain.)
*555b38bf[VHD](20.000G//2.695G|n)
5afd5849[VHD](20.000G//8.000M|n)
*02e8b56b[VHD](20.000G//12.000M|n)
*1a240d45[VHD](20.000G//136.000M|n) <- corrupted
*c99e711b[VHD](20.000G//136.000M|n)
34a03c9a[VHD](20.000G//8.000M|a)
5c12db0a[VHD](20.000G//8.000M|n)
I have noticed that above numbers are part of UUID so I did additional search
on the whole database and these are the results:
*555b38bf[VHD](20.000G//2.695G|n) -> not foud
5afd5849[VHD](20.000G//8.000M|n) -> found in template_spool_ref.
status ready, template ID 292
*02e8b56b[VHD](20.000G//12.000M|n) -> not found
*1a240d45[VHD](20.000G//136.000M|n) <- corrupted and not found
*c99e711b[VHD](20.000G//136.000M|n) -> not found
34a03c9a[VHD](20.000G//8.000M|a) -> found in
shapshot_store_ref. state destroyed, volume id 661, from template id 292
5c12db0a[VHD](20.000G//8.000M|n) -> found in
shapshot_store_ref. state destroyed
Here are my assumptions / questions. Please advise / comment.
Is last snapshot: 5c12db0a in chain linked to the first: 555b38bf or the chain
ends with 34a03c9a?
Does anyone have a link to documentation explaining this output?
What do the starts ie. * mean?
To which VM based on the above output is the *1a240d45[VHD]related?
How to make the error go away? How to safely delete *1a240d45[VHD]?
All suggestions are welcome!
Thank you.
F.
On 08 Dec 2015, at 09:46, Dag Sonstebo <[email protected]> wrote:
> Hi France,
>
> have a look at the following writeup I did a few weeks back -
> http://www.shapeblue.com/recovery-of-vms-to-new-cloudstack-instance/
>
> It should give you an idea how to troubleshoot the VHD chain and how to tie
> it back to DB entries and potential VM owner.
>
>
> Regards,
>
> Dag Sonstebo
> Cloud Architect
> ShapeBlue
>
>
>
>
>
> On 07/12/2015, 22:14, "France" <[email protected]> wrote:
>
>> Hi Guys,
>>
>> after battling with cloudstack (4.3.2+) for the whole day with another issue
>> (unable to create snapshot’s for one VM, which I also failed to fix) I ran
>> into message on XenServer 6.0.2+Hotfixes SMlog (copy pasted at the bottom).
>>
>> Even thou i kinda understand XenServers relations of VM -> VBD -> VDI -> SR
>> -> PBD, i failed to find out to which VM does this corrupted .vhd snapshot
>> belong.
>> Can you please help me find it? Afterwards, we can find the appropriate
>> action in order to remove the error and make coalesce work.
>>
>> The physical file with corruption is:
>> /dev/VG_XenStorage-3b6e386d-8736-6b6b-7006-f3f4df9bd586/VHD-1a240d45-ee0a-4c30-809b-3114dfaf85ba
>>
>> This is the VDI it supposedly belongs to:
>> [root@x3 log]# xe vdi-list uuid=02e8b56b-279c-4d5b-8870-b3c5f2255dc7
>> params=all
>> uuid ( RO) : 02e8b56b-279c-4d5b-8870-b3c5f2255dc7
>> name-label ( RW): base copy
>> name-description ( RW):
>> is-a-snapshot ( RO): false
>> snapshot-of ( RO): <not in database>
>> snapshots ( RO):
>> snapshot-time ( RO): 19700101T00:00:00Z
>> allowed-operations (SRO): generate_config; update; resize; destroy;
>> clone; copy; snapshot
>> current-operations (SRO):
>> sr-uuid ( RO): 3b6e386d-8736-6b6b-7006-f3f4df9bd586
>> sr-name-label ( RO): 1c8dbf77-2b23-3244-91ee-5037cb2a55a8
>> vbd-uuids (SRO):
>> crashdump-uuids (SRO):
>> virtual-size ( RO): 21474836480
>> physical-utilisation ( RO): 12582912
>> location ( RO): 02e8b56b-279c-4d5b-8870-b3c5f2255dc7
>> type ( RO): User
>> sharable ( RO): false
>> read-only ( RO): true
>> storage-lock ( RO): false
>> managed ( RO): false
>> parent ( RO): <not in database>
>> missing ( RO): false
>> other-config (MRW):
>> xenstore-data (MRO):
>> sm-config (MRO): vhd-parent:
>> 555b38bf-fe63-41b2-9609-e16f4e26b274; vdi_type: vhd; vhd-blocks:
>> eJxjYBgFo2AUjIKRCQAFAAAB
>> on-boot ( RW): persist
>> allow-caching ( RW): false
>> metadata-latest ( RO): false
>> metadata-of-pool ( RO): <not in database>
>> tags (SRW):
>>
>> (It supposedly resides on this SR:
>> uuid ( RO) : 3b6e386d-8736-6b6b-7006-f3f4df9bd586
>> name-label ( RW): 1c8dbf77-2b23-3244-91ee-5037cb2a55a8
>> name-description ( RW): 208 - censored
>> host ( RO): <shared>
>> type ( RO): lvmoiscsi
>> content-type ( RO): user
>>
>> Which is on one of many HA iscsi targets mounted on the XenServer cluster.
>> uuid ( RO) : f471ef16-4670-2f5e-967e-7e98d4721fe7
>> host-uuid ( RO): f6d699aa-f58f-449e-9a51-6e965562f178
>> sr-uuid ( RO): 3b6e386d-8736-6b6b-7006-f3f4df9bd586
>> device-config (MRO): targetIQN: censoredi:storage.censored; target:
>> censored; SCSIid: 1target2lun1
>> currently-attached ( RO): true)
>>
>> Cleaned exempt from SMlog on cluster master:
>>
>> [8756] 2015-12-07 21:50:45.118758 ['/usr/sbin/vhd-util', 'check',
>> '--debug', '-n',
>> '/dev/VG_XenStorage-3b6e386d-8736-6b6b-7006-f3f4df9bd586/VHD-1a240d45-ee0a-4c30-809b-3114dfaf85ba']
>> [8756] 2015-12-07 21:50:45.131975 FAILED: (rc 22) stdout: 'primary
>> footer invalid: invalid cookie
>> /dev/VG_XenStorage-3b6e386d-8736-6b6b-7006-f3f4df9bd586/VHD-1a240d45-ee0a-4c30-809b-3114dfaf85ba
>> appears invalid; dumping metadata
>> VHD Footer Summary:
>> -------------------
>> Cookie : conectix
>> Features : (0x00000002) <RESV>
>> File format version : Major: 1, Minor: 0
>> Data offset : 512
>> Timestamp : Mon Dec 7 18:19:46 2015
>> Creator Application : 'tap'
>> Creator version : Major: 1, Minor: 3
>> Creator OS : Unknown!
>> Original disk size : 20480 MB (21474836480 Bytes)
>> Current disk size : 20480 MB (21474836480 Bytes)
>> Geometry : Cyl: 41610, Hds: 16, Sctrs: 63
>> : = 20479 MB (21474754560 Bytes)
>> Disk type : Differencing hard disk
>> Checksum : 0xffffef85|0xffffef85 (Good!)
>> UUID : 8f25299d-d7cc-44fe-a42b-95b4e9a31a47
>> Saved state : No
>> Hidden : 1
>>
>> VHD Header Summary:
>> -------------------
>> Cookie : cxsparse
>> Data offset (unusd) : 18446744073709
>> Table offset : 1536
>> Header version : 0x00010000
>> Max BAT size : 1048576
>> Block size : 2097152 (2 MB)
>> Parent name :
>> VG_XenStorage--3b6e386d--8736--6b6b--7006--f3f4df9bd586-VHD--02e8b56b--279c--4d5b--8870--b3c5f2255dc7
>> Parent UUID : 8cece4c9-8c94-47b9-9f7a-cb6023da9b72
>> Parent timestamp : Mon Dec 7 18:19:46 2015
>> Checksum : 0xffffc6ed|0xffffc6ed (Good!)
>>
>> VHD Parent Locators:
>> --------------------
>> locator: : 0
>> code : PLAT_CODE_MACX
>> data_space : 512
>> data_length : 110
>> data_offset : 4327424
>> decoded name :
>> ./VG_XenStorage--3b6e386d--8736--6b6b--7006--f3f4df9bd586-VHD--02e8b56b--279c--4d5b--8870--b3c5f2255dc7
>>
>> locator: : 1
>> code : PLAT_CODE_W2KU
>> data_space : 512
>> data_length : 206
>> data_offset : 4327936
>> decoded name :
>> ./VG_XenStorage--3b6e386d--8736--6b6b--7006--f3f4df9bd586-VHD--02e8b56b--279c--4d5b--8870--b3c5f2255dc7
>>
>> locator: : 2
>> code : PLAT_CODE_W2RU
>> data_space : 512
>> data_length : 206
>> data_offset : 4328448
>> decoded name :
>> ./VG_XenStorage--3b6e386d--8736--6b6b--7006--f3f4df9bd586-VHD--02e8b56b--279c--4d5b--8870--b3c5f2255dc7
>>
>> VHD Batmap Summary:
>> -------------------
>> Batmap offset : 4196352
>> Batmap size (secs) : 256
>> Batmap version : 0x00010002
>> Checksum : 0xffffff7f|0xffffff7f (Good!)
>>
>> ', stderr: ''
>> [8756] 2015-12-07 21:50:45.132293 ['/usr/sbin/vhd-util', 'query',
>> '--debug', '-s', '-n',
>> '/dev/VG_XenStorage-3b6e386d-8736-6b6b-7006-f3f4df9bd586/VHD-1a240d45-ee0a-4c30-809b-3114dfaf85ba']
>> [8756] 2015-12-07 21:50:45.145222 SUCCESS
>> [8756] 2015-12-07 21:50:45.145502 lock: tried lock
>> /var/lock/sm/3b6e386d-8736-6b6b-7006-f3f4df9bd586/sr, acquired: True
>> (exists: True)
>> [8756] 2015-12-07 21:50:45.145635 lock: released
>> /var/lock/sm/3b6e386d-8736-6b6b-7006-f3f4df9bd586/sr
>> [8756] 2015-12-07 21:50:45.145752 ['/usr/sbin/lvchange',
>> '/dev/VG_XenStorage-3b6e386d-8736-6b6b-7006-f3f4df9bd586/VHD-1a240d45-ee0a-4c30-809b-3114dfaf85ba',
>> '-p', 'r']
>> <8756> 2015-12-07 21:50:45.229864
>> *~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*
>> <8756> 2015-12-07 21:50:45.229919 ***********************
>> <8756> 2015-12-07 21:50:45.229969 * E X C E P T I O N *
>> <8756> 2015-12-07 21:50:45.230017 ***********************
>> <8756> 2015-12-07 21:50:45.230077 coalesce: EXCEPTION util.SMException,
>> VHD *1a240d45[VHD](20.000G//136.000M|n) corrupted
>> <8756> 2015-12-07 21:50:45.230129 File
>> "/opt/xensource/sm/cleanup.py", line 1397, in coalesce
>> self._coalesce(vdi)
>> File "/opt/xensource/sm/cleanup.py", line 1587, in _coalesce
>> vdi._doCoalesce()
>> File "/opt/xensource/sm/cleanup.py", line 1050, in _doCoalesce
>> self.parent.validate()
>> File "/opt/xensource/sm/cleanup.py", line 1043, in validate
>> VDI.validate(self, fast)
>> File "/opt/xensource/sm/cleanup.py", line 633, in validate
>> raise util.SMException("VHD %s corrupted" % self)
>>
>> <8756> 2015-12-07 21:50:45.230181
>> *~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*~*
>> <8756> 2015-12-07 21:50:45.230234 Coalesce failed, skipping
>>
>>
>> Sorry for possible mistakes, but after working for 13 hours, my
>> concentration has dropped and thank you for your help.
>>
>> Regards,
>> F.
> Find out more about ShapeBlue and our range of CloudStack related services
>
> IaaS Cloud Design & Build<http://shapeblue.com/iaas-cloud-design-and-build//>
> CSForge – rapid IaaS deployment framework<http://shapeblue.com/csforge/>
> CloudStack Consulting<http://shapeblue.com/cloudstack-consultancy/>
> CloudStack Software
> Engineering<http://shapeblue.com/cloudstack-software-engineering/>
> CloudStack Infrastructure
> Support<http://shapeblue.com/cloudstack-infrastructure-support/>
> CloudStack Bootcamp Training
> Courses<http://shapeblue.com/cloudstack-training/>
>
> This email and any attachments to it may be confidential and are intended
> solely for the use of the individual to whom it is addressed. Any views or
> opinions expressed are solely those of the author and do not necessarily
> represent those of Shape Blue Ltd or related companies. If you are not the
> intended recipient of this email, you must neither take any action based upon
> its contents, nor copy or show it to anyone. Please contact the sender if you
> believe you have received this email in error. Shape Blue Ltd is a company
> incorporated in England & Wales. ShapeBlue Services India LLP is a company
> incorporated in India and is operated under license from Shape Blue Ltd.
> Shape Blue Brasil Consultoria Ltda is a company incorporated in Brasil and is
> operated under license from Shape Blue Ltd. ShapeBlue SA Pty Ltd is a company
> registered by The Republic of South Africa and is traded under license from
> Shape Blue Ltd. ShapeBlue is a registered trademark.
