Nevermind, problem was the dom_md/metadata file was bad (Pool uid of the
datacenter from which I could not detach was missing).
On 17/4/19 13:39, Eduardo Mayoral wrote:
> Hi.
>
> When trying to detach an ISO datastore I get the following error. Also
> fails if I try to switch the datastore from maintenance to active.
>
>
> 2019-04-17 10:07:23,100+0000 INFO (jsonrpc/6) [jsonrpc.JsonRpcServer]
> RPC call StoragePool.connectStorageServer succeeded in 0.01 seconds
> (__init__:312)
> 2019-04-17 10:07:23,104+0000 INFO (jsonrpc/4) [vdsm.api] START
> detachStorageDomain(sdUUID=u'2722942e-0887-439c-81f0-70b60d860060',
> spUUID=u'6901a281-c414-42c6-8b41-3cf6e6bcd788',
> msdUUID=u'00000000-0000-0000-0000-000000000000', masterVersion=14,
> options=None) from=::fff
> f:10.5.72.63,58618, flow_id=0420c8d0-fb37-4b22-9fea-9ee8bac7d41d,
> task_id=9301405a-eaf6-4f1c-a1b6-6537740e2daf (api:48)
> 2019-04-17 10:07:23,105+0000 INFO (jsonrpc/4) [storage.StoragePool]
> sdUUID=2722942e-0887-439c-81f0-70b60d860060
> spUUID=6901a281-c414-42c6-8b41-3cf6e6bcd788 (sp:1048)
> 2019-04-17 10:07:23,316+0000 INFO (itmap/0) [IOProcessClient]
> (/pre1-svm-templates.por-ngcs.lan:_ds__pre1__templates__iso01_ovirt3)
> Starting client (__init__:308)
> 2019-04-17 10:07:23,324+0000 INFO (itmap/1) [IOProcessClient]
> (/pre1-svm-templates.por-ngcs.lan:_ds__pre1__templates__iso01_ovirt)
> Starting client (__init__:308)
> 2019-04-17 10:07:23,327+0000 INFO (ioprocess/74639) [IOProcess]
> (/pre1-svm-templates.por-ngcs.lan:_ds__pre1__templates__iso01_ovirt3)
> Starting ioprocess (__init__:434)
> 2019-04-17 10:07:23,333+0000 INFO (ioprocess/74645) [IOProcess]
> (/pre1-svm-templates.por-ngcs.lan:_ds__pre1__templates__iso01_ovirt)
> Starting ioprocess (__init__:434)
> 2019-04-17 10:07:23,335+0000 INFO (jsonrpc/4) [IOProcessClient]
> (2722942e-0887-439c-81f0-70b60d860060) Starting client (__init__:308)
> 2019-04-17 10:07:23,344+0000 INFO (ioprocess/74653) [IOProcess]
> (2722942e-0887-439c-81f0-70b60d860060) Starting ioprocess (__init__:434)
> 2019-04-17 10:07:23,350+0000 INFO (jsonrpc/4) [storage.StorageDomain]
> Removing remnants of deleted images [] (fileSD:740)
> 2019-04-17 10:07:23,352+0000 INFO (jsonrpc/4) [vdsm.api] FINISH
> detachStorageDomain error=Storage domain not in pool:
> u'domain=2722942e-0887-439c-81f0-70b60d860060,
> pool=6901a281-c414-42c6-8b41-3cf6e6bcd788' from=::ffff:10.5.72.63,58618,
> flow_id=0420c8d0-fb37-4b22-9fea-
> 9ee8bac7d41d, task_id=9301405a-eaf6-4f1c-a1b6-6537740e2daf (api:52)
> 2019-04-17 10:07:23,352+0000 ERROR (jsonrpc/4)
> [storage.TaskManager.Task] (Task='9301405a-eaf6-4f1c-a1b6-6537740e2daf')
> Unexpected error (task:875)
> Traceback (most recent call last):
> File "/usr/lib/python2.7/site-packages/vdsm/storage/task.py", line
> 882, in _run
> return fn(*args, **kargs)
> File "<string>", line 2, in detachStorageDomain
> File "/usr/lib/python2.7/site-packages/vdsm/common/api.py", line 50,
> in method
> ret = func(*args, **kwargs)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/hsm.py", line 891,
> in detachStorageDomain
> pool.detachSD(sdUUID)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py",
> line 79, in wrapper
> return method(self, *args, **kwargs)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 1053,
> in detachSD
> self.validateAttachedDomain(dom)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/securable.py",
> line 79, in wrapper
> return method(self, *args, **kwargs)
> File "/usr/lib/python2.7/site-packages/vdsm/storage/sp.py", line 551,
> in validateAttachedDomain
> raise se.StorageDomainNotInPool(self.spUUID, dom.sdUUID)
> StorageDomainNotInPool: Storage domain not in pool:
> u'domain=2722942e-0887-439c-81f0-70b60d860060,
> pool=6901a281-c414-42c6-8b41-3cf6e6bcd788'
> 2019-04-17 10:07:23,352+0000 INFO (jsonrpc/4)
> [storage.TaskManager.Task] (Task='9301405a-eaf6-4f1c-a1b6-6537740e2daf')
> aborting: Task is aborted: "Storage domain not in pool:
> u'domain=2722942e-0887-439c-81f0-70b60d860060,
> pool=6901a281-c414-42c6-8b41-3cf6e6bcd788'" - co
> de 353 (task:1181)
> 2019-04-17 10:07:23,353+0000 ERROR (jsonrpc/4) [storage.Dispatcher]
> FINISH detachStorageDomain error=Storage domain not in pool:
> u'domain=2722942e-0887-439c-81f0-70b60d860060,
> pool=6901a281-c414-42c6-8b41-3cf6e6bcd788' (dispatcher:83)
> 2019-04-17 10:07:23,353+0000 INFO (jsonrpc/4) [jsonrpc.JsonRpcServer]
> RPC call StorageDomain.detach failed (error 353) in 0.25 seconds
> (__init__:312)
> 2019-04-17 10:07:26,275+0000 INFO (jsonrpc/2) [api.host] START
> getAllVmStats() from=::ffff:10.5.72.63,58604 (api:48)
> 2019-04-17 10:07:26,275+0000 INFO (jsonrpc/2) [api.host] FINISH
> getAllVmStats return={'status': {'message': 'Done', 'code': 0},
> 'statsList': (suppressed)} from=::ffff:10.5.72.63,58604 (api:54)
> 2019-04-17 10:07:26,275+0000 INFO (jsonrpc/2) [jsonrpc.JsonRpcServer]
> RPC call Host.getAllVmStats succeeded in 0.01 seconds (__init__:312)
> 2019-04-17 10:07:29,675+0000 INFO (jsonrpc/7) [api.host] START
> getStats() from=::ffff:10.5.72.63,58604 (api:48)
> 2019-04-17 10:07:29,739+0000 INFO (jsonrpc/7) [vdsm.api] START
> repoStats(domains=()) from=::ffff:10.5.72.63,58604,
> task_id=3ec94f9f-7c98-4c49-8710-ffbfd9910ff2 (api:48)
> 2019-04-17 10:07:29,739+0000 INFO (jsonrpc/7) [vdsm.api] FINISH
> repoStats return={u'a97bbda5-33e4-4e78-890c-f7d2a572179e': {'code': 0,
> 'actual': True, 'version': 4, 'acquired': True, 'delay': '0.000505524',
> 'lastCheck': '3.6', 'valid': True}, u'4121f681-45c3-413a-9c6e-6
> 9d154710ace': {'code': 0, 'actual': True, 'version': 4, 'acquired':
> True, 'delay': '0.000790301', 'lastCheck': '2.7', 'valid': True}}
> from=::ffff:10.5.72.63,58604,
> task_id=3ec94f9f-7c98-4c49-8710-ffbfd9910ff2 (api:54)
> 2019-04-17 10:07:29,740+0000 INFO (jsonrpc/7) [vdsm.api] START
> multipath_health() from=::ffff:10.5.72.63,58604,
> task_id=f567e03a-8cf4-49dd-84eb-628acb10e1d3 (api:48)
> 2019-04-17 10:07:29,740+0000 INFO (jsonrpc/7) [vdsm.api] FINISH
> multipath_health return={} from=::ffff:10.5.72.63,58604,
> task_id=f567e03a-8cf4-49dd-84eb-628acb10e1d3 (api:54)
>
> Interestingly, I have 2 datacenters, I can attach, detach, set to
> maintenance or activate this same datastore on the second datacenter,
> but not on this one.
>
> So far I have tried updating OVF, setting master datastore and hosts to
> maintenance and re-activating them.
>
> Started happening on 4.3.2, updating to 4.3.3 makes no difference.
>
> This is what I see on the database. Looks OK to me:
>
> engine=# select * from storage_pool;
> id | name | description
> | storage_pool_type | storage_pool_format_type | status |
> master_domain_version | spm_vds_id |
> compatibility_version | _create_date |
> _update_date
> | quota_enforcement_type | free_text_comment | is_local
> --------------------------------------+-------------------+-------------+-------------------+--------------------------+--------+-----------------------+--------------------------------------+-----------------------+-------------------------------+-----------------------
> --------+------------------------+-------------------+----------
> 039aef77-77c8-4f04-8d17-22f4b655df28 | dc-pre1-vc03-cl02 |
> | | 4 | 1
> | 1 | 42c969f3-a2a6-4784-8506-e6fbc731da9d |
> 4.3 | 2019-03-14 12:13:24.411468+00 | 2019-04-17
> 08:38:05.53
> 8042+00 | 0 | | f
> 6901a281-c414-42c6-8b41-3cf6e6bcd788 | dc-pre1-vc03-cl01 |
> | | 4 | 1
> | 15 | 089cdde4-ecbf-4054-82f9-c106b046d73a |
> 4.3 | 2019-03-14 12:13:24.365529+00 | 2019-04-17
> 11:10:46.42
> 2305+00 | 0 | | f
> (2 rows)
>
> engine=# select * from storage_pool_iso_map ;
> storage_id |
> storage_pool_id | status
> --------------------------------------+--------------------------------------+--------
> 2722942e-0887-439c-81f0-70b60d860060 |
> 039aef77-77c8-4f04-8d17-22f4b655df28 | 6
> 2722942e-0887-439c-81f0-70b60d860060 |
> 6901a281-c414-42c6-8b41-3cf6e6bcd788 | 6
> 5d390949-24ec-45e2-bb6d-43b3f30bd286 |
> 039aef77-77c8-4f04-8d17-22f4b655df28 | 3
> 94cb1a4f-a267-479c-88e7-ea9c32e3fd24 |
> 039aef77-77c8-4f04-8d17-22f4b655df28 | 3
> 4121f681-45c3-413a-9c6e-69d154710ace |
> 6901a281-c414-42c6-8b41-3cf6e6bcd788 | 3
> a97bbda5-33e4-4e78-890c-f7d2a572179e |
> 6901a281-c414-42c6-8b41-3cf6e6bcd788 | 3
> (6 rows)
>
>
> Apart from root cause. I would be interested in recovering from the
> error, detaching the ISO storage, even if it is manually. This
> datacenter is not in production yet, so I have options on the things I
> can try.
>
>
>
--
Eduardo Mayoral Jimeno
Systems engineer, platform department. Arsys Internet.
[email protected] - +34 941 620 105 - ext 2153
_______________________________________________
Users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
Privacy Statement: https://www.ovirt.org/site/privacy-policy/
oVirt Code of Conduct:
https://www.ovirt.org/community/about/community-guidelines/
List Archives:
https://lists.ovirt.org/archives/list/[email protected]/message/RQOKJXZBDGMQM25QXIN5DBIKHMCWTSRF/
