** Changed in: glance
Importance: Undecided => High
** Changed in: glance-store
Importance: Undecided => High
** Changed in: glance-store
Status: In Progress => Fix Committed
** Changed in: glance
Status: Fix Released => Triaged
** Changed in: glance
Milestone: kilo-2 => None
** Changed in: glance-store
Milestone: None => v0.1.11
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Glance.
https://bugs.launchpad.net/bugs/1401778
Title:
500 error returned while uploading image using multi filesystem store
and 'filesystem_store_metadata_file' option enabled
Status in OpenStack Image Registry and Delivery Service (Glance):
Triaged
Status in OpenStack Glance backend store-drivers library (glance_store):
Fix Committed
Bug description:
When we try to upload image with multi filesystem store enabled and
'filesystem_store_metadata_file'
(containing list of all directories configured for multi filesystem store) is
provided
then glance throws 'HTTPInternalServerError (HTTP 500)' error.
- Glance Configuration:
1. /etc/glance/glance-api.conf
[DEFAULT]
show_multiple_locations = True
filesystem_store_metadata_file = /etc/glance/metadata.json
[glance_store]
filesystem_store_datadirs = /var/lib/glance/images1/:1
filesystem_store_datadirs = /var/lib/glance/images2/:2
2. /etc/glance/metadata.json
[
{
"id": "f0781415-cf81-47cd-8860-b83f9c2a415c",
"mountpoint": "/var/lib/glance/images1/"
},
{
"id": "5d2dd1db-8684-46bb-880f-b94a1942cfd2",
"mountpoint": "/var/lib/glance/images2/"
}
]
3. 'df -ha' command result:
openstack@openstack:/opt/stack$ df -ha
Filesystem Size Used Avail Use% Mounted on
proc 0 0 0 - /proc
sysfs 0 0 0 - /sys
tmpfs 799M 600K 798M 1% /run
/dev/sda1 236M 41M 183M 19% /boot
nfsd 0 0 0 - /proc/fs/nfsd
10.69.4.173:/export/images2 443G 23G 398G 6%
/var/lib/glance/images2
10.69.4.172:/export/images1 447G 8.7G 415G 3%
/var/lib/glance/images1
openstack@openstack:/opt/stack$
- Steps to reproduce:
1. Create image:
glance --os-image-api-version 2 image-create --name Test123 --disk-format raw
--container-format ami
2. Upload image data:
openstack@openstack-150:~$ glance --os-image-api-version 2 image-upload
47d39050-cc7e-498a-a800-4faf80a72c93 < /home/openstack/workbook/test.py
HTTPInternalServerError (HTTP 500)
openstack@openstack-150:~$
- glance-api.log :
2014-12-11 22:16:59.586 3495 ERROR glance_store.backend
[95987e95-dcae-4516-b57e-87fbd9135ff3 0080647f6a2145f8a40bace67654a058
48f94106d3b24ca2a0a9e2951c505bf9 - - -] The storage driver
<glance_store._drivers.filesystem.Store
object at 0x7fa1b184e390> returned invalid metadata [{u'mountpoint':
u'/opt/stack/data/glance/images/', u'id':
u'f0781415-cf81-47cd-8860-b83f9c2a415c'}, {u'mountpoint':
u'/opt/stack/data/glance/images1/', u'id': u'5d2dd1db-
8684-46bb-880f-b94a1942cfd2'}]. This must be a dictionary type
2014-12-11 22:16:59.591 3495 ERROR glance.api.v2.image_data
[95987e95-dcae-4516-b57e-87fbd9135ff3 0080647f6a2145f8a40bace67654a058
48f94106d3b24ca2a0a9e2951c505bf9 - - -] Failed to upload image data due to
internal error
2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data Traceback (most
recent call last):
2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data File
"/opt/stack/glance/glance/api/v2/image_data.py", line 74, in upload
2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data
image.set_data(data, size)
2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data File
"/opt/stack/glance/glance/domain/proxy.py", line 160, in set_data
2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data
self.base.set_data(data, size)
2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data File
"/opt/stack/glance/glance/notifier.py", line 252, in set_data
2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data
self.notifier.error('image.upload', msg)
2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data File
"/usr/local/lib/python2.7/dist-packages/oslo/utils/excutils.py", line 82, in
__exit
2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data
six.reraise(self.type_, self.value, self.tb)
2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data File
"/opt/stack/glance/glance/notifier.py", line 201, in set_data
2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data
self.image.set_data(data, size)
2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data File
"/opt/stack/glance/glance/api/policy.py", line 176, in set_data
2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data return
self.image.set_data(*args, **kwargs)
2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data File
"/opt/stack/glance/glance/quota/__init__.py", line 296, in set_data
2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data
self.image.set_data(data, size=size)
2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data File
"/opt/stack/glance/glance/location.py", line 364, in set_data
2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data
context=self.context)
2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data File
"/usr/local/lib/python2.7/dist-packages/glance_store/backend.py", line 357, in
add_to_backend
2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data return
store_add_to_backend(image_id, data, size, store, context)
2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data File
"/usr/local/lib/python2.7/dist-packages/glance_store/backend.py", line 338, in
store_add_to_backend
2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data raise
exceptions.BackendException(msg)
2014-12-11 22:16:59.591 3495 TRACE glance.api.v2.image_data BackendException:
The storage driver <glance_store._drivers.filesystem.Store object at
0x7fa1b184e390> returned invalid metadata [{u'mountpoint':
u'/opt/stack/data/glance/images/', u'id':
u'f0781415-cf81-47cd-8860-b83f9c2a415c'}, {u'mountpoint':
u'/opt/stack/data/glance/images1/', u'id':
u'5d2dd1db-8684-46bb-880f-b94a1942cfd2'}]. This must be a dictionary type
When we enable 'filesystem_store_metadata_file' file option in
glance-api.conf then it adds this json data as metadata for all glance images.
This data is then used by NOVA to directly copy image using 'cp' command.
(when glance image directories are shared between nova-compute and glance)
For direct download in nova.conf we need to enable
'allowed_direct_url_schemes' option.
Also nova needs 'id' and 'moutpoint' keys to be present in glance-image
metadata for direct image copy.
If there is only one glance image directory (ie- one nfs mountpoint) then we
do not get this '500 internal server error'.
But if there are multiple image data directories configured and in metadata
JSON file we give list of dicts containing all
mountpoints then we get above error.
IMO we need to validate glance metadata JSON file (for 'id' and
'mountpoint' keys) before adding it as image-metadata and if there are
multiple mountpoints (directories) configured for glance then we need
to add appropriate metadata to glance-image instead of adding all
image metadata which is provided in JSON file.
To manage notifications about this bug go to:
https://bugs.launchpad.net/glance/+bug/1401778/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
