Public bug reported: UpdateImage call in v2 ( PATCH /v2/images/%imageid%) fetches the whole image object from database (https://github.com/openstack/glance/blob/master/glance/api/v2/images.py#L119), modifies some of its properties with appropriate matching methods (_do_replace, _do_add etc) and then saves the object in ImageRepo
The imageRepo takes the object, renders it to dictionary (https://github.com/openstack/glance/blob/master/glance/db/__init__.py#L169) and then passes this dictionary to db_api.image_update(), which eventually executes UPDATE query and replaces all the fields in database with these values. This is not transactional, and other operations may interfere between image fetching and update query. For example, imagine we have an image with name "test" and min_ram "10". If we concurrently execute two image-update requests: 'image-update --name test2' and 'image-update --min-ram 100' A race condition occurs. The result image is undetermined: it may be either {'name': 'test', 'min_ram': 100} or {'name': 'test2', 'min_ram': 10} or {'name': 'test2', 'min_ram': 100'} - depending on which request fetches the object first. It may be had to reproduce this in normal circumstances, but once the database load becomes higher, it may show up. To reproduce it for sure one may add a event let.sleep() command after line 119 of glance/api/v2/images.py, to increase the timespan between image fetching and image update. ** Affects: glance Importance: Undecided Assignee: Alexander Tivelkov (ativelkov) Status: New ** Description changed: UpdateImage call in v2 ( PATCH /v2/images/%imageid%) fetches the whole image object from database (https://github.com/openstack/glance/blob/master/glance/api/v2/images.py#L119), modifies some of its properties with appropriate matching methods - (_do_raplace, _do_add etc) and then saves the object in ImageRepo + (_do_replace, _do_add etc) and then saves the object in ImageRepo The imageRepo takes the object, renders it to dictionary (https://github.com/openstack/glance/blob/master/glance/db/__init__.py#L169) and then passes this dictionary to db_api.image_update(), which eventually executes UPDATE query and replaces all the fields in database with these values. This is not transactional, and other operations may interfere between image fetching and update query. - For example, imagine we have an image with name "test" and min_ram "10". + For example, imagine we have an image with name "test" and min_ram "10". If we concurrently execute two image-update requests: - 'image-update --name test2' + 'image-update --name test2' and - 'image-update --min-ram 100' + 'image-update --min-ram 100' A race condition occurs. The result image is undetermined: it may be either {'name': 'test', 'min_ram': 100} or {'name': 'test2', 'min_ram': 10} or {'name': 'test2', 'min_ram': 100'} - depending on which request fetches the object first. It may be had to reproduce this in normal circumstances, but once the database load becomes higher, it may show up. To reproduce it for sure one may add a event let.sleep() command after line 119 of glance/api/v2/images.py, to increase the timespan between image fetching and image update. ** Changed in: glance Assignee: (unassigned) => Alexander Tivelkov (ativelkov) -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to Glance. https://bugs.launchpad.net/bugs/1371728 Title: Concurrency Update issue in v2 Status in OpenStack Image Registry and Delivery Service (Glance): New Bug description: UpdateImage call in v2 ( PATCH /v2/images/%imageid%) fetches the whole image object from database (https://github.com/openstack/glance/blob/master/glance/api/v2/images.py#L119), modifies some of its properties with appropriate matching methods (_do_replace, _do_add etc) and then saves the object in ImageRepo The imageRepo takes the object, renders it to dictionary (https://github.com/openstack/glance/blob/master/glance/db/__init__.py#L169) and then passes this dictionary to db_api.image_update(), which eventually executes UPDATE query and replaces all the fields in database with these values. This is not transactional, and other operations may interfere between image fetching and update query. For example, imagine we have an image with name "test" and min_ram "10". If we concurrently execute two image-update requests: 'image-update --name test2' and 'image-update --min-ram 100' A race condition occurs. The result image is undetermined: it may be either {'name': 'test', 'min_ram': 100} or {'name': 'test2', 'min_ram': 10} or {'name': 'test2', 'min_ram': 100'} - depending on which request fetches the object first. It may be had to reproduce this in normal circumstances, but once the database load becomes higher, it may show up. To reproduce it for sure one may add a event let.sleep() command after line 119 of glance/api/v2/images.py, to increase the timespan between image fetching and image update. To manage notifications about this bug go to: https://bugs.launchpad.net/glance/+bug/1371728/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
