Public bug reported: Hi, For the past few releases (Caracal, Bobcat, maybe earlier) we have noticed that listing images is particularly slow. I've finally had some time to dig into this, and I believe I've tracked it to https://review.opendev.org/c/openstack/glance/+/886811
We're running the RBD back end primarily, but also have HTTP and Cinder listed in 'enabled_backends', meaning that sort_image_locations executes in full. It appears that when this function executes for the RBD back end, it causes a connection to be opened to the back end. When doing a full image list operation, this happens once for every image in the list (the connection is not re-used). This appears to carry a 20-30ms time penalty per image. As such, for any reasonable set of images the response ends up taking several seconds. In our case, images are unlikely to be held in more than one back end at a time, and I noted that adding a length check to the locations list in https://github.com/openstack/glance/blob/master/glance/common/utils.py#L718 so that the sorting doesn't occur when the list has just one element resolves the performance issue entirely. Whilst a length check is a workaround, does the sorting operation actually require connections to be opened to the RBD back end? If they are required, could the connections at least be re-used to avoid this time penalty growing linearly with the number of images held by Glance? Thanks ** Affects: glance Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to Glance. https://bugs.launchpad.net/bugs/2086675 Title: Suspected performance regression for RBD back end linked to location sorting Status in Glance: New Bug description: Hi, For the past few releases (Caracal, Bobcat, maybe earlier) we have noticed that listing images is particularly slow. I've finally had some time to dig into this, and I believe I've tracked it to https://review.opendev.org/c/openstack/glance/+/886811 We're running the RBD back end primarily, but also have HTTP and Cinder listed in 'enabled_backends', meaning that sort_image_locations executes in full. It appears that when this function executes for the RBD back end, it causes a connection to be opened to the back end. When doing a full image list operation, this happens once for every image in the list (the connection is not re-used). This appears to carry a 20-30ms time penalty per image. As such, for any reasonable set of images the response ends up taking several seconds. In our case, images are unlikely to be held in more than one back end at a time, and I noted that adding a length check to the locations list in https://github.com/openstack/glance/blob/master/glance/common/utils.py#L718 so that the sorting doesn't occur when the list has just one element resolves the performance issue entirely. Whilst a length check is a workaround, does the sorting operation actually require connections to be opened to the RBD back end? If they are required, could the connections at least be re-used to avoid this time penalty growing linearly with the number of images held by Glance? Thanks To manage notifications about this bug go to: https://bugs.launchpad.net/glance/+bug/2086675/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
