Public bug reported: When creating an image from a server using the `openstack server image create` command, a snapshot is created by nova and uploaded to S3 via glanceclient upload method.
The size of the image is not supplied to glanceclient (however it can be known to nova by checking the size of temporary file nova created), and therefore it will be set to 0 when its None. This causes the S3 driver to assume the size is smaller than the store_large_object_size and call the _add_singlepart method instead of _add_multipart. This results in poor upload performance and high memory usage on the glance-api server because the whole upload is read in memory. The (virtual) size will eventually be calculated when the upload to S3 is completed, and then get saved in the database. Steps to reproduce: * Configure an S3 backend like MinIO as default_backend in devstack * openstack server create --flavor m1.small --image debian12 --network public vm1 * openstack server image create vm1 Expected result: * Image is uploaded to S3 in parts using multipart Actual result: * Image is uploaded to S3 using _add_singlepart, causing high memory consumption and duration. Performance impact singlepart vs multipart: singlepart: May 01 19:54:23 devstack-1 nova-compute[502984]: INFO nova.virt.libvirt.driver [None req-cf59e8a9-d12a-4424-8221-9e64fb62b5f7 demo admin] [instance: c3f80570-8ce5-40c4-a815-6acad873bba5] Snapshot extracted, beginning image upload May 01 19:55:49 devstack-1 nova-compute[502984]: INFO nova.virt.libvirt.driver [None req-cf59e8a9-d12a-4424-8221-9e64fb62b5f7 demo admin] [instance: c3f80570-8ce5-40c4-a815-6acad873bba5] Snapshot image upload complete May 01 19:55:49 devstack-1 nova-compute[502984]: INFO nova.compute.manager [None req-cf59e8a9-d12a-4424-8221-9e64fb62b5f7 demo admin] [instance: c3f80570-8ce5-40c4-a815-6acad873bba5] Took 89.70 seconds to snapshot the instance on the hypervisor. multipart: May 01 19:57:24 devstack-1 nova-compute[502984]: INFO nova.virt.libvirt.driver [None req-272205be-2fea-417d-8412-2a2a45daefb5 demo admin] [instance: c3f80570-8ce5-40c4-a815-6acad873bba5] Snapshot extracted, beginning image upload May 01 19:57:38 devstack-1 nova-compute[502984]: INFO nova.virt.libvirt.driver [None req-272205be-2fea-417d-8412-2a2a45daefb5 demo admin] [instance: c3f80570-8ce5-40c4-a815-6acad873bba5] Snapshot image upload complete May 01 19:57:38 devstack-1 nova-compute[502984]: INFO nova.compute.manager [None req-272205be-2fea-417d-8412-2a2a45daefb5 demo admin] [instance: c3f80570-8ce5-40c4-a815-6acad873bba5] Took 18.83 seconds to snapshot the instance on the hypervisor I've tried forcing multipart uploads by using this patch https://review.opendev.org/c/openstack/glance_store/+/946382/1/glance_store/_drivers/s3.py and setting the large_object_size to 0. This fixes the issue, but creates another issue when creating an image of volume-backed instances. In that case, a 0 byte dummy object is created as placeholder, and metadata with a reference to the volume snapshot id is saved in glance. This errors out: ERROR glance.common.wsgi botocore.errorfactory.InvalidRequest: An error occurred (InvalidRequest) when calling the CompleteMultipartUpload operation: You must specify at least one part Related to this similar bug: https://bugs.launchpad.net/glance-store/+bug/2043740 ** Affects: glance-store Importance: Undecided Status: New ** Affects: nova Importance: Undecided Status: New ** Tags: glance glance-store glanceclient multipart singlepart ** Also affects: nova Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/2110222 Title: Nova server image uploaded to S3 uses singlepart instead of multipart upload Status in glance_store: New Status in OpenStack Compute (nova): New Bug description: When creating an image from a server using the `openstack server image create` command, a snapshot is created by nova and uploaded to S3 via glanceclient upload method. The size of the image is not supplied to glanceclient (however it can be known to nova by checking the size of temporary file nova created), and therefore it will be set to 0 when its None. This causes the S3 driver to assume the size is smaller than the store_large_object_size and call the _add_singlepart method instead of _add_multipart. This results in poor upload performance and high memory usage on the glance-api server because the whole upload is read in memory. The (virtual) size will eventually be calculated when the upload to S3 is completed, and then get saved in the database. Steps to reproduce: * Configure an S3 backend like MinIO as default_backend in devstack * openstack server create --flavor m1.small --image debian12 --network public vm1 * openstack server image create vm1 Expected result: * Image is uploaded to S3 in parts using multipart Actual result: * Image is uploaded to S3 using _add_singlepart, causing high memory consumption and duration. Performance impact singlepart vs multipart: singlepart: May 01 19:54:23 devstack-1 nova-compute[502984]: INFO nova.virt.libvirt.driver [None req-cf59e8a9-d12a-4424-8221-9e64fb62b5f7 demo admin] [instance: c3f80570-8ce5-40c4-a815-6acad873bba5] Snapshot extracted, beginning image upload May 01 19:55:49 devstack-1 nova-compute[502984]: INFO nova.virt.libvirt.driver [None req-cf59e8a9-d12a-4424-8221-9e64fb62b5f7 demo admin] [instance: c3f80570-8ce5-40c4-a815-6acad873bba5] Snapshot image upload complete May 01 19:55:49 devstack-1 nova-compute[502984]: INFO nova.compute.manager [None req-cf59e8a9-d12a-4424-8221-9e64fb62b5f7 demo admin] [instance: c3f80570-8ce5-40c4-a815-6acad873bba5] Took 89.70 seconds to snapshot the instance on the hypervisor. multipart: May 01 19:57:24 devstack-1 nova-compute[502984]: INFO nova.virt.libvirt.driver [None req-272205be-2fea-417d-8412-2a2a45daefb5 demo admin] [instance: c3f80570-8ce5-40c4-a815-6acad873bba5] Snapshot extracted, beginning image upload May 01 19:57:38 devstack-1 nova-compute[502984]: INFO nova.virt.libvirt.driver [None req-272205be-2fea-417d-8412-2a2a45daefb5 demo admin] [instance: c3f80570-8ce5-40c4-a815-6acad873bba5] Snapshot image upload complete May 01 19:57:38 devstack-1 nova-compute[502984]: INFO nova.compute.manager [None req-272205be-2fea-417d-8412-2a2a45daefb5 demo admin] [instance: c3f80570-8ce5-40c4-a815-6acad873bba5] Took 18.83 seconds to snapshot the instance on the hypervisor I've tried forcing multipart uploads by using this patch https://review.opendev.org/c/openstack/glance_store/+/946382/1/glance_store/_drivers/s3.py and setting the large_object_size to 0. This fixes the issue, but creates another issue when creating an image of volume-backed instances. In that case, a 0 byte dummy object is created as placeholder, and metadata with a reference to the volume snapshot id is saved in glance. This errors out: ERROR glance.common.wsgi botocore.errorfactory.InvalidRequest: An error occurred (InvalidRequest) when calling the CompleteMultipartUpload operation: You must specify at least one part Related to this similar bug: https://bugs.launchpad.net/glance-store/+bug/2043740 To manage notifications about this bug go to: https://bugs.launchpad.net/glance-store/+bug/2110222/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
