Public bug reported: Description: After upgrading Glance from Dalmatian to Epoxy, uploading an image to an S3-compatible storage backend fails. The failure only occurs on Epoxy and correlates with the boto3 1.36.x upgrade, which enabled new default S3 checksum behaviors. Many S3-compatible storage services (e.g., Ceph RGW) do not yet support these behaviors, which causes boto3/botocore to raise errors and the Glance upload to fail.
This is a regression from Dalmatian (uploads succeeded there). Environment: Glance - Epoxy boto3 - 1.36.26 (installed via Epoxy upper-constraints; https://github.com/openstack/requirements/blob/bfd91c04caf498be2bfda0732a2a0f9fa9306e07/upper-constraints.txt#L306) Storage backend - Ceph RGW (S3-compatible storage) Steps to Reproduce: 1. Configure Glance to use an S3-compatible storage backend (e.g., Ceph RGW). 2. Attempt to upload any image and observe that the upload fails, for example; ``` $ openstack image create "cirros-0.6.2-x86_64" --file ./cirros-0.6.2-x86_64-disk.img --disk-format qcow2 --container-format bare HttpException: 500: Server Error for url: https://XXXXXXXXXX/v2/images/258cdf6f-ba48-4c03-8cf1-d90114f5da8e/file, Internal Server Error Error: exit status 1 ``` error log; ``` 2025-08-21T07:08:05.467 13 ERROR glance.api.v2.image_data [F] [None req-1139a1ef-1339-4f34-aa19-844de43261cb f209e8b906b549fe987161b2c42dbbfd 604177366ff8401a911cf16aacfe6b4b - - default default] Failed to upload image data due to internal error: botocore.exceptions.ClientError: An error occurred (MissingContentLength) when calling the PutObject operation: The internal error code is -2011 ``` Suspected Root Cause: Starting in boto3/botocore 1.36.0, the SDK enables new S3 “data integrity” protections by default (request checksum calculation and response checksum validation). S3-compatible storage that does not implement these features rejects the new headers or fails validation, causing Glance uploads to fail. Related: https://github.com/boto/boto3/issues/4392 Workaround: Set the following environment variables for the Glance API process so that checksum behavior is only used when required by the API; ``` export AWS_REQUEST_CHECKSUM_CALCULATION=WHEN_REQUIRED export AWS_RESPONSE_CHECKSUM_VALIDATION=WHEN_REQUIRED ``` Guidance: https://github.com/boto/boto3/issues/4398#issuecomment-2727003405 Proposed Fix (ideas): - In Glance/glance_store’s S3 path, default boto3’s S3 client config to request_checksum_calculation="when_required" and response_checksum_validation="when_required". - Document the required environment variables prominently in Glance Epoxy release notes. ** Affects: glance Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to Glance. https://bugs.launchpad.net/bugs/2121144 Title: Image uploads to S3-compatible storage fail with Epoxy version after upgrade from Dalmatian due to boto3 checksum behavior changes Status in Glance: New Bug description: Description: After upgrading Glance from Dalmatian to Epoxy, uploading an image to an S3-compatible storage backend fails. The failure only occurs on Epoxy and correlates with the boto3 1.36.x upgrade, which enabled new default S3 checksum behaviors. Many S3-compatible storage services (e.g., Ceph RGW) do not yet support these behaviors, which causes boto3/botocore to raise errors and the Glance upload to fail. This is a regression from Dalmatian (uploads succeeded there). Environment: Glance - Epoxy boto3 - 1.36.26 (installed via Epoxy upper-constraints; https://github.com/openstack/requirements/blob/bfd91c04caf498be2bfda0732a2a0f9fa9306e07/upper-constraints.txt#L306) Storage backend - Ceph RGW (S3-compatible storage) Steps to Reproduce: 1. Configure Glance to use an S3-compatible storage backend (e.g., Ceph RGW). 2. Attempt to upload any image and observe that the upload fails, for example; ``` $ openstack image create "cirros-0.6.2-x86_64" --file ./cirros-0.6.2-x86_64-disk.img --disk-format qcow2 --container-format bare HttpException: 500: Server Error for url: https://XXXXXXXXXX/v2/images/258cdf6f-ba48-4c03-8cf1-d90114f5da8e/file, Internal Server Error Error: exit status 1 ``` error log; ``` 2025-08-21T07:08:05.467 13 ERROR glance.api.v2.image_data [F] [None req-1139a1ef-1339-4f34-aa19-844de43261cb f209e8b906b549fe987161b2c42dbbfd 604177366ff8401a911cf16aacfe6b4b - - default default] Failed to upload image data due to internal error: botocore.exceptions.ClientError: An error occurred (MissingContentLength) when calling the PutObject operation: The internal error code is -2011 ``` Suspected Root Cause: Starting in boto3/botocore 1.36.0, the SDK enables new S3 “data integrity” protections by default (request checksum calculation and response checksum validation). S3-compatible storage that does not implement these features rejects the new headers or fails validation, causing Glance uploads to fail. Related: https://github.com/boto/boto3/issues/4392 Workaround: Set the following environment variables for the Glance API process so that checksum behavior is only used when required by the API; ``` export AWS_REQUEST_CHECKSUM_CALCULATION=WHEN_REQUIRED export AWS_RESPONSE_CHECKSUM_VALIDATION=WHEN_REQUIRED ``` Guidance: https://github.com/boto/boto3/issues/4398#issuecomment-2727003405 Proposed Fix (ideas): - In Glance/glance_store’s S3 path, default boto3’s S3 client config to request_checksum_calculation="when_required" and response_checksum_validation="when_required". - Document the required environment variables prominently in Glance Epoxy release notes. To manage notifications about this bug go to: https://bugs.launchpad.net/glance/+bug/2121144/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
