Public bug reported: Glance may loose some image data while transferring it to the backing store thus corrupting the image when ALL the following conditions are met:
- Image is being created by copying data from remote source (--copy-from CLI parameter or appropriate API call) - Backing store is Swift - Image size is larger then configured "swift_store_large_object_size" In such scenarios the last chunk stored in Swift will have the size significantly less then expected. An attempt to download the image will result in a checksum verification error, however the checksum stored in Glance (with image metadata) is correct, and so is the size. This is easily reproducible even on devstack (if the devstack is configured to run Swift as Glance backend). Just decrease 'swift_store_large_object_size' to some reasonably low value (i.e. 200 Mb) and try to copy-from any image which is larger then that value. After the upload is successful, check the object size in swift (by either summing the sizes of all the chunks or by looking to the size of virtual large object) - they will be lower then expected: glance image-create --name tst --disk-format qcow2 --container-format bare --copy-from http://192.168.56.1:8000/F18-x86_64-cfntools.qcow2 ... glance image-list +--------------------------------------+---------------------------------+-------------+------------------+-----------+--------+ | ID | Name | Disk Format | Container Format | Size | Status | +--------------------------------------+---------------------------------+-------------+------------------+-----------+--------+ | fc34ec49-4bd3-40dd-918f-44d3254f2ac9 | tst | qcow2 | bare | 536412160 | active | +--------------------------------------+---------------------------------+-------------+------------------+-----------+--------+ ... swift stat glance fc34ec49-4bd3-40dd-918f-44d3254f2ac9 --os-tenant-name service --os-username admin Account: AUTH_cce6e9c12fa34c63b64ef29e84861554 Container: glance Object: fc34ec49-4bd3-40dd-918f-44d3254f2ac9 Content Type: application/octet-stream Content Length: 509804544 <---- see, the size is different! Last Modified: Mon, 19 Jan 2015 15:52:18 GMT ETag: "6d0612f82db9a531b34d25823a45073d" Manifest: glance/fc34ec49-4bd3-40dd-918f-44d3254f2ac9- Accept-Ranges: bytes X-Timestamp: 1421682737.01148 X-Trans-Id: tx01a19f7476a541808c9a1-0054bd28e1 .... glance image-download tst --file out.qcow2 [Errno 32] Corrupt image download. Checksum was 0eeddae1007f01b0029136d28518f538 expected 3ecddfe0787a392960d230c87a421c6a ** Affects: glance Importance: Undecided Assignee: Alexander Tivelkov (ativelkov) Status: New ** Changed in: glance Assignee: (unassigned) => Alexander Tivelkov (ativelkov) -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to Glance. https://bugs.launchpad.net/bugs/1412802 Title: copy-from broken for large files and swift Status in OpenStack Image Registry and Delivery Service (Glance): New Bug description: Glance may loose some image data while transferring it to the backing store thus corrupting the image when ALL the following conditions are met: - Image is being created by copying data from remote source (--copy-from CLI parameter or appropriate API call) - Backing store is Swift - Image size is larger then configured "swift_store_large_object_size" In such scenarios the last chunk stored in Swift will have the size significantly less then expected. An attempt to download the image will result in a checksum verification error, however the checksum stored in Glance (with image metadata) is correct, and so is the size. This is easily reproducible even on devstack (if the devstack is configured to run Swift as Glance backend). Just decrease 'swift_store_large_object_size' to some reasonably low value (i.e. 200 Mb) and try to copy-from any image which is larger then that value. After the upload is successful, check the object size in swift (by either summing the sizes of all the chunks or by looking to the size of virtual large object) - they will be lower then expected: glance image-create --name tst --disk-format qcow2 --container-format bare --copy-from http://192.168.56.1:8000/F18-x86_64-cfntools.qcow2 ... glance image-list +--------------------------------------+---------------------------------+-------------+------------------+-----------+--------+ | ID | Name | Disk Format | Container Format | Size | Status | +--------------------------------------+---------------------------------+-------------+------------------+-----------+--------+ | fc34ec49-4bd3-40dd-918f-44d3254f2ac9 | tst | qcow2 | bare | 536412160 | active | +--------------------------------------+---------------------------------+-------------+------------------+-----------+--------+ ... swift stat glance fc34ec49-4bd3-40dd-918f-44d3254f2ac9 --os-tenant-name service --os-username admin Account: AUTH_cce6e9c12fa34c63b64ef29e84861554 Container: glance Object: fc34ec49-4bd3-40dd-918f-44d3254f2ac9 Content Type: application/octet-stream Content Length: 509804544 <---- see, the size is different! Last Modified: Mon, 19 Jan 2015 15:52:18 GMT ETag: "6d0612f82db9a531b34d25823a45073d" Manifest: glance/fc34ec49-4bd3-40dd-918f-44d3254f2ac9- Accept-Ranges: bytes X-Timestamp: 1421682737.01148 X-Trans-Id: tx01a19f7476a541808c9a1-0054bd28e1 .... glance image-download tst --file out.qcow2 [Errno 32] Corrupt image download. Checksum was 0eeddae1007f01b0029136d28518f538 expected 3ecddfe0787a392960d230c87a421c6a To manage notifications about this bug go to: https://bugs.launchpad.net/glance/+bug/1412802/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
