Hi Casey,
Thanks for the quick response. I have just confirmed that if I set the
PartSize to 500MB the file uploads correctly. I am hesitant to do this in
production but I think we are on the right track. Interestingly enough,
when I set the PartSize to 5242880 the file did not store correctly (it
retrieved with a different hash, and not even the same different hash I saw
before).
Here are my details of my client. Please forgive me for not seeing how to
attach this to the tracker item you mentioned in your email.
I am using the AWSSDK.S3 nuget package version 3.3.31.24, with ceph version
"ceph version 12.2.10-551-gbb089269ea
(bb089269ea0c1272294c6b9777123ac81662b6d2) luminous (stable)"
My relevant c# code is as follows:
using (var client = new AmazonS3Client(AccessKey, SecretKey,
GetS3Config()))
using (var transferUtility = new
Amazon.S3.Transfer.TransferUtility(client))
{
var transferRequest = new
Amazon.S3.Transfer.TransferUtilityUploadRequest
{
BucketName = bucketName,
Key = key,
InputStream = stream,
ServerSideEncryptionCustomerProvidedKey = EncryptionKey,
ServerSideEncryptionCustomerMethod =
string.IsNullOrWhiteSpace(EncryptionKey) ?
ServerSideEncryptionCustomerMethod.None :
ServerSideEncryptionCustomerMethod.AES256
};
transferUtility.Upload(transferRequest);
}
I am having hard time locating the default PartSize this library uses, but
I think it is 5MB as the content-length header is a bit more than that
size. Perhaps this sanitized header is useful:
PUT
[redacted]/delete-me?partNumber=18&uploadId=2~2dt4pYGY3vfKxBb9FcbVlAnbz_z3HTV
HTTP/1.1
Expect: 100-continue
x-amz-server-side-encryption-customer-algorithm: AES256
x-amz-server-side-encryption-customer-key: [redacted]
x-amz-server-side-encryption-customer-key-MD5: [redacted]
User-Agent: aws-sdk-dotnet-coreclr/3.3.31.24 aws-sdk-dotnet-core/3.3.32.2
.NET_Core/4.6.27317.07 OS/Microsoft_Windows_10.0.17763 ClientAsync
TransferManager/MultipartUploadCommand
Host: [redacted]
X-Amz-Date: [redacted]
X-Amz-Decoded-Content-Length: 5242880
X-Amz-Content-SHA256: STREAMING-AWS4-HMAC-SHA256-PAYLOAD
Authorization: [redacted]
Content-Length: 5248726
Content-Type: text/plain
14000;chunk-signature=[redacted]
[payload here]
Thank you again for help on this matter. I'll point our vendor towards this
bug as well.
Cheers,
Dan
On Mon, Mar 18, 2019 at 12:05 PM Casey Bodley <cbod...@redhat.com> wrote:
> Hi Dan,
>
> We just got a similar report about SSE-C in
> http://tracker.ceph.com/issues/38700 that seems to be related to
> multipart uploads. Could you please add some details there about your s3
> client, its multipart chunk size, and your ceph version?
>
> On 3/18/19 2:38 PM, Dan Smith wrote:
> > Hello,
> >
> > I have stored more than 167 million files in ceph using the S3 api.
> > Out of those 167 million+ files, one file is not storing correctly.
> >
> > The file is 92MB in size. I have stored files much larger and much
> > smaller. If I store the file WITHOUT using the Customer Provided
> > 256-bit AES key using Server Side encryption, the file stores and
> > retrieves just fine (SHA256 hashes match).
> >
> > If I store the file USING the 256-bit AES key using Server Side
> > encryption, the file stores without error, however, when I retrieve
> > the file and compare the hash of the file I retrieve from ceph against
> > the hash of the original file, the hashes differ.
> >
> > If I store the file using Amazon S3, using the same AES key and their
> > server side encryption the file stores are retrieves using out issue
> > (hashes match).
> >
> > I can reproduce this issue in two different ceph environments.
> > Thankfully, the file I am storing is not confidential, so I can share
> > it out to anyone interested in this
> > issue.(
> https://s3.amazonaws.com/aws-website-afewgoodmenrankedfantasyfootballcom-j5gvt/delete-me
> )
> >
> > I have opened a ticket with our vendor for support, but I am hoping
> > someone might be able to give me some ideas on what might be going on
> > as well.
> >
> > Cheers,
> > Dan
> >
> > _______________________________________________
> > ceph-users mailing list
> > ceph-users@lists.ceph.com
> > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
>
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
