Gargi-jais11 opened a new pull request, #11856: URL: https://github.com/apache/ozone/pull/11856
## What changes were proposed in this pull request? S3 Gateway accepts a **CompleteMultipartUpload** request that lists the same part number more than once and returns success, which can create an object that does not match the request. `ObjectEndpoint.completeMultipartUpload` copies the part list into a map keyed by part number before calling OM. A repeated part number overwrites the earlier entry, so OM never sees the duplicate: - `[1, 1]` becomes `[1]`: the request succeeds with a 5 MiB object instead of failing, and part 2 is discarded. - `[1, 2, 1]` becomes `[1, 2]`: the request succeeds even though the list is invalid. - `[1 (wrong ETag), 1 (right ETag)]` becomes `[1 (right ETag)]`: the wrong ETag is never checked. OM already rejects part lists that are not in ascending order with `INVALID_PART_ORDER`, but S3 Gateway removes the duplicates first. AWS S3 rejects such requests with `400 InvalidPartOrder` ([CompleteMultipartUpload](https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html)). This change makes S3 Gateway return `InvalidPartOrder` when a part number repeats in the request. The multipart upload stays open, so the client can retry with a correct part list. ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-17554 ## How was this patch tested? Added unit and integration tests. also tested manually. **Before fix-:** ``` bash-5.1$ new_mpu() { ash-5.1$ new_mpu() {ate +%s)" KEY="mpu-dup-$1-$(date +%s)"multipart-upload --bucket "$BUCKET" --key "$KEY" --query UploadId --output text) UPLOAD_ID=$($A s3api create-multipart-upload --bucket "$BUCKET" --key "$KEY" --query UploadId --output text) ETAG1=$($A s3api upload-part --bucket "$BUCKET" --key "$KEY" --upload-id "$UPLOAD_ID" \ --part-number 1 --body part1.bin --query ETag --output text)-upload-id "$UPLOAD_ID" \ ETAG2=$($A s3api upload-part --bucket "$BUCKET" --key "$KEY" --upload-id "$UPLOAD_ID" \ --part-number 2 --body part2.bin --query ETag --output text) echo "KEY=$KEY UPLOAD_ID=$UPLOAD_ID ETAG1=$ETAG1 ETAG2=$ETAG2"d "$UPLOAD_ID" --query 'Parts[].[PartNumber,Size]' $A s3api list-parts --bucket "$BUCKET" --key "$KEY" --upload-id "$UPLOAD_ID" --query 'Parts[].[PartNumber,Size]' }omplete_mpu() { complete_mpu() {ete-multipart-upload --bucket "$BUCKET" --key "$KEY" --upload-id "$UPLOAD_ID" \ $A s3api complete-multipart-upload --bucket "$BUCKET" --key "$KEY" --upload-id "$UPLOAD_ID" \ --multipart-upload "{\"Parts\":$1}"T" --key "$KEY" --query '{ContentLength:ContentLength,ETag:ETag,PartsCount:PartsCount}' $A s3api head-object --bucket "$BUCKET" --key "$KEY" --query '{ContentLength:ContentLength,ETag:ETag,PartsCount:PartsCount}' } # Case 1 (the report): [1,1]. Expected InvalidPartOrder; actually succeeds with 5242880 bytes and PartsCount 2. bash-5.1$ new_mpu dup11 KEY=mpu-dup-dup11-1793254086 UPLOAD_ID=01a18645-d6d2-79f2-b9af-c5f6b05e75c0-117522699803361280 ETAG1="37fc190d719da7dc45591ee32529e8e9" ETAG2="4ee14b29dc6ffaab761f9664d8f549a9" [ [ 1, 5242880 ], [ 2, 5242880 ] ] bash-5.1$ complete_mpu "[{\"ETag\":$ETAG1,\"PartNumber\":1},{\"ETag\":$ETAG1,\"PartNumber\":1}]" $A s3api get-object --bucket "$BUCKET" --key "$KEY" /tmp/downloaded.bin >/dev/null && wc -c /tmp/downloaded.bin ozone sh key info "/s3v/$BUCKET/$KEY" | grep -E '"dataSize"|"ETag"' { "Location": "mpu-dup-repro", "Bucket": "mpu-dup-repro", "Key": "mpu-dup-dup11-1793254086", "ETag": "\"30baf772d9aba3bfed62c978b4a406f5-2\"" } { "ContentLength": 5242880, "ETag": "\"30baf772d9aba3bfed62c978b4a406f5-2\"", "PartsCount": 2 } 5242880 /tmp/downloaded.bin "dataSize" : 5242880, "ETag" : "30baf772d9aba3bfed62c978b4a406f5-2" # Case 2: [1,2,1]. Expected InvalidPartOrder; actually succeeds with 10 MiB. bash-5.1$ new_mpu dup121 complete_mpu "[{\"ETag\":$ETAG1,\"PartNumber\":1},{\"ETag\":$ETAG2,\"PartNumber\":2},{\"ETag\":$ETAG1,\"PartNumber\":1}]" KEY=mpu-dup-dup121-1793254109 UPLOAD_ID=01a18646-309a-7970-88a6-bf94b658bc9e-117522701309575171 ETAG1="37fc190d719da7dc45591ee32529e8e9" ETAG2="4ee14b29dc6ffaab761f9664d8f549a9" [ [ 1, 5242880 ], [ 2, 5242880 ] ] { "Location": "mpu-dup-repro", "Bucket": "mpu-dup-repro", "Key": "mpu-dup-dup121-1793254109", "ETag": "\"30baf772d9aba3bfed62c978b4a406f5-2\"" } { "ContentLength": 10485760, "ETag": "\"30baf772d9aba3bfed62c978b4a406f5-2\"", "PartsCount": 2 } # Case 3: a wrong ETag on the first copy of part 1 is ignored because the last entry wins. Actually succeeds. bash-5.1$ new_mpu wrongetag complete_mpu "[{\"ETag\":$ETAG2,\"PartNumber\":1},{\"ETag\":$ETAG1,\"PartNumber\":1}]" KEY=mpu-dup-wrongetag-1793254119 UPLOAD_ID=01a18646-586b-7056-98d2-5a3af7550ca7-117522701977583622 ETAG1="37fc190d719da7dc45591ee32529e8e9" ETAG2="4ee14b29dc6ffaab761f9664d8f549a9" [ [ 1, 5242880 ], [ 2, 5242880 ] ] { "Location": "mpu-dup-repro", "Bucket": "mpu-dup-repro", "Key": "mpu-dup-wrongetag-1793254119", "ETag": "\"30baf772d9aba3bfed62c978b4a406f5-2\"" } { "ContentLength": 5242880, "ETag": "\"30baf772d9aba3bfed62c978b4a406f5-2\"", "PartsCount": 2 } # Control: [2,1] is correctly rejected with InvalidPartOrder, which shows OM's order check works. bash-5.1$ new_mpu order21 complete_mpu "[{\"ETag\":$ETAG2,\"PartNumber\":2},{\"ETag\":$ETAG1,\"PartNumber\":1}]" KEY=mpu-dup-order21-1793254127 UPLOAD_ID=01a18646-79e0-79b3-a3ce-c4b7a4959475-117522702538899465 ETAG1="37fc190d719da7dc45591ee32529e8e9" ETAG2="4ee14b29dc6ffaab761f9664d8f549a9" [ [ 1, 5242880 ], [ 2, 5242880 ] ] An error occurred (InvalidPartOrder) when calling the CompleteMultipartUpload operation: The list of parts was not in ascending order. The parts list must be specified in order by part number. An error occurred (404) when calling the HeadObject operation: Not Found # Case 4 (Bug 2 alone): the valid request [1] succeeds, but the ETag ends in -2 and PartsCount is 2 instead of 1. bash-5.1$ new_mpu only1 complete_mpu "[{\"ETag\":$ETAG1,\"PartNumber\":1}]" KEY=mpu-dup-only1-1793254141 UPLOAD_ID=01a18646-aecb-7356-ba0a-9454cf88d00c-117522703426715660 ETAG1="37fc190d719da7dc45591ee32529e8e9" ETAG2="4ee14b29dc6ffaab761f9664d8f549a9" [ [ 1, 5242880 ], [ 2, 5242880 ] ] { "Location": "mpu-dup-repro", "Bucket": "mpu-dup-repro", "Key": "mpu-dup-only1-1793254141", "ETag": "\"30baf772d9aba3bfed62c978b4a406f5-2\"" } { "ContentLength": 5242880, "ETag": "\"30baf772d9aba3bfed62c978b4a406f5-2\"", "PartsCount": 2 } bash-5.1$ ``` After Fix: ``` bash-5.1$ new_mpu dup11 complete_mpu "[{\"ETag\":$ETAG1,\"PartNumber\":1},{\"ETag\":$ETAG1,\"PartNumber\":1}]" $A s3api get-object --bucket "$BUCKET" --key "$KEY" /tmp/downloaded.bin >/dev/null && wc -c /tmp/downloaded.bin ozone sh key info "/s3v/$BUCKET/$KEY" | grep -E '"dataSize"|"ETag"' KEY=mpu-dup-dup11-1793253529 UPLOAD_ID=01a1863d-58c2-7b11-bb75-d91bfdcb3f79-117522663328579584 ETAG1="d18c2fce808e435abc90552bd67342dd" ETAG2="05f6665701e7cbbf15c8115d7a6cdae3" [ [ 1, 5242880 ], [ 2, 5242880 ] ] An error occurred (InvalidPartOrder) when calling the CompleteMultipartUpload operation: The list of parts was not in ascending order. The parts list must be specified in order by part number. An error occurred (404) when calling the HeadObject operation: Not Found An error occurred (NoSuchKey) when calling the GetObject operation: The specified key does not exist KEY_NOT_FOUND Key:mpu-dup-dup11-1793253529 not found bash-5.1$ new_mpu dup121 complete_mpu "[{\"ETag\":$ETAG1,\"PartNumber\":1},{\"ETag\":$ETAG2,\"PartNumber\":2},{\"ETag\":$ETAG1,\"PartNumber\":1}]" KEY=mpu-dup-dup121-1793253558 UPLOAD_ID=01a1863d-ca7e-7f1b-b560-bf50d839c937-117522665236725763 ETAG1="d18c2fce808e435abc90552bd67342dd" ETAG2="05f6665701e7cbbf15c8115d7a6cdae3" [ [ 1, 5242880 ], [ 2, 5242880 ] ] An error occurred (InvalidPartOrder) when calling the CompleteMultipartUpload operation: The list of parts was not in ascending order. The parts list must be specified in order by part number. An error occurred (404) when calling the HeadObject operation: Not Found bash-5.1$ new_mpu wrongetag complete_mpu "[{\"ETag\":$ETAG2,\"PartNumber\":1},{\"ETag\":$ETAG1,\"PartNumber\":1}]" KEY=mpu-dup-wrongetag-1793253578 UPLOAD_ID=01a1863e-1847-71df-812e-2349ff539f79-117522666541744134 ETAG1="d18c2fce808e435abc90552bd67342dd" ETAG2="05f6665701e7cbbf15c8115d7a6cdae3" [ [ 1, 5242880 ], [ 2, 5242880 ] ] An error occurred (InvalidPartOrder) when calling the CompleteMultipartUpload operation: The list of parts was not in ascending order. The parts list must be specified in order by part number. An error occurred (404) when calling the HeadObject operation: Not Found bash-5.1$ new_mpu order21 complete_mpu "[{\"ETag\":$ETAG2,\"PartNumber\":2},{\"ETag\":$ETAG1,\"PartNumber\":1}]" KEY=mpu-dup-order21-1793253593 UPLOAD_ID=01a1863e-50e1-727b-9934-cfc32ce6dbb5-117522667491360777 ETAG1="d18c2fce808e435abc90552bd67342dd" ETAG2="05f6665701e7cbbf15c8115d7a6cdae3" [ [ 1, 5242880 ], [ 2, 5242880 ] ] An error occurred (InvalidPartOrder) when calling the CompleteMultipartUpload operation: The list of parts was not in ascending order. The parts list must be specified in order by part number. An error occurred (404) when calling the HeadObject operation: Not Found bash-5.1$ new_mpu only1 complete_mpu "[{\"ETag\":$ETAG1,\"PartNumber\":1}]" KEY=mpu-dup-only1-1793253619 UPLOAD_ID=01a1863e-b7e8-7ace-9ca5-87916214fd16-117522669219872780 ETAG1="d18c2fce808e435abc90552bd67342dd" ETAG2="05f6665701e7cbbf15c8115d7a6cdae3" [ [ 1, 5242880 ], [ 2, 5242880 ] ] { "Location": "mpu-dup-repro", "Bucket": "mpu-dup-repro", "Key": "mpu-dup-only1-1793253619", "ETag": "\"c615133815fe76482ad413aa613ec0cd-2\"" } { "ContentLength": 5242880, "ETag": "\"c615133815fe76482ad413aa613ec0cd-2\"", "PartsCount": 2 } ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
