tanvipenumudy opened a new pull request, #8695: URL: https://github.com/apache/ozone/pull/8695
## What changes were proposed in this pull request? The PR adds support in Recon to expose information regarding multipart upload (MPU) keys or files that are still in progress or uncommitted. - As of today, the `Recon Overview Page` - `Open Keys Summary Tile` shows the summary of open key / file entries (count / unreplicated size / replicated size) based on `OpenKeyTable` and `OpenFileTable`, but does not include data from `MultipartInfoTable`. - This change introduces a new endpoint, `GET` `/api/v1/keys/open/mpu/summary`, which exposes the MPU-related key / file information from the `MultipartInfoTable`. - The total size of open (uncommitted) keys and files can be derived by combining entries from the `OpenKeyTable`, `OpenFileTable`, and `MultipartInfoTable`. ## What is the link to the Apache JIRA https://issues.apache.org/jira/browse/HDDS-13187 ## How was this patch tested? - Added Unit Tests - Manually tested the endpoint on a local docker-compose deployment: Create a new OBS bucket with `RATIS` `THREE` replication: ``` bash-5.1$ ozone sh bucket create s3v/testbuck --layout=OBJECT_STORE --replication=THREE --type=RATIS bash-5.1$ ozone sh bucket info s3v/testbuck { "metadata" : { }, "volumeName" : "s3v", "name" : "testbuck", "storageType" : "DISK", "versioning" : false, "listCacheSize" : 1000, "usedBytes" : 0, "usedNamespace" : 0, "creationTime" : "2025-06-19T18:22:47.103Z", "modificationTime" : "2025-06-19T18:22:47.103Z", "sourcePathExist" : true, "quotaInBytes" : -1, "quotaInNamespace" : -1, "bucketLayout" : "OBJECT_STORE", "owner" : "hadoop", "link" : false, "replicationConfig" : { "replicationFactor" : "THREE", "requiredNodes" : 3, "minimumNodes" : 1, "replicationType" : "RATIS" } } ``` Create a 1 GB test file filled with zeros: ``` dd if=/dev/zero of=my_1GB_file bs=1M count=1024 1024+0 records in 1024+0 records out 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 2.26283 s, 475 MB/s ``` Split the test file into ~500 MB parts: `xaa`, `xab`, `xac`: ``` split -b 500MB my_1GB_file ``` Initiate MPU for key1: ``` aws s3api create-multipart-upload --bucket testbuck --key key1 --endpoint http://localhost:9878/ --no-verify-ssl { "Bucket": "testbuck", "Key": "key1", "UploadId": "0197896f-1d59-7019-8c93-d1868313176e-114711425847984128" } ``` Upload 3 parts (~500 MB each): ``` aws s3api upload-part --bucket testbuck --key key1 --endpoint http://localhost:9878/ --part-number 1 --body xaa --upload-id 0197896f-1d59-7019-8c93-d1868313176e-114711425847984128 --no-verify-ssl { "ETag": "d8b61b2c0025919d5321461045c8226f" } aws s3api upload-part --bucket testbuck --key key1 --endpoint http://localhost:9878/ --part-number 2 --body xab --upload-id 0197896f-1d59-7019-8c93-d1868313176e-114711425847984128 --no-verify-ssl { "ETag": "d8b61b2c0025919d5321461045c8226f" } aws s3api upload-part --bucket testbuck --key key1 --endpoint http://localhost:9878/ --part-number 3 --body xac --upload-id 0197896f-1d59-7019-8c93-d1868313176e-114711425847984128 --no-verify-ssl { "ETag": "77377273b0a4b61febdbf7bbf52b9db9" } ``` Response of `GET` `/api/v1/keys/open/mpu/summary`: ``` { "totalUnreplicatedDataSize": 1073741824, "totalOpenMPUKeys": 1, "totalReplicatedDataSize": 3221225472 } ``` Initiate MPU for key2: ``` aws s3api create-multipart-upload --bucket testbuck --key key2 --endpoint http://localhost:9878/ --no-verify-ssl { "Bucket": "testbuck", "Key": "key2", "UploadId": "01978971-9b42-7afb-96cf-a5faef242898-114711436550340612" } ``` Upload 2 parts (~500 MB each): ``` aws s3api upload-part --bucket testbuck --key key2 --endpoint http://localhost:9878/ --part-number 1 --body xaa --upload-id 01978971-9b42-7afb-96cf-a5faef242898-114711436550340612 --no-verify-ssl { "ETag": "d8b61b2c0025919d5321461045c8226f" } aws s3api upload-part --bucket testbuck --key key2 --endpoint http://localhost:9878/ --part-number 2 --body xab --upload-id 01978971-9b42-7afb-96cf-a5faef242898-114711436550340612 --no-verify-ssl { "ETag": "d8b61b2c0025919d5321461045c8226f" } ``` Response of `GET` `/api/v1/keys/open/mpu/summary`: ``` { "totalUnreplicatedDataSize": 2122317824, "totalOpenMPUKeys": 2, "totalReplicatedDataSize": 6366953472 } ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
