[
https://issues.apache.org/jira/browse/HDDS-16363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prince Raj updated HDDS-16363:
------------------------------
Description:
h2. Summary
Storage distribution QE test multipart_upload_openkeybytes validates that Recon
reports multipartOpenKeyBytes correctly for an {*}incomplete S3 multipart
upload (MPU){*}. After uploading *one 16MB part* and leaving the MPU open, the
test expects Recon to report baseline + 48MB (16MB logical × RATIS RF=3). On
upstream runs, multipartOpenKeyBytes {*}never reaches 50331648 bytes{*};
polling exhausts retries and actual *≠* expected.
h2. What this testcase does
h3. Purpose
Verify Recon *Capacity / storageDistribution* API correctly tracks bytes held
by {*}open (incomplete) multipart uploads{*}, and that those bytes are cleared
after MPU container cleanup.
h3. Scenario configuration
{{StorageScenario(
test_id="multipart_upload_openkeybytes",
num_keys=1,
key_sizes=['16MB'], # not used for MPU upload path
multipart_upload=True,
mpu_open_key_bytes=True, # leave MPU incomplete
mpu_part_sizes=['16MB', '16MB'], # 2 parts defined
mpu_parts_to_upload=1, # only part 1 uploaded → 16MB raw
keys_to_delete=None, # no normal delete workflow
)}}
h3. Data created
* *1 incomplete MPU* on volume s3v, key name mpu-open-key-test
* *Uploaded:* 1 part × *16MB* = *16,777,216 bytes* (part 2 never uploaded, MPU
never completed)
* *Expected Recon metric (replicated):* 16MB × *RF 3* = *50,331,648 bytes*
h3. Actual test flow
||Step||Action||
|1|Read multipartOpenKeyBytes *BEFORE* MPU (GET /api/v1/storageDistribution) —
baseline|
|2|Capture OM/SCM/DN Recon metrics baseline|
|3|Create incomplete MPU via S3 API (1 × 16MB part, no complete)|
|4|*Poll* until multipartOpenKeyBytes == baseline + 50331648 (up to 100 × 5s) ←
*FAILS here on upstream*|
|5|Resolve MPU container IDs from OM multipartInfoTable|
|6|closeContainer() on those containers|
|7|Poll until multipartOpenKeyBytes == 0|
|8|Verify SCM/DN pending-deletion metrics reflect cleanup|
h3.
How expected value is computed
{{uploaded_bytes = 16 * 1024 * 1024 # from created_keys[0]['size']
expected_open_delta = Ozone.calculate_total_repl_size(
sizes=[16777216],
replication_type="RATIS", # default RF=3
return_bytes=True,
) # → 50331648
expected_multipart_open = multipart_open_before + 50331648}}
h3. How actual value is read
{{GET Recon storageDistribution
→ usedSpaceBreakdown.openKeyBytes.multipartOpenKeyBytes}}
(Recon backend uses totalReplicatedDataSize from MPU open-key summary — see
StorageDistributionEndpoint.calculateOpenKeySizes().)
----
h2. Steps to reproduce
# Run storage distribution E2E with scenario multipart_upload_openkeybytes:
{{pytest
tests/ozone/ozone_18/storage_capacity_distribution/test_delete_workflow.py \
-k multipart_upload_openkeybytes}}
# Confirm S3 Gateway is up and kinit / S3 creds are configured.
3. Observe logs from poll_multipart_open_key_bytes_until_match:
{{2026-08-23 07:56:36,294|INFO|MainThread|ozone_storage_utils.py:704 -
poll_multipart_open_key_bytes_until_match()|multipartOpenKeyBytes attempt 22:
actual=0 expected=50331648}}
----
h2. Observed behavior (upstream)
* Incomplete MPU is created successfully (S3 upload part succeeds).
* multipartOpenKeyBytes *does not increase to expected 48MB replicated size*
(50331648).
* Poll loop runs full retries (max_retries=100, retry_interval=5s ≈ {*}8+
minutes{*}) without match.
* Example log pattern:
{{2026-08-23 07:56:36,294|INFO|MainThread|ozone_storage_utils.py:704 -
poll_multipart_open_key_bytes_until_match()|multipartOpenKeyBytes attempt 22:
actual=0 expected=50331648}}
----
h2. Expected behavior
After uploading *one 16MB MPU part* (incomplete):
* Recon multipartOpenKeyBytes should become baseline + 50,331,648 bytes (16MB
× RF 3).
* Test assertion at step 4 should pass within poll window.
----
h3. Root Cause:
This Change was causing the failure :
<https://github.com/apache/ozone/pull/10588>
was:
h2. Summary
Storage distribution QE test multipart_upload_openkeybytes validates that Recon
reports multipartOpenKeyBytes correctly for an {*}incomplete S3 multipart
upload (MPU){*}. After uploading *one 16MB part* and leaving the MPU open, the
test expects Recon to report baseline + 48MB (16MB logical × RATIS RF=3). On
upstream runs, multipartOpenKeyBytes \{*}never reaches 50331648 bytes{*};
polling exhausts retries and actual *≠* expected.
h2. What this testcase does
h3. Purpose
Verify Recon *Capacity / storageDistribution* API correctly tracks bytes held
by {*}open (incomplete) multipart uploads{*}, and that those bytes are cleared
after MPU container cleanup.
h3. Scenario configuration
{{StorageScenario(
test_id="multipart_upload_openkeybytes",
num_keys=1,
key_sizes=['16MB'], # not used for MPU upload path
multipart_upload=True,
mpu_open_key_bytes=True, # leave MPU incomplete
mpu_part_sizes=['16MB', '16MB'], # 2 parts defined
mpu_parts_to_upload=1, # only part 1 uploaded → 16MB raw
keys_to_delete=None, # no normal delete workflow
)}}
h3. Data created
* *1 incomplete MPU* on volume s3v, key name mpu-open-key-test
* *Uploaded:* 1 part × *16MB* = *16,777,216 bytes* (part 2 never uploaded, MPU
never completed)
* *Expected Recon metric (replicated):* 16MB × *RF 3* = *50,331,648 bytes*
h3. Actual test flow
||Step||Action||
|1|Read multipartOpenKeyBytes *BEFORE* MPU (GET /api/v1/storageDistribution) —
baseline|
|2|Capture OM/SCM/DN Recon metrics baseline|
|3|Create incomplete MPU via S3 API (1 × 16MB part, no complete)|
|4|*Poll* until multipartOpenKeyBytes == baseline + 50331648 (up to 100 × 5s) ←
*FAILS here on upstream*|
|5|Resolve MPU container IDs from OM multipartInfoTable|
|6|closeContainer() on those containers|
|7|Poll until multipartOpenKeyBytes == 0|
|8|Verify SCM/DN pending-deletion metrics reflect cleanup|
h3. How expected value is computed
{{uploaded_bytes = 16 * 1024 * 1024 # from created_keys[0]['size']
expected_open_delta = Ozone.calculate_total_repl_size(
sizes=[16777216],
replication_type="RATIS", # default RF=3
return_bytes=True,
) # → 50331648
expected_multipart_open = multipart_open_before + 50331648}}
h3. How actual value is read
{{GET Recon storageDistribution
→ usedSpaceBreakdown.openKeyBytes.multipartOpenKeyBytes}}
(Recon backend uses totalReplicatedDataSize from MPU open-key summary — see
StorageDistributionEndpoint.calculateOpenKeySizes().)
----
h2. Steps to reproduce
# Run storage distribution E2E with scenario multipart_upload_openkeybytes:
{{pytest
tests/ozone/ozone_18/storage_capacity_distribution/test_delete_workflow.py \
-k multipart_upload_openkeybytes}}
# Confirm S3 Gateway is up and kinit / S3 creds are configured.
# Observe logs from poll_multipart_open_key_bytes_until_match:
{{2026-08-23 07:56:36,294|INFO|MainThread|ozone_storage_utils.py:704 -
poll_multipart_open_key_bytes_until_match()|multipartOpenKeyBytes attempt 22:
actual=0 expected=50331648}}
----
h2. Observed behavior (upstream)
* Incomplete MPU is created successfully (S3 upload part succeeds).
* multipartOpenKeyBytes *does not increase to expected 48MB replicated size*
(50331648).
* Poll loop runs full retries (max_retries=100, retry_interval=5s ≈ {*}8+
minutes{*}) without match.
* Example log pattern:
{{2026-08-23 07:56:36,294|INFO|MainThread|ozone_storage_utils.py:704 -
poll_multipart_open_key_bytes_until_match()|multipartOpenKeyBytes attempt 22:
actual=0 expected=50331648}}
----
h2. Expected behavior
After uploading *one 16MB MPU part* (incomplete):
* Recon multipartOpenKeyBytes should become baseline + 50,331,648 bytes (16MB
× RF 3).
* Test assertion at step 4 should pass within poll window.
----
h2. Possible root causes (for investigation)
# *Recon MPU open-key summary not updated* for incomplete S3 MPU on upstream
(getMPUKeySummary() / totalReplicatedDataSize stays 0).
# *Recon sync lag* beyond test poll window (less likely if 500s exhausted).
> Storage-Capacity-Distribution: multipart_upload_openkeybytes fails on
> upstream — Recon metric does not reach expected 48MB
> --------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16363
> URL: https://issues.apache.org/jira/browse/HDDS-16363
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Prince Raj
> Assignee: Priyesh Karatha
> Priority: Major
>
> h2. Summary
> Storage distribution QE test multipart_upload_openkeybytes validates that
> Recon reports multipartOpenKeyBytes correctly for an {*}incomplete S3
> multipart upload (MPU){*}. After uploading *one 16MB part* and leaving the
> MPU open, the test expects Recon to report baseline + 48MB (16MB logical ×
> RATIS RF=3). On upstream runs, multipartOpenKeyBytes {*}never reaches
> 50331648 bytes{*}; polling exhausts retries and actual *≠* expected.
> h2. What this testcase does
> h3. Purpose
> Verify Recon *Capacity / storageDistribution* API correctly tracks bytes held
> by {*}open (incomplete) multipart uploads{*}, and that those bytes are
> cleared after MPU container cleanup.
> h3. Scenario configuration
>
> {{StorageScenario(
> test_id="multipart_upload_openkeybytes",
> num_keys=1,
> key_sizes=['16MB'], # not used for MPU upload path
> multipart_upload=True,
> mpu_open_key_bytes=True, # leave MPU incomplete
> mpu_part_sizes=['16MB', '16MB'], # 2 parts defined
> mpu_parts_to_upload=1, # only part 1 uploaded → 16MB raw
> keys_to_delete=None, # no normal delete workflow
> )}}
>
>
> h3. Data created
> * *1 incomplete MPU* on volume s3v, key name mpu-open-key-test
> * *Uploaded:* 1 part × *16MB* = *16,777,216 bytes* (part 2 never uploaded,
> MPU never completed)
> * *Expected Recon metric (replicated):* 16MB × *RF 3* = *50,331,648 bytes*
> h3. Actual test flow
>
> ||Step||Action||
> |1|Read multipartOpenKeyBytes *BEFORE* MPU (GET /api/v1/storageDistribution)
> — baseline|
> |2|Capture OM/SCM/DN Recon metrics baseline|
> |3|Create incomplete MPU via S3 API (1 × 16MB part, no complete)|
> |4|*Poll* until multipartOpenKeyBytes == baseline + 50331648 (up to 100 × 5s)
> ← *FAILS here on upstream*|
> |5|Resolve MPU container IDs from OM multipartInfoTable|
> |6|closeContainer() on those containers|
> |7|Poll until multipartOpenKeyBytes == 0|
> |8|Verify SCM/DN pending-deletion metrics reflect cleanup|
> h3.
> How expected value is computed
> {{uploaded_bytes = 16 * 1024 * 1024 # from created_keys[0]['size']
> expected_open_delta = Ozone.calculate_total_repl_size(
> sizes=[16777216],
> replication_type="RATIS", # default RF=3
> return_bytes=True,
> ) # → 50331648
> expected_multipart_open = multipart_open_before + 50331648}}
>
> h3. How actual value is read
>
> {{GET Recon storageDistribution
> → usedSpaceBreakdown.openKeyBytes.multipartOpenKeyBytes}}
>
>
> (Recon backend uses totalReplicatedDataSize from MPU open-key summary — see
> StorageDistributionEndpoint.calculateOpenKeySizes().)
> ----
> h2. Steps to reproduce
> # Run storage distribution E2E with scenario multipart_upload_openkeybytes:
> {{pytest
> tests/ozone/ozone_18/storage_capacity_distribution/test_delete_workflow.py \
> -k multipart_upload_openkeybytes}}
> # Confirm S3 Gateway is up and kinit / S3 creds are configured.
> 3. Observe logs from poll_multipart_open_key_bytes_until_match:
>
>
>
>
> {{2026-08-23 07:56:36,294|INFO|MainThread|ozone_storage_utils.py:704 -
> poll_multipart_open_key_bytes_until_match()|multipartOpenKeyBytes attempt 22:
> actual=0 expected=50331648}}
>
>
> ----
> h2. Observed behavior (upstream)
> * Incomplete MPU is created successfully (S3 upload part succeeds).
> * multipartOpenKeyBytes *does not increase to expected 48MB replicated size*
> (50331648).
> * Poll loop runs full retries (max_retries=100, retry_interval=5s ≈ {*}8+
> minutes{*}) without match.
> * Example log pattern:
>
>
>
>
> {{2026-08-23 07:56:36,294|INFO|MainThread|ozone_storage_utils.py:704 -
> poll_multipart_open_key_bytes_until_match()|multipartOpenKeyBytes attempt 22:
> actual=0 expected=50331648}}
>
>
> ----
> h2. Expected behavior
> After uploading *one 16MB MPU part* (incomplete):
> * Recon multipartOpenKeyBytes should become baseline + 50,331,648 bytes
> (16MB × RF 3).
> * Test assertion at step 4 should pass within poll window.
> ----
> h3. Root Cause:
> This Change was causing the failure :
> <https://github.com/apache/ozone/pull/10588>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
