[
https://issues.apache.org/jira/browse/HDDS-16363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prince Raj reassigned HDDS-16363:
---------------------------------
Assignee: Priyesh Karatha
> Storage-Capacity-Distribution: multipart_upload_openkeybytes fails on
> upstream — Recon metric does not reach expected 48MB
> --------------------------------------------------------------------------------------------------------------------------
>
> Key: HDDS-16363
> URL: https://issues.apache.org/jira/browse/HDDS-16363
> Project: Apache Ozone
> Issue Type: Bug
> Reporter: Prince Raj
> Assignee: Priyesh Karatha
> Priority: Major
>
> h2. Summary
> Storage distribution QE test multipart_upload_openkeybytes validates that
> Recon reports multipartOpenKeyBytes correctly for an {*}incomplete S3
> multipart upload (MPU){*}. After uploading *one 16MB part* and leaving the
> MPU open, the test expects Recon to report baseline + 48MB (16MB logical ×
> RATIS RF=3). On upstream runs, multipartOpenKeyBytes \{*}never reaches
> 50331648 bytes{*}; polling exhausts retries and actual *≠* expected.
> h2. What this testcase does
> h3. Purpose
> Verify Recon *Capacity / storageDistribution* API correctly tracks bytes held
> by {*}open (incomplete) multipart uploads{*}, and that those bytes are
> cleared after MPU container cleanup.
> h3. Scenario configuration
>
> {{StorageScenario(
> test_id="multipart_upload_openkeybytes",
> num_keys=1,
> key_sizes=['16MB'], # not used for MPU upload path
> multipart_upload=True,
> mpu_open_key_bytes=True, # leave MPU incomplete
> mpu_part_sizes=['16MB', '16MB'], # 2 parts defined
> mpu_parts_to_upload=1, # only part 1 uploaded → 16MB raw
> keys_to_delete=None, # no normal delete workflow
> )}}
>
>
> h3. Data created
> * *1 incomplete MPU* on volume s3v, key name mpu-open-key-test
> * *Uploaded:* 1 part × *16MB* = *16,777,216 bytes* (part 2 never uploaded,
> MPU never completed)
> * *Expected Recon metric (replicated):* 16MB × *RF 3* = *50,331,648 bytes*
> h3. Actual test flow
>
>
> ||Step||Action||
> |1|Read multipartOpenKeyBytes *BEFORE* MPU (GET /api/v1/storageDistribution)
> — baseline|
> |2|Capture OM/SCM/DN Recon metrics baseline|
> |3|Create incomplete MPU via S3 API (1 × 16MB part, no complete)|
> |4|*Poll* until multipartOpenKeyBytes == baseline + 50331648 (up to 100 × 5s)
> ← *FAILS here on upstream*|
> |5|Resolve MPU container IDs from OM multipartInfoTable|
> |6|closeContainer() on those containers|
> |7|Poll until multipartOpenKeyBytes == 0|
> |8|Verify SCM/DN pending-deletion metrics reflect cleanup|
>
>
>
> h3. How expected value is computed
>
>
>
>
> {{uploaded_bytes = 16 * 1024 * 1024 # from created_keys[0]['size']
> expected_open_delta = Ozone.calculate_total_repl_size(
> sizes=[16777216],
> replication_type="RATIS", # default RF=3
> return_bytes=True,
> ) # → 50331648
> expected_multipart_open = multipart_open_before + 50331648}}
>
>
> h3. How actual value is read
>
>
>
>
> {{GET Recon storageDistribution
> → usedSpaceBreakdown.openKeyBytes.multipartOpenKeyBytes}}
>
>
> (Recon backend uses totalReplicatedDataSize from MPU open-key summary — see
> StorageDistributionEndpoint.calculateOpenKeySizes().)
> ----
> h2. Steps to reproduce
> # Run storage distribution E2E with scenario multipart_upload_openkeybytes:
>
>
>
>
> {{pytest
> tests/ozone/ozone_18/storage_capacity_distribution/test_delete_workflow.py \
> -k multipart_upload_openkeybytes}}
>
>
> # Confirm S3 Gateway is up and kinit / S3 creds are configured.
> # Observe logs from poll_multipart_open_key_bytes_until_match:
>
>
>
>
> {{2026-08-23 07:56:36,294|INFO|MainThread|ozone_storage_utils.py:704 -
> poll_multipart_open_key_bytes_until_match()|multipartOpenKeyBytes attempt 22:
> actual=0 expected=50331648}}
>
>
> ----
> h2. Observed behavior (upstream)
> * Incomplete MPU is created successfully (S3 upload part succeeds).
> * multipartOpenKeyBytes *does not increase to expected 48MB replicated size*
> (50331648).
> * Poll loop runs full retries (max_retries=100, retry_interval=5s ≈ {*}8+
> minutes{*}) without match.
> * Example log pattern:
>
>
>
>
> {{2026-08-23 07:56:36,294|INFO|MainThread|ozone_storage_utils.py:704 -
> poll_multipart_open_key_bytes_until_match()|multipartOpenKeyBytes attempt 22:
> actual=0 expected=50331648}}
>
>
> ----
> h2. Expected behavior
> After uploading *one 16MB MPU part* (incomplete):
> * Recon multipartOpenKeyBytes should become baseline + 50,331,648 bytes
> (16MB × RF 3).
> * Test assertion at step 4 should pass within poll window.
> ----
> h2. Possible root causes (for investigation)
> # *Recon MPU open-key summary not updated* for incomplete S3 MPU on upstream
> (getMPUKeySummary() / totalReplicatedDataSize stays 0).
> # *Recon sync lag* beyond test poll window (less likely if 500s exhausted).
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
