Priyesh Karatha created HDDS-17158:
--------------------------------------
Summary: Datanode under-reports pending deletion bytes for EC
blocks
Key: HDDS-17158
URL: https://issues.apache.org/jira/browse/HDDS-17158
Project: Apache Ozone
Issue Type: Improvement
Components: Ozone Datanode
Reporter: Priyesh Karatha
h3. Problem
For EC keys, the DN metric \{{TotalPendingBlockBytes}} (BlockDeletingService)
under-reports pending deletion size. As a result, the DN value Recon shows on
the storage distribution / pending deletion page
(\{{pendingDeletion?component=dn}}) does not match SCM. The mismatch only shows
up some of the time.
Observed in a test that deleted 3 EC keys (expected 171,966,464 bytes):
* SCM deletion metrics per cycle: [113246208, 58720256] -> matches expected
total
* DN pending deletion per cycle: [113246205, 5872025] -> second cycle is ~10x
too low (~52.8 MB missing)
The first cycle is correct; the -3 bytes come from integer division of the
per-replica size across 5 DNs.
h3. Root cause
HDDS-15076 changed the DN to add \{{totalSizePerReplica}} to a container's
pending deletion bytes when a delete transaction arrives
(\{{DeleteBlocksCommandHandler#updateMetaData}}). The release path was not
updated: when the transaction is processed,
\{{BlockDeletingTask#deleteTransactions}} still subtracts \{{totalBlockSize}}:
{code:java}
bytesProcessed += entry.getTotalBlockSize();
{code}
For RATIS, \{{totalBlockSize == totalSizePerReplica}}, so the bug does not
show. For EC, \{{totalBlockSize}} is the logical block size, about \{{data}}
times the per-replica size (e.g. ~3x for rs-3-2). So each processed EC
transaction releases far more than it added.
\{{ContainerData.BlockByteAndCounts#decDeletion}} then clamps the result at 0.
h3. Why it is intermittent
If all pending transactions of a container are processed in the same run, the
clamp to 0 hides the error. The under-count shows only when a newer transaction
is already pending in the same container while an older one is processed in a
separate run. That can happen when:
* the block deletion limit per interval is reached
* the max lock holding time cutoff is hit
* the next deletion cycle's transaction arrives in between
The extra amount subtracted then removes the newer transaction's pending bytes.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
