Chu Cheng Li created HDDS-16730:
-----------------------------------
Summary: [hsync] FinalizeBlock does not persist the last chunk
merge
Key: HDDS-16730
URL: https://issues.apache.org/jira/browse/HDDS-16730
Project: Apache Ozone
Issue Type: Sub-task
Reporter: Chu Cheng Li
Assignee: Chu Cheng Li
h3. Problem
HDDS-11014 adds a merge step to {{BlockManagerImpl#finalizeBlock}}. For a block
with an incremental chunk list, this step moves the last chunk from the
{{last_chunk_info}} table to the {{block_data}} table.
The code commits the batch before it adds the merge operations to the batch:
{code:java}
db.getStore().getFinalizeBlocksTable().putWithBatch(batch, blockKey, localID);
db.getStore().getBatchHandler().commitBatchOperation(batch);
mergeLastChunkForBlockFinalization(blockId, db, kvContainer, batch, localID);
{code}
No code commits the batch again. {{RDBBatchOperation#close}} discards the merge
operations. Thus, the {{last_chunk_info}} row stays, and the {{block_data}} row
does not get the last chunk.
The datanode logs these warnings for each finalized block that has an
incremental chunk list:
{noformat}
WARN db.RDBBatchOperation: discarding changes Batch-2: block_data
batchSize=94 B, discarded: 0 (0 B), #put=1, #del=0
WARN db.RDBBatchOperation: discarding changes Batch-2: last_chunk_info
batchSize=1 B, discarded: 0 (0 B), #put=0, #del=1
{noformat}
All releases since 2.0.0 have this problem.
h3. Effect
* Only a client with {{ozone.client.incremental.chunk.list=true}} writes an
incremental chunk list. The default is {{false}}.
* Reads return the full block. {{getBlock}} merges the {{last_chunk_info}} row
at read time.
* Some code reads the {{block_data}} rows directly. This code does not see the
last chunk. Thus, the container scanner does not verify the checksum of the
last chunk, and the container merkle tree does not include it.
* Container reconciliation can make the block unreadable on a datanode. When
reconciliation repairs a chunk of the block, it writes the full chunk list to
{{block_data}}. It does not delete the {{last_chunk_info}} row. Thus, the next
read adds the last chunk a second time and fails with {{IllegalStateException:
chunk offset does not match}}.
h3. Fix
Add the merge operations to the batch before the commit. Thus, RocksDB gets the
finalize marker and the merge in one write.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
