[
https://issues.apache.org/jira/browse/HDDS-16731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chu Cheng Li reassigned HDDS-16731:
-----------------------------------
Assignee: Chu Cheng Li
> [hsync] Container scanner and reconciliation ignore last_chunk_info rows
> ------------------------------------------------------------------------
>
> Key: HDDS-16731
> URL: https://issues.apache.org/jira/browse/HDDS-16731
> Project: Apache Ozone
> Issue Type: Sub-task
> Reporter: Chu Cheng Li
> Assignee: Chu Cheng Li
> Priority: Major
> Labels: pull-request-available
>
> h3. Background
> A client with {{ozone.client.incremental.chunk.list=true}} sends an
> incremental chunk list in PutBlock. The datanode keeps the last partial chunk
> of the block in the {{last_chunk_info}} table. The other chunks are in the
> {{block_data}} table. {{getBlock}} reads both tables and merges the chunk
> lists.
> The datanode moves the last chunk to {{block_data}} when the chunk is full,
> at the end of the block, or when FinalizeBlock merges it. Thus, a closed
> container can keep {{last_chunk_info}} rows:
> * The client does not end the block, for example because the client fails.
> Then the container closes before lease recovery. FinalizeBlock fails on a
> closed container. Thus, no merge occurs.
> * A datanode with release 2.0.0 to 2.2.1 finalized the block. These releases
> do not persist the merge (HDDS-16730).
> h3. Problem
> Some code for closed containers reads only the {{block_data}} table:
> * The container scanner does not verify the checksum of the last chunk.
> * The container merkle tree does not include the last chunk. Thus,
> reconciliation cannot find or repair a bad last chunk.
> * When reconciliation repairs a chunk of the block, it writes the full chunk
> list to {{block_data}}. It does not delete the {{last_chunk_info}} row. Thus,
> the next {{getBlock}} adds the last chunk a second time and fails with
> {{IllegalStateException: chunk offset does not match}}.
> h3. Effect
> * The datanode does not find corruption in the last chunk of these blocks.
> * After reconciliation repairs such a block, the block is unreadable on that
> datanode. Clients must read from other datanodes. Other datanodes also cannot
> reconcile the block from this datanode.
> h3. Possible fix
> * Merge the {{last_chunk_info}} rows into {{block_data}} when the datanode
> closes or quasi-closes a container.
> * For containers that are already closed, make the scanner and the merkle
> tree use the merged block data. Make {{putBlockForClosedContainer}} delete
> the {{last_chunk_info}} row.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
