[
https://issues.apache.org/jira/browse/HDDS-16783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siyao Meng updated HDDS-16783:
------------------------------
Description:
h3. Problem
HBase stores its own checksums in HFile blocks. When
{{hbase.regionserver.checksum.verify}} is true (the default), {{HFileSystem}}
opens a second FileSystem instance, calls {{setVerifyChecksum(false)}} on it,
and reads HFile blocks through that instance. On HDFS this skips the file
system checksum.
{{BasicOzoneFileSystem}} and {{BasicRootedOzoneFileSystem}} do not override
{{setVerifyChecksum}}, so the call reaches the empty default in {{FileSystem}}
and the Ozone client still verifies every read. This costs HBase twice:
* the data is checksummed by both Ozone and HBase
* with verification on, each read is widened to checksum boundaries
({{ozone.client.bytes.per.checksum}}), so with a 1MB value a 64KB HFile block
read fetches and checksums 1MB
{{ozone.client.verify.checksum=false}} is not a good substitute. It applies to
the whole process, including the FileSystem instance HBase falls back to when
an HFile checksum fails, so that retry would no longer be verified by anything.
h3. Proposal
Override {{setVerifyChecksum}} in both file systems and pass the flag through
{{OzoneClientAdapter}} and {{ClientProtocol}} to the client's own
{{OzoneClientConfig}}, so that input streams opened afterwards by that
FileSystem instance skip checksum verification. The setting is per instance,
the same as {{DistributedFileSystem}}.
No protocol change is needed. The existing read paths already read this flag
when a stream is opened: ReadChunk, short-circuit read, and streaming
ReadBlock, which passes it to the datanode as {{includeChecksums}} (HDDS-16258).
h3. Expected gain
64KB random positional reads ({{readFully(pos, buf, 0, len)}} on one shared
input stream per file), 16 threads, operations per second, median of three
rounds. "Verify off" was measured with {{ozone.client.verify.checksum=false}},
which is what the API call would toggle per instance.
||Read path||Verify on||Verify off||
|gRPC, 1MB bytes per checksum|6,100|22,600|
|gRPC, 16KB bytes per checksum|22,100|22,600|
|Short-circuit read, 1MB bytes per checksum|38,500|937,800|
|Short-circuit read, 16KB bytes per checksum|534,900|911,400|
Environment:
* Ozone 2.3.0-SNAPSHOT, master at 24cf9aaa28e, Hadoop 3.4.3, Ratis 3.3.1
* MiniOzoneCluster with 3 datanodes in one JVM ({{-Xmx8192m}}, G1), 16 files of
64MB, RATIS/THREE
* {{apache/ozone-runner:20260626-1-jdk25}} container (Rocky Linux 9.8, OpenJDK
25.0.2), with native libhadoop for Hadoop 3.4.3 (linux aarch64) for
short-circuit read
* Docker Desktop Linux VM on arm64 with 14 CPUs and 23.4 GB, on an Apple M4 Pro
host (14 cores, 48 GB)
* 3 s warm up and 8 s per measurement, host at least 89% idle at the start of
each run
* Single machine, loopback network, all data in the page cache, security off.
Treat the figures as an upper bound.
was:HBase calls setVerifyChecksum(false) on its file system, but ofs and o3fs
do not override it, so the Ozone client still verifies every read.
> Support FileSystem#setVerifyChecksum in ofs and o3fs
> ----------------------------------------------------
>
> Key: HDDS-16783
> URL: https://issues.apache.org/jira/browse/HDDS-16783
> Project: Apache Ozone
> Issue Type: Improvement
> Components: Ozone Filesystem
> Reporter: Siyao Meng
> Priority: Major
>
> h3. Problem
> HBase stores its own checksums in HFile blocks. When
> {{hbase.regionserver.checksum.verify}} is true (the default), {{HFileSystem}}
> opens a second FileSystem instance, calls {{setVerifyChecksum(false)}} on it,
> and reads HFile blocks through that instance. On HDFS this skips the file
> system checksum.
> {{BasicOzoneFileSystem}} and {{BasicRootedOzoneFileSystem}} do not override
> {{setVerifyChecksum}}, so the call reaches the empty default in
> {{FileSystem}} and the Ozone client still verifies every read. This costs
> HBase twice:
> * the data is checksummed by both Ozone and HBase
> * with verification on, each read is widened to checksum boundaries
> ({{ozone.client.bytes.per.checksum}}), so with a 1MB value a 64KB HFile block
> read fetches and checksums 1MB
> {{ozone.client.verify.checksum=false}} is not a good substitute. It applies
> to the whole process, including the FileSystem instance HBase falls back to
> when an HFile checksum fails, so that retry would no longer be verified by
> anything.
> h3. Proposal
> Override {{setVerifyChecksum}} in both file systems and pass the flag through
> {{OzoneClientAdapter}} and {{ClientProtocol}} to the client's own
> {{OzoneClientConfig}}, so that input streams opened afterwards by that
> FileSystem instance skip checksum verification. The setting is per instance,
> the same as {{DistributedFileSystem}}.
> No protocol change is needed. The existing read paths already read this flag
> when a stream is opened: ReadChunk, short-circuit read, and streaming
> ReadBlock, which passes it to the datanode as {{includeChecksums}}
> (HDDS-16258).
> h3. Expected gain
> 64KB random positional reads ({{readFully(pos, buf, 0, len)}} on one shared
> input stream per file), 16 threads, operations per second, median of three
> rounds. "Verify off" was measured with
> {{ozone.client.verify.checksum=false}}, which is what the API call would
> toggle per instance.
> ||Read path||Verify on||Verify off||
> |gRPC, 1MB bytes per checksum|6,100|22,600|
> |gRPC, 16KB bytes per checksum|22,100|22,600|
> |Short-circuit read, 1MB bytes per checksum|38,500|937,800|
> |Short-circuit read, 16KB bytes per checksum|534,900|911,400|
> Environment:
> * Ozone 2.3.0-SNAPSHOT, master at 24cf9aaa28e, Hadoop 3.4.3, Ratis 3.3.1
> * MiniOzoneCluster with 3 datanodes in one JVM ({{-Xmx8192m}}, G1), 16 files
> of 64MB, RATIS/THREE
> * {{apache/ozone-runner:20260626-1-jdk25}} container (Rocky Linux 9.8,
> OpenJDK 25.0.2), with native libhadoop for Hadoop 3.4.3 (linux aarch64) for
> short-circuit read
> * Docker Desktop Linux VM on arm64 with 14 CPUs and 23.4 GB, on an Apple M4
> Pro host (14 cores, 48 GB)
> * 3 s warm up and 8 s per measurement, host at least 89% idle at the start of
> each run
> * Single machine, loopback network, all data in the page cache, security off.
> Treat the figures as an upper bound.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
