Andrey Yarovoy created HDDS-16872:
-------------------------------------
Summary: EC: Report EC read bytes and decode time in filesystem
statistics
Key: HDDS-16872
URL: https://issues.apache.org/jira/browse/HDDS-16872
Project: Apache Ozone
Issue Type: Improvement
Components: EC Client
Reporter: Andrey Yarovoy
h2. Problem
{\{OzoneFSInputStream}} only calls
\{{FileSystem.Statistics.incrementBytesRead}}.
Engines and users that read EC data cannot see, per filesystem or per stream:
* how many bytes were read from EC keys;
* how much time was spent decoding;
* how many blocks were read in reconstruction mode.
The EC reconstruction counters that exist today are client-wide, in
{\{XceiverClientMetrics}}.
h2. HDFS behavior
{\{DFSStripedInputStream}} adds EC decoding time to \{{ReadStatistics}}, and
{\{DFSClient.updateFileSystemECReadStats}} calls
{\{FileSystem.Statistics.incrementBytesReadErasureCoded}}.
h2. Proposal
* Call \{{incrementBytesReadErasureCoded}} for reads of EC keys. That method
exists only in Hadoop 3, so put the call in the Hadoop 3 filesystem path (for
example \{{CapableOzoneFSInputStream}}) or guard it, so \{{ozonefs-hadoop2}}
keeps working. The stream needs to know the key is EC, which the key lookup
that opens it already knows.
* Collect per-stream EC counters in the client (bytes decoded, decode time,
blocks read in reconstruction mode, failovers) and expose them on the
filesystem input stream as \{{IOStatistics}} (\{{IOStatisticsSource}}). That
interface is Hadoop 3.3+, so check the oldest Hadoop 3 runtime the
filesystem jar has to load on and guard the class if needed.
* Keep the existing \{{XceiverClientMetrics}} counters unchanged.
h2. Compatibility
Client only. No wire or data change. New statistics only; existing ones keep
their names and values.
h2. Testing
* Unit: reading an EC key increments \{{bytesRead}} and
{\{bytesReadErasureCoded}} by the same amount; a replicated key leaves
{\{bytesReadErasureCoded}} at 0.
* Unit: a degraded read reports non-zero decode time and the reconstruction
block count in the stream's \{{IOStatistics}}.
* The \{{ozonefs-hadoop2}} build and its tests keep passing.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
