[
https://issues.apache.org/jira/browse/HDDS-17412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Soldatov resolved HDDS-17412.
------------------------------------
Resolution: Duplicate
The HDDS-16870 address the issue and patch is already available.
> EC positioned read creates a new reader per call, cutting random pread
> throughput by about 45%
> ----------------------------------------------------------------------------------------------
>
> Key: HDDS-17412
> URL: https://issues.apache.org/jira/browse/HDDS-17412
> Project: Apache Ozone
> Issue Type: Bug
> Components: EC Client
> Reporter: Siyao Meng
> Assignee: Sergey Soldatov
> Priority: Major
>
> h2. Problem
> Since HDDS-16396, {{ECBlockInputStreamProxy#readPositioned}} creates a new EC
> reader for every positioned read and closes it afterwards. The cached
> per-index {{BlockInputStream}} instances are thrown away each time, so every
> pread issues a fresh {{GetBlock}} RPC to the datanode for each EC cell it
> touches before reading any data.
> Compared with its parent commit on a five-datanode mini-cluster (RS-3-2, 1
> MiB cells, single-threaded client):
> ||EC workload||Parent (reads/s)||HDDS-16396 (reads/s)||Change||
> |Random pread 4 KiB|4,926|2,661|-46%|
> |Random pread 64 KiB|3,779|2,192|-42%|
> |Sequential read 64 KiB|4,023|4,019|-0.1%|
> For 2,000 random 64 KiB preads after warm-up, {{GetBlock}} RPCs went from 0
> to 2,121 while {{ReadChunk}} stayed at 2,121. Ratis keys are not affected.
> Reusing the existing reader for preads (seek, read, restore) restores the
> parent's throughput and removes the extra RPCs, which confirms that creating
> a reader per call causes the slowdown. Current master has the same behavior.
> h2. Proposed fix
> Add a fast path for healthy preads in {{ECBlockInputStream}}: map the
> requested range to EC cells and call the existing stateless
> {{readPositioned}} on the cached per-index block streams, without moving the
> sequential cursor. Lazy stream creation stays synchronized; datanode I/O does
> not hold the lock, so concurrent preads still overlap. The existing parallel
> cell read option also applies to preads.
> {{ECBlockInputStreamProxy}} uses this fast path while the block is healthy.
> On failure it restores the destination buffer position and falls back to the
> existing request-owned reader. That reader starts with the failed indexes
> excluded, so it goes straight to a spare replica or reconstruction.
> The fix needs no new configuration, wire change or reader pool.
> With the fix on current master, same benchmark:
> ||EC workload||Unpatched (reads/s)||Fixed (reads/s)||
> |Random pread 4 KiB|2,776|5,001|
> |Random pread 64 KiB|2,283|3,828|
> |Positioned scan 64 KiB|2,448|4,074|
> Warmed EC {{GetBlock}} RPCs drop to zero. Ratis throughput and EC sequential
> reads are unchanged.
> h2. Tests
> Extend {{TestECBlockInputStream}} and {{TestECBlockInputStreamProxy}} to
> cover:
> * reader reuse: 100 preads use one EC reader (without the fix, 101 are
> created);
> * overlapping concurrent preads;
> * cell and stripe boundaries, partial last stripes, EOF, and an unchanged
> cursor;
> * failure after partial output, retry on a spare replica, reconstruction
> fallback, and simultaneous failures on several indexes.
> Related: HDDS-16870 proposes stateless positioned reads for EC keys more
> broadly.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
