[
https://issues.apache.org/jira/browse/KUDU-3778?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18084081#comment-18084081
]
ASF subversion and git services commented on KUDU-3778:
-------------------------------------------------------
Commit 94b246ae31c71ed6a660d966b984e84c8fc3d314 in kudu's branch
refs/heads/master from Yan-Daojiang
[ https://gitbox.apache.org/repos/asf?p=kudu.git;h=94b246ae3 ]
KUDU-3778 [fs] speed up LBM startup by replaying small metadata files from
memory
On startup, LogBlockContainerNativeMeta::ProcessRecords replays a
container's .metadata file through ReadablePBContainerFile::ReadNextPB,
which issues two pread() syscalls per record. The total number of read
syscalls during container loading is therefore proportional to the
number of records across all containers, and on a tserver with many
containers each holding many records this turns the loading phase into
a long stream of tiny sequential reads against the kernel.
Add an opt-in fast path: when the payload fits in a configurable
budget, read the file once into a faststring and let
ReadablePBContainerFile parse it through an in-memory RandomAccessFile
shim. This collapses the per-record preadv pair into a single bulk
read per container, so the number of read syscalls during startup
drops from O(total records) to O(number of containers). Any IO error
or oversized file falls back to the existing streaming reader, so
the optimization can only help and never break startup.
The fast path is gated by
--log_container_metadata_inmem_replay_threshold_bytes (default 64 MiB,
runtime/advanced/experimental); set it to 0 to disable.
Encryption is intentionally left on the streaming path.
Benchmarks (release build, KUDU_ALLOW_SLOW_TESTS=1,
`log_block_manager-test --gtest_filter='*InMemoryReplayStartupBenchmark*'`,
5 reopens per pass, native-meta containers, encryption disabled):
"streaming" = --log_container_metadata_inmem_replay_threshold_bytes=0,
"in-memory" = --log_container_metadata_inmem_replay_threshold_bytes=64MiB.
Workload scale (batches x blocks/batch, dirs, 90% deletion):
200 x 500, 4 dirs ( 10k live): 154 ms -> 66 ms (2.33x, -57%)
500 x 1000, 8 dirs ( 50k live): 811 ms -> 394 ms (2.06x, -51%)
1000 x 1000, 8 dirs (101k live): 1643 ms -> 817 ms (2.01x, -50%)
Deletion-ratio sweep (500k writes, 500x1000 / 8 dirs):
deleted=0% (500k records, 500k live): 597 ms -> 379 ms (1.58x)
deleted=50% (750k records, 250k live): 734 ms -> 416 ms (1.76x)
deleted=90% (950k records, 50k live): 806 ms -> 388 ms (2.08x)
deleted=99% (995k records, 5k live): 836 ms -> 386 ms (2.17x)
The streaming-path time scales with total records (creates+deletes); the
in-memory-path time is nearly flat across deletion ratios (379-416 ms).
That is exactly the "O(records) preadv syscalls -> O(containers) bulk
reads" collapse this patch targets.
Container-density sweep (500x1000 / 8 dirs / 90% deletion, all 50k live;
numbers are steady-state, the first reopen of a fresh on-disk layout is
excluded as it is dominated by cold page-cache):
log_container_max_blocks=-1: 788 ms -> 384 ms (2.05x)
log_container_max_blocks=10000: 38 ms -> 23 ms (1.65x)
log_container_max_blocks=1000: 28 ms -> 22 ms (1.27x)
The optimization's absolute and relative gain both grow with the number
of records per container, as expected: each container's per-record
preadv() pair is replaced by a single bulk read at open time.
Real-world perf trace:
Before (streaming, two preadv() per record):
7 metadata worker threads each issued ~4.2 M preadv() calls,
totaling ~30.8 M preadv() syscalls in ~35 s of per-thread kernel
time (~294 s summed across threads). preadv avg syscall duration: 8 us.
perf trace itself reported "LOST <n> events!" thousands of times
during this window, i.e. the syscall rate overran the perf ring
buffer.
After (in-memory replay,
--log_container_metadata_inmem_replay_threshold_bytes=64MiB):
the same 7 worker threads issued 17-109 preadv() calls each,
totaling 358 preadv() syscalls (the file-management thread did an
additional ~7.7K preadv() on non-metadata files in both runs and is
unchanged). preadv avg syscall duration rose to 60-130 ms because
each call now slurps a whole ~8 MiB metadata file in one go,
but per-thread preadv() wall time dropped from ~35 s to ~5-6.5 s.
perf trace reported zero event loss.
Change-Id: Iacbe12977ae945e7fa2f97a41aef250b03495cd4
Reviewed-on: http://gerrit.cloudera.org:8080/24326
Reviewed-by: Ashwani Raina <[email protected]>
Reviewed-by: Zoltan Martonka <[email protected]>
Reviewed-by: Marton Greber <[email protected]>
Tested-by: Marton Greber <[email protected]>
> Speed up log block manager startup by replaying small container metadata
> files from memory
> ------------------------------------------------------------------------------------------
>
> Key: KUDU-3778
> URL: https://issues.apache.org/jira/browse/KUDU-3778
> Project: Kudu
> Issue Type: Improvement
> Reporter: Daojiang Yan
> Assignee: Daojiang Yan
> Priority: Major
> Fix For: Backlog
>
>
> *Description*
> On startup,LogBlockContainerNativeMeta::ProcessRecords() replays each
> container's. metadata file through ReadablePBContainerFile::ReadNextPB(),
> which issues two preadv() syscalls per record (one for the length+checksum
> prefix, one for the body+checksum). The total number of read syscalls during
> the "Reading filesystem" phase is therefore O(total records across all
> containers). On a tserver with many containers each holding many records,
> this turns startup into a long stream of tiny sequential reads against the
> kernel.
>
> *Proposal*
> Add an opt-in fast path: when a metadata file's payload fits in a
> configurable budget, read it once into a faststring and let
> ReadablePBContainerFile parse it through an in-memory RandomAccessFile shim
> (MemoryReadableFile). This collapses the per-record preadv() pair into a
> single bulk read per container, so the syscall count during startup drops
> from O(records) to O(containers). Any I/O error during preload, or an
> oversized file, transparently falls back to the existing streaming path, so
> the optimization can only help and never widen the failure envelope of
> startup.
>
> The fast path is gated by
> --log_container_metadata_inmem_replay_threshold_bytes (default 64 MiB;
> runtime/advanced/experimental). Set to 0 to disable. Only applies to
> --block_manager=log with native metadata; encrypted metadata files are
> intentionally left on the streaming path.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
