Chu Cheng Li created HDDS-16728:
-----------------------------------
Summary: Limit the Ratis data stream client to 16 event loops by
default
Key: HDDS-16728
URL: https://issues.apache.org/jira/browse/HDDS-16728
Project: Apache Ozone
Issue Type: Sub-task
Components: Ozone Client
Reporter: Chu Cheng Li
Assignee: Chu Cheng Li
*Background*
A Ratis data stream client sends and receives data on netty event loops. All
the Ratis data stream clients in a JVM share one group of event loops.
{{raft.netty.dataStream.client.worker-group.size}} sets the size of this group.
The Ratis default is two times the number of CPU cores (768 on a 384-core
host). Ozone can set it with
{{hdds.ratis.raft.netty.dataStream.client.worker-group.size}}, but Ozone does
not set a default.
The Ratis data stream read of HDDS-9904 reads closed containers through their
read pipelines. SCM gives the read pipeline of each closed container a random
ID. Thus, the client opens one Ratis client and one connection for each closed
container that it reads.
*Problem*
Netty puts each connection on one event loop. The netty pooled allocator gives
each event loop thread a separate arena of 4 MiB chunks. A reply is a slice of
the buffer that received it. Thus, a chunk stays allocated until the client
releases all the replies in that chunk. An arena cannot use the free space of a
different arena.
*Effect*
In a TPC-H SF100 test with HBase on a 384-core host, each RegionServer opened
about 150 data stream connections. The connections used different event loops,
and each event loop used its own arena.
* At idle, the pool kept 30.5 GiB of direct memory in 7,632 chunks. 98.5% of
this memory was free.
* The RegionServers stopped with {{OutOfMemoryError: Cannot reserve ... bytes
of direct buffer memory}}.
* A limit on the read-ahead did not prevent this.
*Changes*
* Set the default of
{{hdds.ratis.raft.netty.dataStream.client.worker-group.size}} to 16.
* In the same test, 16 event loops kept the peak direct memory of SF100 q1 and
q6 at 3 to 9 GiB. The query times were the same as with the gRPC read.
*Open points*
* Datanodes use the same setting for the clients that forward streamed writes
to the other datanodes of a pipeline. Thus, a default of 16 can decrease the
write throughput of a busy datanode. A write benchmark must examine this.
Alternatively, Ozone can set the default only for clients, in
{{RatisHelper.createRaftClientProperties}}.
* If the client shares read connections for each datanode, it opens fewer
connections. Then this limit becomes less important.
*Tests*
* A unit test examines the default.
* A streaming write benchmark compares 16 event loops with the Ratis default.
This sub-task contains the event loop limit of
https://github.com/apache/ozone/pull/10743.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
