[
https://issues.apache.org/jira/browse/HDDS-17491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDDS-17491:
----------------------------------
Labels: pull-request-available (was: )
> XceiverClientManager should close idle clients even when its cache is no
> longer used
> ------------------------------------------------------------------------------------
>
> Key: HDDS-17491
> URL: https://issues.apache.org/jira/browse/HDDS-17491
> Project: Apache Ozone
> Issue Type: Improvement
> Components: Ozone Client
> Reporter: Andrey Yarovoy
> Assignee: Andrey Yarovoy
> Priority: Major
> Labels: pull-request-available
>
> `XceiverClientManager` keeps datanode clients in a Guava cache with
> `expireAfterAccess(scm.container.client.idle.threshold)`
> (default 10 s). Evicting a client marks it evicted, and it is closed once no
> one holds it. Guava doesn't evict on a
> timer, though. It removes expired entries only during cache operations, and
> only in the segment the operation
> touches (the cache has 4 segments by default). As a result:
> - **An idle JVM never closes its clients.** Once the application stops
> writing, nothing uses the cache again, so
> connections, channels and their threads stay open until
> `XceiverClientManager.close()` runs.
> - **A busy JVM closes idle clients only by chance.** When some clients are
> idle and others are still used, each
> cache write cleans up only one segment. With 16 idle clients, one write after
> the threshold closed 5, 5 and 6 of
> them in three runs.
> For Ratis data streaming this means a JVM that doesn't close its FileSystem
> never exits (HDDS-17278). The streaming
> client holds the shared Netty worker group, and its non-daemon threads keep
> the JVM alive. Hadoop's shutdown hook
> would close the FileSystem, but it never runs because the JVM never starts
> shutting down. If idle clients were
> closed, the worker group would be released and its threads would stop.
> #### Proposed change
> - Give `XceiverClientManager` a single-thread `ScheduledExecutorService` that
> runs `clientCache.cleanUp()`
> periodically, for example every idle threshold. Expired clients that no one
> holds are then closed about one
> threshold after their last use.
> - The thread must be a **daemon** thread, so it doesn't keep a JVM alive
> itself. Give it a recognizable name, for
> example `XceiverClientManager-cleanup`.
> - Stop it in `XceiverClientManager.close()`, before the final `cleanUp()`.
> - Clients that are still referenced stay open until they are released; the
> existing ref-count rule is unchanged.
> - As a side benefit, idle clients are no longer closed by whichever thread
> happens to use the cache, under the
> `clientCache` lock (see HDDS-15849), but by the cleanup thread.
> #### Effect on HDDS-17278
> A JVM that wrote with data streaming and doesn't close its FileSystem exits
> about 12 s after its last write. That is
> the 10 s idle threshold plus Netty's default 2 s quiet period for
> `shutdownGracefully`. Today it doesn't exit until
> it is killed: 60 s for a MapReduce task (`mapreduce.task.exit.timeout`), or
> never for other clients. An immediate
> exit needs the Ratis fix (linked RATIS issue).
> #### Testing
> - **`TestXceiverClientManagerIdleEviction`** (`hdds-client`, uses a 200 ms
> threshold and real, lazily connected gRPC
> clients, no datanode needed). The first two tests fail today:
> - a released client is closed without any further cache use;
> - 16 idle clients are all closed when another client is used;
> - an explicit `cleanUp()` closes an expired, released client;
> - an expired client that is still referenced stays open until it is released.
> - Check that the cleanup thread is a daemon thread and stops on `close()`.
> - **Integration:** a child JVM writes a file with
> `ozone.fs.datastream.enabled=true`, returns from `main` without
> closing the FileSystem, and must exit within the idle threshold plus a
> margin. The test should use a short
> threshold.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
