Andrey Yarovoy created HDDS-17491:
-------------------------------------
Summary: XceiverClientManager should close idle clients even when
its cache is no longer used
Key: HDDS-17491
URL: https://issues.apache.org/jira/browse/HDDS-17491
Project: Apache Ozone
Issue Type: Improvement
Components: Ozone Client
Reporter: Andrey Yarovoy
`XceiverClientManager` keeps datanode clients in a Guava cache with
`expireAfterAccess(scm.container.client.idle.threshold)`
(default 10 s). Evicting a client marks it evicted, and it is closed once no
one holds it. Guava doesn't evict on a
timer, though. It removes expired entries only during cache operations, and
only in the segment the operation
touches (the cache has 4 segments by default). As a result:
- **An idle JVM never closes its clients.** Once the application stops writing,
nothing uses the cache again, so
connections, channels and their threads stay open until
`XceiverClientManager.close()` runs.
- **A busy JVM closes idle clients only by chance.** When some clients are idle
and others are still used, each
cache write cleans up only one segment. With 16 idle clients, one write after
the threshold closed 5, 5 and 6 of
them in three runs.
For Ratis data streaming this means a JVM that doesn't close its FileSystem
never exits (HDDS-17278). The streaming
client holds the shared Netty worker group, and its non-daemon threads keep the
JVM alive. Hadoop's shutdown hook
would close the FileSystem, but it never runs because the JVM never starts
shutting down. If idle clients were
closed, the worker group would be released and its threads would stop.
#### Proposed change
- Give `XceiverClientManager` a single-thread `ScheduledExecutorService` that
runs `clientCache.cleanUp()`
periodically, for example every idle threshold. Expired clients that no one
holds are then closed about one
threshold after their last use.
- The thread must be a **daemon** thread, so it doesn't keep a JVM alive
itself. Give it a recognizable name, for
example `XceiverClientManager-cleanup`.
- Stop it in `XceiverClientManager.close()`, before the final `cleanUp()`.
- Clients that are still referenced stay open until they are released; the
existing ref-count rule is unchanged.
- As a side benefit, idle clients are no longer closed by whichever thread
happens to use the cache, under the
`clientCache` lock (see HDDS-15849), but by the cleanup thread.
#### Effect on HDDS-17278
A JVM that wrote with data streaming and doesn't close its FileSystem exits
about 12 s after its last write. That is
the 10 s idle threshold plus Netty's default 2 s quiet period for
`shutdownGracefully`. Today it doesn't exit until
it is killed: 60 s for a MapReduce task (`mapreduce.task.exit.timeout`), or
never for other clients. An immediate
exit needs the Ratis fix (linked RATIS issue).
#### Testing
- **`TestXceiverClientManagerIdleEviction`** (`hdds-client`, uses a 200 ms
threshold and real, lazily connected gRPC
clients, no datanode needed). The first two tests fail today:
- a released client is closed without any further cache use;
- 16 idle clients are all closed when another client is used;
- an explicit `cleanUp()` closes an expired, released client;
- an expired client that is still referenced stays open until it is released.
- Check that the cleanup thread is a daemon thread and stops on `close()`.
- **Integration:** a child JVM writes a file with
`ozone.fs.datastream.enabled=true`, returns from `main` without
closing the FileSystem, and must exit within the idle threshold plus a margin.
The test should use a short
threshold.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
