[
https://issues.apache.org/jira/browse/HDDS-17254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang reassigned HDDS-17254:
--------------------------------------
Assignee: Wei-Chiu Chuang
> Client-side non-daemon executors and timers can prevent JVM shutdown
> --------------------------------------------------------------------
>
> Key: HDDS-17254
> URL: https://issues.apache.org/jira/browse/HDDS-17254
> Project: Apache Ozone
> Issue Type: Bug
> Components: EC Client, Ozone Client
> Reporter: Wei-Chiu Chuang
> Assignee: Wei-Chiu Chuang
> Priority: Major
> Labels: pull-request-available
>
> h2. Problem
> HDDS-10564 / [PR #6418|https://github.com/apache/ozone/pull/6418] made
> RpcClient output-stream executors use daemon threads so an unclosed
> FileSystem/client would not keep a command-line JVM alive. A source audit
> found four other client-side executor/timer creation sites with the same
> potential shutdown risk.
> This report is based on source inspection at
> d0949f291791c399f3e54409fcb92656a80597a4. JVM-exit hangs have not yet been
> reproduced in a subprocess test. Not every invocation starts the relevant
> worker: the conditions for each path are listed below.
> h2. Affected paths
> h3. 1. Ratis streaming writes: BlockDataStreamOutput
> The constructor creates its response executor with
> Executors.newSingleThreadExecutor(), whose default thread factory creates
> non-daemon threads. This is separate from RpcClient.writeExecutor, so
> HDDS-10564 does not cover it. After an asynchronous response starts the
> worker, an abandoned/unclosed streaming output can keep the JVM alive. Normal
> close calls cleanup in finally, and cleanup shuts down the executor.
> * Executor creation:
> https://github.com/apache/ozone/blob/d0949f291791c399f3e54409fcb92656a80597a4/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockDataStreamOutput.java#L195
> * Normal close/cleanup:
> https://github.com/apache/ozone/blob/d0949f291791c399f3e54409fcb92656a80597a4/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockDataStreamOutput.java#L689
> * Executor shutdown:
> https://github.com/apache/ozone/blob/d0949f291791c399f3e54409fcb92656a80597a4/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/BlockDataStreamOutput.java#L736
> h3. 2. Short-circuit reads: per-connection timer
> XceiverClientShortCircuit.connect() creates new Timer(prefix + "-Timer")
> without the daemon argument. The timer thread starts when the connection is
> established. If client cleanup is omitted, it can keep the JVM alive. Normal
> close cancels the timer.
> *
> https://github.com/apache/ozone/blob/d0949f291791c399f3e54409fcb92656a80597a4/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/XceiverClientShortCircuit.java#L143
> h3. 3. Short-circuit reads: singleton DomainSocketFactory timer
> When short-circuit reads are configured and native support is available,
> DomainSocketFactory creates new Timer(name) without the daemon argument. This
> is a second non-daemon timer, separate from the connection timer above.
> Factory close cancels it, but an unclosed client/factory can leave it alive.
> * Creation:
> https://github.com/apache/ozone/blob/d0949f291791c399f3e54409fcb92656a80597a4/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/DomainSocketFactory.java#L160
> * Cancellation:
> https://github.com/apache/ozone/blob/d0949f291791c399f3e54409fcb92656a80597a4/hadoop-hdds/client/src/main/java/org/apache/hadoop/hdds/scm/storage/DomainSocketFactory.java#L272
> h3. 4. Default low-level EC reader factory
> BlockInputStreamFactoryImpl's no-argument constructor supplies
> Executors::newSingleThreadExecutor. If an EC reconstruction reader is
> instantiated through this default factory and executes reconstruction work,
> it can start non-daemon workers. The reconstruction stream's close method
> does not shut down the supplied executor. The normal RpcClient path supplies
> its own daemon executors and is not affected by this default-factory creation
> site.
> * Default factory:
> https://github.com/apache/ozone/blob/d0949f291791c399f3e54409fcb92656a80597a4/hadoop-hdds/client/src/main/java/org/apache/hadoop/ozone/client/io/BlockInputStreamFactoryImpl.java#L70
> * Reconstruction executor acquisition:
> https://github.com/apache/ozone/blob/d0949f291791c399f3e54409fcb92656a80597a4/hadoop-hdds/client/src/main/java/org/apache/hadoop/ozone/client/io/ECBlockInputStreamFactoryImpl.java#L95
> * Reconstruction stream close:
> https://github.com/apache/ozone/blob/d0949f291791c399f3e54409fcb92656a80597a4/hadoop-hdds/client/src/main/java/org/apache/hadoop/ozone/client/io/ECBlockReconstructedStripeInputStream.java#L738
> * A production caller of the default factory is the checksum-verification
> tool:
> https://github.com/apache/ozone/blob/d0949f291791c399f3e54409fcb92656a80597a4/hadoop-ozone/cli-debug/src/main/java/org/apache/hadoop/ozone/debug/replicas/ChecksumVerifier.java#L61
> h2. Expected behavior / suggested fix
> Client housekeeping workers should not prevent JVM termination solely because
> a caller has finished its work but omitted explicit cleanup. Apply the
> daemon-thread principle used by HDDS-10564 to the affected background
> executors/timers, while preserving normal close/cancel behavior.
> * Use a named daemon ThreadFactory for the streaming response executor.
> * Use daemon timers for both short-circuit timer sites.
> * Give the default EC reconstruction executor an explicit lifecycle/owner and
> daemon threads. Do not indiscriminately shut down caller-supplied shared
> executors from an individual stream's close method.
> h2. Validation needed
> Add isolated subprocess tests with bounded exit timeouts:
> # Start streaming response work, omit stream/client close, return from main,
> and verify the JVM exits.
> # Establish a short-circuit connection, omit client/factory close, return
> from main, and verify both timer threads do not retain the JVM (requires
> native short-circuit prerequisites).
> # Trigger EC reconstruction through the no-argument factory, close the
> stream, and verify no orphan executor keeps the JVM alive; also check the
> abandoned-stream case.
> # Verify explicit cleanup still terminates/releases the owned resources and
> does not break shared executors.
> Daemon workers do not replace application responsibility to flush and close
> streams. No data-durability guarantee is inferred from changing thread daemon
> status.
> Related prior fix: HDDS-10564.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
