[
https://issues.apache.org/jira/browse/IGNITE-28992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nikita Amelchev updated IGNITE-28992:
-------------------------------------
Description:
A lock inversion can occur when an SSL client handshake races with its timeout.
The timeout worker holds the {{CancelableTask}} monitor and waits for the SSL
lock while closing the session, whereas the NIO worker holds the SSL lock and
waits for the task monitor in {{{}CancelableTask.close(){}}}. This deadlocks
both threads and may prevent node shutdown.
{code:java}
Found one Java-level deadlock: =============================
"grid-timeout-worker-#118": waiting for ownable synchronizer
0x00001006e2545cb0, (a java.util.concurrent.locks.ReentrantLock$NonfairSync),
which is held by "grid-nio-worker-client-listener-17-#168"
"grid-nio-worker-client-listener-17-#168": waiting to lock monitor
0x00007f81cc0468f0 (object 0x00001007bfed44e0, a
org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor$CancelableTask),
which is held by "grid-timeout-worker-#118" Java stack information for the
threads listed above: ===================================================
"grid-timeout-worker-#118":
at jdk.internal.misc.Unsafe.park([email protected]/Native Method) - parking to
wait for <0x00001006e2545cb0> (a
java.util.concurrent.locks.ReentrantLock$NonfairSync)
at
java.util.concurrent.locks.LockSupport.park([email protected]/LockSupport.java:211)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire([email protected]/AbstractQueuedSynchronizer.java:715)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire([email protected]/AbstractQueuedSynchronizer.java:938)
at
java.util.concurrent.locks.ReentrantLock$Sync.lock([email protected]/ReentrantLock.java:153)
at
java.util.concurrent.locks.ReentrantLock.lock([email protected]/ReentrantLock.java:322)
at
org.apache.ignite.internal.util.nio.ssl.GridNioSslFilter.onSessionClose(GridNioSslFilter.java:432)
at
org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedSessionClose(GridNioFilterAdapter.java:128)
at
org.apache.ignite.internal.util.nio.GridNioCodecFilter.onSessionClose(GridNioCodecFilter.java:137)
at
org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedSessionClose(GridNioFilterAdapter.java:128)
at
org.apache.ignite.internal.util.nio.GridNioAsyncNotifyFilter.onSessionClose(GridNioAsyncNotifyFilter.java:124)
at
org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedSessionClose(GridNioFilterAdapter.java:128)
at
org.apache.ignite.internal.util.nio.GridNioFilterChain$TailFilter.onSessionClose(GridNioFilterChain.java:274)
at
org.apache.ignite.internal.util.nio.GridNioFilterChain.onSessionClose(GridNioFilterChain.java:203)
at
org.apache.ignite.internal.util.nio.GridNioSessionImpl.close(GridNioSessionImpl.java:169)
at
org.apache.ignite.internal.util.nio.GridSelectorNioSessionImpl.close(GridSelectorNioSessionImpl.java:498)
at
org.apache.ignite.internal.processors.odbc.ClientListenerNioListener$1.run(ClientListenerNioListener.java:324)
at
org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor$CancelableTask.onTimeout(GridTimeoutProcessor.java:371)
- locked <0x00001007bfed44e0> (a
org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor$CancelableTask)
at
org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor$TimeoutWorker.body(GridTimeoutProcessor.java:240)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:125)
at java.lang.Thread.run([email protected]/Thread.java:840)
"grid-nio-worker-client-listener-17-#168":
at
org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor$CancelableTask.close(GridTimeoutProcessor.java:388)
- waiting to lock <0x00001007bfed44e0> (a
org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor$CancelableTask)
at
org.apache.ignite.internal.processors.odbc.ClientListenerNioListener.cancelHandshakeTimeout(ClientListenerNioListener.java:345)
at
org.apache.ignite.internal.processors.odbc.ClientListenerNioListener.onHandshake(ClientListenerNioListener.java:407)
at
org.apache.ignite.internal.processors.odbc.ClientListenerNioListener.onMessage(ClientListenerNioListener.java:186)
at
org.apache.ignite.internal.processors.odbc.ClientListenerNioListener.onMessage(ClientListenerNioListener.java:60)
at
org.apache.ignite.internal.util.nio.GridNioFilterChain$TailFilter.onMessageReceived(GridNioFilterChain.java:279)
at
org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:109)
at
org.apache.ignite.internal.processors.odbc.ClientListenerProcessor$1.onMessageReceived(ClientListenerProcessor.java:460)
at
org.apache.ignite.internal.processors.odbc.ClientListenerProcessor$1.onMessageReceived(ClientListenerProcessor.java:448)
at
org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:109)
at
org.apache.ignite.internal.util.nio.GridNioCodecFilter.onMessageReceived(GridNioCodecFilter.java:116)
at
org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:109)
at
org.apache.ignite.internal.util.nio.ssl.GridNioSslFilter.processApplicationBuffer(GridNioSslFilter.java:423)
at
org.apache.ignite.internal.util.nio.ssl.GridNioSslFilter.onMessageReceived(GridNioSslFilter.java:401)
at
org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:109)
at
org.apache.ignite.internal.util.nio.GridNioServer$HeadFilter.onMessageReceived(GridNioServer.java:3749)
at
org.apache.ignite.internal.util.nio.GridNioFilterChain.onMessageReceived(GridNioFilterChain.java:175)
at
org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processRead(GridNioServer.java:1379)
at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2518)
at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2273)
at
org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1904)
at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:125)
at java.lang.Thread.run([email protected]/Thread.java:840) Found 1 deadlock.
{code}
was:A lock inversion can occur when an SSL client handshake races with its
timeout. The timeout worker holds the {{CancelableTask}} monitor and waits for
the SSL lock while closing the session, whereas the NIO worker holds the SSL
lock and waits for the task monitor in {{{}CancelableTask.close(){}}}. This
deadlocks both threads and may prevent node shutdown.
> Fix SSL handshake timeout deadlock in the client connector
> ----------------------------------------------------------
>
> Key: IGNITE-28992
> URL: https://issues.apache.org/jira/browse/IGNITE-28992
> Project: Ignite
> Issue Type: Bug
> Reporter: Nikita Amelchev
> Assignee: Nikita Amelchev
> Priority: Major
> Labels: ise
> Fix For: 2.19
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> A lock inversion can occur when an SSL client handshake races with its
> timeout. The timeout worker holds the {{CancelableTask}} monitor and waits
> for the SSL lock while closing the session, whereas the NIO worker holds the
> SSL lock and waits for the task monitor in {{{}CancelableTask.close(){}}}.
> This deadlocks both threads and may prevent node shutdown.
>
> {code:java}
> Found one Java-level deadlock: =============================
> "grid-timeout-worker-#118": waiting for ownable synchronizer
> 0x00001006e2545cb0, (a java.util.concurrent.locks.ReentrantLock$NonfairSync),
> which is held by "grid-nio-worker-client-listener-17-#168"
> "grid-nio-worker-client-listener-17-#168": waiting to lock monitor
> 0x00007f81cc0468f0 (object 0x00001007bfed44e0, a
> org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor$CancelableTask),
> which is held by "grid-timeout-worker-#118" Java stack information for the
> threads listed above: ===================================================
> "grid-timeout-worker-#118":
> at jdk.internal.misc.Unsafe.park([email protected]/Native Method) - parking
> to wait for <0x00001006e2545cb0> (a
> java.util.concurrent.locks.ReentrantLock$NonfairSync)
> at
> java.util.concurrent.locks.LockSupport.park([email protected]/LockSupport.java:211)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire([email protected]/AbstractQueuedSynchronizer.java:715)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire([email protected]/AbstractQueuedSynchronizer.java:938)
> at
> java.util.concurrent.locks.ReentrantLock$Sync.lock([email protected]/ReentrantLock.java:153)
> at
> java.util.concurrent.locks.ReentrantLock.lock([email protected]/ReentrantLock.java:322)
> at
> org.apache.ignite.internal.util.nio.ssl.GridNioSslFilter.onSessionClose(GridNioSslFilter.java:432)
> at
> org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedSessionClose(GridNioFilterAdapter.java:128)
> at
> org.apache.ignite.internal.util.nio.GridNioCodecFilter.onSessionClose(GridNioCodecFilter.java:137)
> at
> org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedSessionClose(GridNioFilterAdapter.java:128)
> at
> org.apache.ignite.internal.util.nio.GridNioAsyncNotifyFilter.onSessionClose(GridNioAsyncNotifyFilter.java:124)
> at
> org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedSessionClose(GridNioFilterAdapter.java:128)
> at
> org.apache.ignite.internal.util.nio.GridNioFilterChain$TailFilter.onSessionClose(GridNioFilterChain.java:274)
> at
> org.apache.ignite.internal.util.nio.GridNioFilterChain.onSessionClose(GridNioFilterChain.java:203)
> at
> org.apache.ignite.internal.util.nio.GridNioSessionImpl.close(GridNioSessionImpl.java:169)
> at
> org.apache.ignite.internal.util.nio.GridSelectorNioSessionImpl.close(GridSelectorNioSessionImpl.java:498)
> at
> org.apache.ignite.internal.processors.odbc.ClientListenerNioListener$1.run(ClientListenerNioListener.java:324)
> at
> org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor$CancelableTask.onTimeout(GridTimeoutProcessor.java:371)
> - locked <0x00001007bfed44e0> (a
> org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor$CancelableTask)
> at
> org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor$TimeoutWorker.body(GridTimeoutProcessor.java:240)
> at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:125)
> at java.lang.Thread.run([email protected]/Thread.java:840)
> "grid-nio-worker-client-listener-17-#168":
> at
> org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor$CancelableTask.close(GridTimeoutProcessor.java:388)
> - waiting to lock <0x00001007bfed44e0> (a
> org.apache.ignite.internal.processors.timeout.GridTimeoutProcessor$CancelableTask)
> at
> org.apache.ignite.internal.processors.odbc.ClientListenerNioListener.cancelHandshakeTimeout(ClientListenerNioListener.java:345)
> at
> org.apache.ignite.internal.processors.odbc.ClientListenerNioListener.onHandshake(ClientListenerNioListener.java:407)
> at
> org.apache.ignite.internal.processors.odbc.ClientListenerNioListener.onMessage(ClientListenerNioListener.java:186)
> at
> org.apache.ignite.internal.processors.odbc.ClientListenerNioListener.onMessage(ClientListenerNioListener.java:60)
> at
> org.apache.ignite.internal.util.nio.GridNioFilterChain$TailFilter.onMessageReceived(GridNioFilterChain.java:279)
> at
> org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:109)
> at
> org.apache.ignite.internal.processors.odbc.ClientListenerProcessor$1.onMessageReceived(ClientListenerProcessor.java:460)
> at
> org.apache.ignite.internal.processors.odbc.ClientListenerProcessor$1.onMessageReceived(ClientListenerProcessor.java:448)
> at
> org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:109)
> at
> org.apache.ignite.internal.util.nio.GridNioCodecFilter.onMessageReceived(GridNioCodecFilter.java:116)
> at
> org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:109)
> at
> org.apache.ignite.internal.util.nio.ssl.GridNioSslFilter.processApplicationBuffer(GridNioSslFilter.java:423)
> at
> org.apache.ignite.internal.util.nio.ssl.GridNioSslFilter.onMessageReceived(GridNioSslFilter.java:401)
> at
> org.apache.ignite.internal.util.nio.GridNioFilterAdapter.proceedMessageReceived(GridNioFilterAdapter.java:109)
> at
> org.apache.ignite.internal.util.nio.GridNioServer$HeadFilter.onMessageReceived(GridNioServer.java:3749)
> at
> org.apache.ignite.internal.util.nio.GridNioFilterChain.onMessageReceived(GridNioFilterChain.java:175)
> at
> org.apache.ignite.internal.util.nio.GridNioServer$DirectNioClientWorker.processRead(GridNioServer.java:1379)
> at
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.processSelectedKeysOptimized(GridNioServer.java:2518)
> at
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.bodyInternal(GridNioServer.java:2273)
> at
> org.apache.ignite.internal.util.nio.GridNioServer$AbstractNioClientWorker.body(GridNioServer.java:1904)
> at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:125)
> at java.lang.Thread.run([email protected]/Thread.java:840) Found 1 deadlock.
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
