Octavian Ciubotaru created KAFKA-16051:
------------------------------------------
Summary: Deadlock on connector initialization
Key: KAFKA-16051
URL: https://issues.apache.org/jira/browse/KAFKA-16051
Project: Kafka
Issue Type: Bug
Components: connect
Affects Versions: 3.6.1, 2.6.3
Reporter: Octavian Ciubotaru
Tested with Kafka 3.6.1 and 2.6.3.
The only plugin installed is confluentinc-kafka-connect-jdbc-10.7.4.
Stack trace for Kafka 3.6.1:
{noformat}
Found one Java-level deadlock:
=============================
"pool-3-thread-1":
waiting to lock monitor 0x00007fbc88006300 (object 0x0000000091002aa0, a
org.apache.kafka.connect.runtime.standalone.StandaloneHerder),
which is held by "Thread-9"
"Thread-9":
waiting to lock monitor 0x00007fbc88008800 (object 0x000000009101ccd8, a
org.apache.kafka.connect.storage.MemoryConfigBackingStore),
which is held by "pool-3-thread-1"Java stack information for the threads
listed above:
===================================================
"pool-3-thread-1":
at
org.apache.kafka.connect.runtime.standalone.StandaloneHerder$ConfigUpdateListener.onTaskConfigUpdate(StandaloneHerder.java:516)
- waiting to lock <0x0000000091002aa0> (a
org.apache.kafka.connect.runtime.standalone.StandaloneHerder)
at
org.apache.kafka.connect.storage.MemoryConfigBackingStore.putTaskConfigs(MemoryConfigBackingStore.java:137)
- locked <0x000000009101ccd8> (a
org.apache.kafka.connect.storage.MemoryConfigBackingStore)
at
org.apache.kafka.connect.runtime.standalone.StandaloneHerder.updateConnectorTasks(StandaloneHerder.java:483)
at
org.apache.kafka.connect.runtime.standalone.StandaloneHerder.lambda$null$2(StandaloneHerder.java:229)
at
org.apache.kafka.connect.runtime.standalone.StandaloneHerder$$Lambda$692/0x0000000840557440.run(Unknown
Source)
at
java.util.concurrent.Executors$RunnableAdapter.call(java.base@11.0.21/Executors.java:515)
at
java.util.concurrent.FutureTask.run(java.base@11.0.21/FutureTask.java:264)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(java.base@11.0.21/ScheduledThreadPoolExecutor.java:304)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.21/ThreadPoolExecutor.java:1128)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.21/ThreadPoolExecutor.java:628)
at java.lang.Thread.run(java.base@11.0.21/Thread.java:829)
"Thread-9":
at
org.apache.kafka.connect.storage.MemoryConfigBackingStore.putTaskConfigs(MemoryConfigBackingStore.java:129)
- waiting to lock <0x000000009101ccd8> (a
org.apache.kafka.connect.storage.MemoryConfigBackingStore)
at
org.apache.kafka.connect.runtime.standalone.StandaloneHerder.updateConnectorTasks(StandaloneHerder.java:483)
at
org.apache.kafka.connect.runtime.standalone.StandaloneHerder.requestTaskReconfiguration(StandaloneHerder.java:255)
- locked <0x0000000091002aa0> (a
org.apache.kafka.connect.runtime.standalone.StandaloneHerder)
at
org.apache.kafka.connect.runtime.HerderConnectorContext.requestTaskReconfiguration(HerderConnectorContext.java:50)
at
org.apache.kafka.connect.runtime.WorkerConnector$WorkerConnectorContext.requestTaskReconfiguration(WorkerConnector.java:548)
at
io.confluent.connect.jdbc.source.TableMonitorThread.run(TableMonitorThread.java:86)
Found 1 deadlock.
{noformat}
The jdbc source connector is loading tables from the database and updates the
configuration once the list is available. The deadlock is very consistent in my
environment, probably because the database is on the same machine.
Maybe it is possible to avoid this situation by always locking the herder first
and the config backing store second. From what I see, updateConnectorTasks
sometimes is called before locking on herder and other times it is not.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
