Hi,
Running Kafka 2.5.0_2.12.
We run 3 identically configured kafka clusters (one in each region) and on
one of them we've recently started having regular issues with spikes in CPU
utilization lasting a few hours. In some cases we see large numbers of what
look like replication failures, with the logs filled with:
[2020-08-06 04:05:53,766] ERROR [ReplicaManager broker=0] Error
processing append operation on partition __consumer_offsets-20
(kafka.server.ReplicaManager)org.apache.kafka.common.errors.NotEnoughReplicasException:
The size of the current ISR Set(0) is insufficient to satisfy the
min.isr requirement of 2 for partition __consumer_offsets-20
Once in this state the other brokers in the cluster also begin to fail
with similar messages and our producers being to error.
We've found that if we turn off all our producers temporarily the
cluster quickly returns to normal, which indicates some additional
load is being placed on the system, however all of our metrics show no
increased message rates.
Each cluster has 3 brokers, each with 4 CPU and 4 GiB memory, under
normal load we see 1000-5000 message/s being produced, and we see
maybe 10% utilization (the intention is to load this cluster more in
future).
During these periods of high utilization though we see CPU maxing out
at 100% utilization for long periods of time.
Searching around previous issues, I noticed
https://issues.apache.org/jira/browse/KAFKA-4477 and
https://issues.apache.org/jira/browse/KAFKA-6582 sounded similar,
however both marked as fixed long before this release.
I've attached a thread dump of one of the brokers during a high load
period. It looks to my untrained eye like everything is just waiting
on locks.
Any assistance in further diagnosing this issue would be really
appreciated. Our next step will probably be to completely rebuild this
cluster, as we've only ever seen these issues on one of our instances.
Regards,
Toby
2020-08-18 17:12:33
Full thread dump OpenJDK 64-Bit Server VM (25.252-b09 mixed mode):
"Attach Listener" #68 daemon prio=9 os_prio=0 tid=0x00007fd8c0001000 nid=0x5a3
waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Controller-0-to-broker-2-send-thread" #67 prio=5 os_prio=0
tid=0x00007fd788079000 nid=0x58e waiting on condition [0x00007fd75336c000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000074d15f9b8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:232)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)
Locked ownable synchronizers:
- None
"executor-Heartbeat" #66 prio=5 os_prio=0 tid=0x00007fd780003000 nid=0x58d
waiting on condition [0x00007fd7538fd000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007095aaa88> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- None
"executor-Rebalance" #65 prio=5 os_prio=0 tid=0x00007fd774003000 nid=0x58c
waiting on condition [0x00007fd7a01e6000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007095a27f8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- None
"kafka-scheduler-0" #64 daemon prio=5 os_prio=0 tid=0x00007fd78807c000
nid=0x58b waiting on condition [0x00007fd7a02e7000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000708503368> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- None
"data-plane-kafka-network-thread-0-ListenerName(PLAINTEXT)-PLAINTEXT-2" #63
prio=5 os_prio=0 tid=0x00007fd919cf8800 nid=0x58a runnable [0x00007fd7a05e8000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.NativeThread.current(Native Method)
at sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:46)
at sun.nio.ch.FileChannelImpl.size(FileChannelImpl.java:306)
- locked <0x0000000709433618> (a java.lang.Object)
at
org.apache.kafka.common.record.FileRecords.writeTo(FileRecords.java:271)
at
org.apache.kafka.common.record.DefaultRecordsSend.writeTo(DefaultRecordsSend.java:33)
at
org.apache.kafka.common.record.RecordsSend.writeTo(RecordsSend.java:58)
at
org.apache.kafka.common.record.MultiRecordsSend.writeTo(MultiRecordsSend.java:93)
at
org.apache.kafka.common.network.KafkaChannel.write(KafkaChannel.java:426)
at org.apache.kafka.common.network.Selector.write(Selector.java:648)
at
org.apache.kafka.common.network.Selector.attemptWrite(Selector.java:641)
at
org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:597)
at org.apache.kafka.common.network.Selector.poll(Selector.java:485)
at kafka.network.Processor.poll(SocketServer.scala:861)
at kafka.network.Processor.run(SocketServer.scala:760)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- None
"data-plane-kafka-network-thread-0-ListenerName(PLAINTEXT)-PLAINTEXT-1" #62
prio=5 os_prio=0 tid=0x00007fd919cf6800 nid=0x589 runnable [0x00007fd7a06e9000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.FileDispatcherImpl.writev0(Native Method)
at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:51)
at sun.nio.ch.IOUtil.write(IOUtil.java:148)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:501)
- locked <0x000000074cdb2cf8> (a java.lang.Object)
at java.nio.channels.SocketChannel.write(SocketChannel.java:502)
at
org.apache.kafka.common.network.PlaintextTransportLayer.write(PlaintextTransportLayer.java:152)
at
org.apache.kafka.common.network.ByteBufferSend.writeTo(ByteBufferSend.java:60)
at
org.apache.kafka.common.network.KafkaChannel.write(KafkaChannel.java:426)
at org.apache.kafka.common.network.Selector.write(Selector.java:648)
at
org.apache.kafka.common.network.Selector.attemptWrite(Selector.java:641)
at
org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:597)
at org.apache.kafka.common.network.Selector.poll(Selector.java:485)
at kafka.network.Processor.poll(SocketServer.scala:861)
at kafka.network.Processor.run(SocketServer.scala:760)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- None
"data-plane-kafka-network-thread-0-ListenerName(PLAINTEXT)-PLAINTEXT-0" #61
prio=5 os_prio=0 tid=0x00007fd9191fd000 nid=0x588 runnable [0x00007fd7a07ea000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.FileDispatcherImpl.writev0(Native Method)
at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:51)
at sun.nio.ch.IOUtil.write(IOUtil.java:148)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:501)
- locked <0x000000074d1195f8> (a java.lang.Object)
at java.nio.channels.SocketChannel.write(SocketChannel.java:502)
at
org.apache.kafka.common.network.PlaintextTransportLayer.write(PlaintextTransportLayer.java:152)
at
org.apache.kafka.common.network.ByteBufferSend.writeTo(ByteBufferSend.java:60)
at
org.apache.kafka.common.network.KafkaChannel.write(KafkaChannel.java:426)
at org.apache.kafka.common.network.Selector.write(Selector.java:648)
at
org.apache.kafka.common.network.Selector.attemptWrite(Selector.java:641)
at
org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:597)
at org.apache.kafka.common.network.Selector.poll(Selector.java:485)
at kafka.network.Processor.poll(SocketServer.scala:861)
at kafka.network.Processor.run(SocketServer.scala:760)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- None
"Controller-0-to-broker-0-send-thread" #60 prio=5 os_prio=0
tid=0x00007fd78803b000 nid=0x587 waiting on condition [0x00007fd7a08eb000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007095a86a0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:232)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)
Locked ownable synchronizers:
- None
"/config/changes-event-process-thread" #59 prio=5 os_prio=0
tid=0x00007fd91af19000 nid=0x586 waiting on condition [0x00007fd7a0bec000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007095a88a0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
kafka.common.ZkNodeChangeNotificationListener$ChangeEventProcessThread.doWork(ZkNodeChangeNotificationListener.scala:146)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)
Locked ownable synchronizers:
- None
"data-plane-kafka-request-handler-7" #58 daemon prio=5 os_prio=0
tid=0x00007fd91af09800 nid=0x585 runnable [0x00007fd7a0ced000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007095a8aa8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
at kafka.network.RequestChannel.receiveRequest(RequestChannel.scala:339)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:55)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- None
"data-plane-kafka-request-handler-6" #57 daemon prio=5 os_prio=0
tid=0x00007fd91af07800 nid=0x584 runnable [0x00007fd7a0dee000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007095a8aa8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
at kafka.network.RequestChannel.receiveRequest(RequestChannel.scala:339)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:55)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- None
"data-plane-kafka-request-handler-5" #56 daemon prio=5 os_prio=0
tid=0x00007fd91af06000 nid=0x583 runnable [0x00007fd7a0eef000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007095a8aa8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
at kafka.network.RequestChannel.receiveRequest(RequestChannel.scala:339)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:55)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- None
"data-plane-kafka-request-handler-4" #55 daemon prio=5 os_prio=0
tid=0x00007fd91af04000 nid=0x582 runnable [0x00007fd7a0ff0000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007095a8aa8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
at kafka.network.RequestChannel.receiveRequest(RequestChannel.scala:339)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:55)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- None
"data-plane-kafka-request-handler-3" #54 daemon prio=5 os_prio=0
tid=0x00007fd91af02000 nid=0x581 runnable [0x00007fd7a10f1000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007095a8aa8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
at kafka.network.RequestChannel.receiveRequest(RequestChannel.scala:339)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:55)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- None
"data-plane-kafka-request-handler-2" #53 daemon prio=5 os_prio=0
tid=0x00007fd91af00000 nid=0x580 runnable [0x00007fd7a11f2000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007095a8aa8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
at kafka.network.RequestChannel.receiveRequest(RequestChannel.scala:339)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:55)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- None
"data-plane-kafka-request-handler-1" #52 daemon prio=5 os_prio=0
tid=0x00007fd91aef8800 nid=0x57f runnable [0x00007fd7a12f3000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007095a8aa8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
at kafka.network.RequestChannel.receiveRequest(RequestChannel.scala:339)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:55)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- None
"data-plane-kafka-request-handler-0" #51 daemon prio=5 os_prio=0
tid=0x00007fd91aefb000 nid=0x57e runnable [0x00007fd7a13f4000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007095a8aa8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:418)
at kafka.network.RequestChannel.receiveRequest(RequestChannel.scala:339)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:55)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- None
"ExpirationReaper-0-AlterAcls" #50 prio=5 os_prio=0 tid=0x00007fd91aef3000
nid=0x57d runnable [0x00007fd7a14f5000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007095ad2e0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.DelayQueue.poll(DelayQueue.java:259)
at kafka.utils.timer.SystemTimer.advanceClock(Timer.scala:103)
at
kafka.server.DelayedOperationPurgatory.advanceClock(DelayedOperation.scala:423)
at
kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper.doWork(DelayedOperation.scala:449)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)
Locked ownable synchronizers:
- None
"TxnMarkerSenderThread-0" #48 prio=5 os_prio=0 tid=0x00007fd91aedc000 nid=0x57c
runnable [0x00007fd7a15f6000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000007095ab330> (a sun.nio.ch.Util$3)
- locked <0x00000007095ab340> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000007095ab2e8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at org.apache.kafka.common.network.Selector.select(Selector.java:847)
at org.apache.kafka.common.network.Selector.poll(Selector.java:469)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:549)
at
kafka.common.InterBrokerSendThread.doWork(InterBrokerSendThread.scala:66)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)
Locked ownable synchronizers:
- None
"transaction-log-manager-0" #49 daemon prio=5 os_prio=0 tid=0x00007fd91aeda000
nid=0x57b waiting on condition [0x00007fd7a16f7000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000070959cc88> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- None
"group-metadata-manager-0" #46 daemon prio=5 os_prio=0 tid=0x00007fd91ae88000
nid=0x57a waiting on condition [0x00007fd7a17f8000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000070959cd50> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- None
"ExpirationReaper-0-Rebalance" #45 prio=5 os_prio=0 tid=0x00007fd91ae45000
nid=0x579 runnable [0x00007fd7a1af9000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007095a90f8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.DelayQueue.poll(DelayQueue.java:268)
at kafka.utils.timer.SystemTimer.advanceClock(Timer.scala:103)
at
kafka.server.DelayedOperationPurgatory.advanceClock(DelayedOperation.scala:423)
at
kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper.doWork(DelayedOperation.scala:449)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)
Locked ownable synchronizers:
- None
"ExpirationReaper-0-Heartbeat" #44 prio=5 os_prio=0 tid=0x00007fd91ae44000
nid=0x578 runnable [0x00007fd7a1bfa000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007095ab5c8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.DelayQueue.poll(DelayQueue.java:268)
at kafka.utils.timer.SystemTimer.advanceClock(Timer.scala:103)
at
kafka.server.DelayedOperationPurgatory.advanceClock(DelayedOperation.scala:423)
at
kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper.doWork(DelayedOperation.scala:449)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)
Locked ownable synchronizers:
- None
"ExpirationReaper-0-topic" #43 prio=5 os_prio=0 tid=0x00007fd91ae3d800
nid=0x577 runnable [0x00007fd7a1cfb000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007095a9340> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.DelayQueue.poll(DelayQueue.java:259)
at kafka.utils.timer.SystemTimer.advanceClock(Timer.scala:103)
at
kafka.server.DelayedOperationPurgatory.advanceClock(DelayedOperation.scala:423)
at
kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper.doWork(DelayedOperation.scala:449)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)
Locked ownable synchronizers:
- None
"controller-event-thread" #42 prio=5 os_prio=0 tid=0x00007fd91ae3c800 nid=0x576
waiting on condition [0x00007fd7a1dfc000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007095a9588> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at
kafka.controller.ControllerEventManager$ControllerEventThread.doWork(ControllerEventManager.scala:119)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)
Locked ownable synchronizers:
- None
"LogDirFailureHandler" #41 prio=5 os_prio=0 tid=0x00007fd91adb6000 nid=0x575
waiting on condition [0x00007fd7a1efd000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007091001e8> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:403)
at
kafka.server.LogDirFailureChannel.takeNextOfflineLogDir(LogDirFailureChannel.scala:56)
at
kafka.server.ReplicaManager$LogDirFailureHandler.doWork(ReplicaManager.scala:225)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)
Locked ownable synchronizers:
- None
"ExpirationReaper-0-ElectLeader" #40 prio=5 os_prio=0 tid=0x00007fd918fa9000
nid=0x574 runnable [0x00007fd7a21fe000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007095ad390> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.DelayQueue.poll(DelayQueue.java:259)
at kafka.utils.timer.SystemTimer.advanceClock(Timer.scala:103)
at
kafka.server.DelayedOperationPurgatory.advanceClock(DelayedOperation.scala:423)
at
kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper.doWork(DelayedOperation.scala:449)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)
Locked ownable synchronizers:
- None
"ExpirationReaper-0-DeleteRecords" #39 prio=5 os_prio=0 tid=0x00007fd918ef0000
nid=0x573 runnable [0x00007fd8581fc000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007095a95f0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.DelayQueue.poll(DelayQueue.java:259)
at kafka.utils.timer.SystemTimer.advanceClock(Timer.scala:103)
at
kafka.server.DelayedOperationPurgatory.advanceClock(DelayedOperation.scala:423)
at
kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper.doWork(DelayedOperation.scala:449)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)
Locked ownable synchronizers:
- None
"ExpirationReaper-0-Fetch" #38 prio=5 os_prio=0 tid=0x00007fd918eee800
nid=0x572 runnable [0x00007fd8582fd000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007095ad790> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.DelayQueue.poll(DelayQueue.java:259)
at kafka.utils.timer.SystemTimer.advanceClock(Timer.scala:103)
at
kafka.server.DelayedOperationPurgatory.advanceClock(DelayedOperation.scala:423)
at
kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper.doWork(DelayedOperation.scala:449)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)
Locked ownable synchronizers:
- None
"ExpirationReaper-0-Produce" #37 prio=5 os_prio=0 tid=0x00007fd918eed000
nid=0x571 runnable [0x00007fd8583fe000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007095ac240> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.DelayQueue.poll(DelayQueue.java:259)
at kafka.utils.timer.SystemTimer.advanceClock(Timer.scala:103)
at
kafka.server.DelayedOperationPurgatory.advanceClock(DelayedOperation.scala:423)
at
kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper.doWork(DelayedOperation.scala:449)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)
Locked ownable synchronizers:
- None
"data-plane-kafka-socket-acceptor-ListenerName(PLAINTEXT)-PLAINTEXT-9092" #36
prio=5 os_prio=0 tid=0x00007fd918f73800 nid=0x570 runnable [0x00007fd8681fc000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x00000007095ac500> (a sun.nio.ch.Util$3)
- locked <0x00000007095ac510> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000007095ac4b8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at kafka.network.Acceptor.run(SocketServer.scala:534)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- None
"kafka-log-cleaner-thread-0" #35 prio=5 os_prio=0 tid=0x00007fd91aca6800
nid=0x56f waiting on condition [0x00007fd8d523e000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000709100400> (a
java.util.concurrent.CountDownLatch$Sync)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedNanos(AbstractQueuedSynchronizer.java:1037)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireSharedNanos(AbstractQueuedSynchronizer.java:1328)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:277)
at kafka.utils.ShutdownableThread.pause(ShutdownableThread.scala:82)
at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:305)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)
Locked ownable synchronizers:
- None
"kafka-scheduler-9" #34 daemon prio=5 os_prio=0 tid=0x00007fd880098800
nid=0x56e waiting on condition [0x00007fd8682fd000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007091005f0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- None
"kafka-scheduler-8" #33 daemon prio=5 os_prio=0 tid=0x00007fd880096800
nid=0x56d waiting on condition [0x00007fd8683fe000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007091005f0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- None
"kafka-scheduler-7" #32 daemon prio=5 os_prio=0 tid=0x00007fd880095000
nid=0x56c waiting on condition [0x00007fd8781fc000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007091005f0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- None
"kafka-scheduler-6" #31 daemon prio=5 os_prio=0 tid=0x00007fd880093800
nid=0x56b waiting on condition [0x00007fd8782fd000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007091005f0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- None
"kafka-scheduler-5" #30 daemon prio=5 os_prio=0 tid=0x00007fd880091800
nid=0x56a waiting on condition [0x00007fd8783fe000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007091005f0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- None
"kafka-scheduler-4" #29 daemon prio=5 os_prio=0 tid=0x00007fd88008f800
nid=0x569 waiting on condition [0x00007fd8d4115000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007091005f0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- None
"kafka-scheduler-3" #28 daemon prio=5 os_prio=0 tid=0x00007fd88008e000
nid=0x568 waiting on condition [0x00007fd8d4217000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007091005f0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- None
"kafka-scheduler-2" #27 daemon prio=5 os_prio=0 tid=0x00007fd880085000
nid=0x567 waiting on condition [0x00007fd8d431a000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007091005f0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- None
"kafka-scheduler-1" #26 daemon prio=5 os_prio=0 tid=0x00007fd880083000
nid=0x566 waiting on condition [0x00007fd8d443b000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007091005f0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- None
"kafka-scheduler-0" #25 daemon prio=5 os_prio=0 tid=0x00007fd880081800
nid=0x565 waiting on condition [0x00007fd8d453d000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007091005f0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- None
"ThrottledChannelReaper-Request" #23 prio=5 os_prio=0 tid=0x00007fd91ac51000
nid=0x563 runnable [0x00007fd8d533f000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000709101780> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.DelayQueue.poll(DelayQueue.java:259)
at
kafka.server.ClientQuotaManager$ThrottledChannelReaper.doWork(ClientQuotaManager.scala:196)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)
Locked ownable synchronizers:
- None
"ThrottledChannelReaper-Produce" #22 prio=5 os_prio=0 tid=0x00007fd91ac4f800
nid=0x562 runnable [0x00007fd8d5440000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000709101960> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.DelayQueue.poll(DelayQueue.java:259)
at
kafka.server.ClientQuotaManager$ThrottledChannelReaper.doWork(ClientQuotaManager.scala:196)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)
Locked ownable synchronizers:
- None
"ThrottledChannelReaper-Fetch" #21 prio=5 os_prio=0 tid=0x00007fd91ac4c800
nid=0x561 runnable [0x00007fd8d5541000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000709101b40> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at java.util.concurrent.DelayQueue.poll(DelayQueue.java:259)
at
kafka.server.ClientQuotaManager$ThrottledChannelReaper.doWork(ClientQuotaManager.scala:196)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)
Locked ownable synchronizers:
- None
"SensorExpiryThread" #20 daemon prio=5 os_prio=0 tid=0x00007fd91ac35800
nid=0x560 waiting on condition [0x00007fd8d5642000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000709101d08> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- None
"main-EventThread" #19 daemon prio=5 os_prio=0 tid=0x00007fd91aa92800 nid=0x55f
waiting on condition [0x00007fd8d5d67000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000708725b78> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:506)
Locked ownable synchronizers:
- None
"main-SendThread(zk-hs:2181)" #18 daemon prio=5 os_prio=0
tid=0x00007fd91aa96800 nid=0x55e runnable [0x00007fd8d5e68000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x0000000708729b38> (a sun.nio.ch.Util$3)
- locked <0x0000000708729b28> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000708729ae0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at
org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:345)
at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1223)
Locked ownable synchronizers:
- None
"metrics-meter-tick-thread-2" #17 daemon prio=5 os_prio=0
tid=0x00007fd91aa5a000 nid=0x55d waiting on condition [0x00007fd8d5f69000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000708725be0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- None
"metrics-meter-tick-thread-1" #16 daemon prio=5 os_prio=0
tid=0x00007fd91aa51800 nid=0x55c runnable [0x00007fd8d606a000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x0000000708725be0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
at
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
at
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- None
"Service Thread" #13 daemon prio=9 os_prio=0 tid=0x00007fd91917e000 nid=0x55a
runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C1 CompilerThread2" #12 daemon prio=9 os_prio=0 tid=0x00007fd91917b000
nid=0x559 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread1" #11 daemon prio=9 os_prio=0 tid=0x00007fd919179800
nid=0x558 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C2 CompilerThread0" #10 daemon prio=9 os_prio=0 tid=0x00007fd919178800
nid=0x557 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Thread-2" #9 daemon prio=5 os_prio=0 tid=0x00007fd8b8001000 nid=0x556 runnable
[0x00007fd8d6b6f000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0x0000000708731e48> (a sun.nio.ch.Util$3)
- locked <0x0000000708731e38> (a java.util.Collections$UnmodifiableSet)
- locked <0x0000000708731df0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at sun.net.httpserver.ServerImpl$Dispatcher.run(ServerImpl.java:352)
at java.lang.Thread.run(Thread.java:748)
Locked ownable synchronizers:
- None
"server-timer" #7 daemon prio=5 os_prio=0 tid=0x00007fd9191ac000 nid=0x554 in
Object.wait() [0x00007fd8d6d71000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x0000000708731300> (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:552)
- locked <0x0000000708731300> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)
Locked ownable synchronizers:
- None
"Signal Dispatcher" #5 daemon prio=9 os_prio=0 tid=0x00007fd918c8c000 nid=0x553
runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Surrogate Locker Thread (Concurrent GC)" #4 daemon prio=9 os_prio=0
tid=0x00007fd918c7a000 nid=0x552 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007fd918c59000 nid=0x551 in
Object.wait() [0x00007fd9082ab000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000007087312f0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
- locked <0x00000007087312f0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:216)
Locked ownable synchronizers:
- None
"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007fd918c54800
nid=0x550 in Object.wait() [0x00007fd9083ac000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x0000000708729c50> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
- locked <0x0000000708729c50> (a java.lang.ref.Reference$Lock)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
Locked ownable synchronizers:
- None
"main" #1 prio=5 os_prio=0 tid=0x00007fd91800f800 nid=0x543 waiting on
condition [0x00007fd91f3ca000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x00000007095ad840> (a
java.util.concurrent.CountDownLatch$Sync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
at kafka.server.KafkaServer.awaitShutdown(KafkaServer.scala:688)
at
kafka.server.KafkaServerStartable.awaitShutdown(KafkaServerStartable.scala:71)
at kafka.Kafka$.main(Kafka.scala:83)
at kafka.Kafka.main(Kafka.scala)
Locked ownable synchronizers:
- None
"VM Thread" os_prio=0 tid=0x00007fd918c4b000 nid=0x54f runnable
"Gang worker#0 (Parallel GC Threads)" os_prio=0 tid=0x00007fd918024000
nid=0x544 runnable
"Gang worker#1 (Parallel GC Threads)" os_prio=0 tid=0x00007fd918025800
nid=0x545 runnable
"Gang worker#2 (Parallel GC Threads)" os_prio=0 tid=0x00007fd918027000
nid=0x546 runnable
"Gang worker#3 (Parallel GC Threads)" os_prio=0 tid=0x00007fd918029000
nid=0x547 runnable
"G1 Main Concurrent Mark GC Thread" os_prio=0 tid=0x00007fd91805d000 nid=0x54d
runnable
"Gang worker#0 (G1 Parallel Marking Threads)" os_prio=0 tid=0x00007fd91805e800
nid=0x54e runnable
"G1 Concurrent Refinement Thread#0" os_prio=0 tid=0x00007fd918032800 nid=0x54c
runnable
"G1 Concurrent Refinement Thread#1" os_prio=0 tid=0x00007fd918030800 nid=0x54b
runnable
"G1 Concurrent Refinement Thread#2" os_prio=0 tid=0x00007fd91802f000 nid=0x54a
runnable
"G1 Concurrent Refinement Thread#3" os_prio=0 tid=0x00007fd91802d000 nid=0x549
runnable
"G1 Concurrent Refinement Thread#4" os_prio=0 tid=0x00007fd91802b800 nid=0x548
runnable
"VM Periodic Task Thread" os_prio=0 tid=0x00007fd919180800 nid=0x55b waiting on
condition
JNI global references: 2420
