Thanks. I managed to get a cpu dump from staging.
The output:
THREAD START (obj=50000427, id = 200004, name="RMI TCP Accept-0",
group="system")
THREAD START (obj=50000427, id = 200001, name="main", group="main")
THREAD START (obj=50000427, id = 200005, name="SensorExpiryThread",
group="main")
THREAD START (obj=500008e6, id = 200006,
name="ThrottledRequestReaper-Fetch", group="main")
THREAD START (obj=500008e6, id = 200007,
name="ThrottledRequestReaper-Produce", group="main")
THREAD START (obj=50000914, id = 200008,
name="ZkClient-EventThread-18-zookeeper:2181", group="main")
THREAD START (obj=500008e6, id = 200009, name="main-SendThread()",
group="main")
THREAD START (obj=50000950, id = 200010, name="main-EventThread",
group="main")
THREAD START (obj=50000427, id = 200011, name="pool-3-thread-1",
group="main")
THREAD END (id = 200011)
THREAD START (obj=50000427, id = 200012,
name="metrics-meter-tick-thread-1", group="main")
THREAD START (obj=50000427, id = 200014, name="kafka-scheduler-0",
group="main")
THREAD START (obj=50000427, id = 200013, name="kafka-scheduler-1",
group="main")
THREAD START (obj=50000427, id = 200015, name="kafka-scheduler-2",
group="main")
THREAD START (obj=50000c33, id = 200016, name="kafka-log-cleaner-thread-0",
group="main")
THREAD START (obj=50000427, id = 200017,
name="kafka-network-thread-2-PLAINTEXT-0", group="main")
THREAD START (obj=50000427, id = 200018,
name="kafka-network-thread-2-PLAINTEXT-1", group="main")
THREAD START (obj=50000427, id = 200019,
name="kafka-network-thread-2-PLAINTEXT-2", group="main")
THREAD START (obj=50000427, id = 200020,
name="kafka-socket-acceptor-PLAINTEXT-9092", group="main")
THREAD START (obj=500008e6, id = 200021, name="ExpirationReaper-2",
group="main")
THREAD START (obj=500008e6, id = 200022, name="ExpirationReaper-2",
group="main")
THREAD START (obj=50000427, id = 200023,
name="metrics-meter-tick-thread-2", group="main")
THREAD START (obj=50000427, id = 200024, name="kafka-scheduler-3",
group="main")
THREAD START (obj=50000427, id = 200025, name="kafka-scheduler-4",
group="main")
THREAD START (obj=50000427, id = 200026, name="kafka-scheduler-5",
group="main")
THREAD START (obj=50000427, id = 200027, name="kafka-scheduler-6",
group="main")
THREAD START (obj=500008e6, id = 200028, name="ExpirationReaper-2",
group="main")
THREAD START (obj=500008e6, id = 200029, name="ExpirationReaper-2",
group="main")
THREAD START (obj=500008e6, id = 200030, name="ExpirationReaper-2",
group="main")
THREAD START (obj=50000427, id = 200031, name="group-metadata-manager-0",
group="main")
THREAD START (obj=50000427, id = 200032, name="kafka-request-handler-0",
group="main")
THREAD START (obj=50000427, id = 200037, name="kafka-request-handler-5",
group="main")
THREAD START (obj=50000427, id = 200036, name="kafka-request-handler-4",
group="main")
THREAD START (obj=50000427, id = 200035, name="kafka-request-handler-3",
group="main")
THREAD START (obj=50000427, id = 200034, name="kafka-request-handler-2",
group="main")
THREAD START (obj=50000427, id = 200033, name="kafka-request-handler-1",
group="main")
THREAD START (obj=50000427, id = 200038, name="kafka-request-handler-6",
group="main")
THREAD START (obj=50000427, id = 200039, name="kafka-request-handler-7",
group="main")
THREAD START (obj=50000427, id = 200040, name="kafka-scheduler-7",
group="main")
THREAD START (obj=50000427, id = 200041, name="kafka-scheduler-8",
group="main")
THREAD START (obj=50000ee2, id = 200042, name="ReplicaFetcherThread-0-0",
group="main")
THREAD START (obj=50000ee2, id = 200043, name="ReplicaFetcherThread-0-1",
group="main")
THREAD START (obj=50000427, id = 200044, name="kafka-scheduler-9",
group="main")
THREAD START (obj=50000427, id = 200045, name="executor-Fetch",
group="main")
TRACE 300920:
sun.nio.ch.EPollArrayWrapper.epollWait(EPollArrayWrapper.java:Unknown line)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
TRACE 300518:
java.net.PlainSocketImpl.socketAccept(PlainSocketImpl.java:Unknown line)
java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
java.net.ServerSocket.implAccept(ServerSocket.java:545)
java.net.ServerSocket.accept(ServerSocket.java:513)
TRACE 300940:
sun.nio.ch.FileDispatcherImpl.write0(FileDispatcherImpl.java:Unknown line)
sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
sun.nio.ch.IOUtil.write(IOUtil.java:65)
TRACE 301003:
org.xerial.snappy.SnappyNative.rawUncompress(SnappyNative.java:Unknown line)
org.xerial.snappy.Snappy.rawUncompress(Snappy.java:474)
org.xerial.snappy.Snappy.uncompress(Snappy.java:513)
org.xerial.snappy.SnappyInputStream.readFully(SnappyInputStream.java:147)
TRACE 300979:
sun.nio.ch.FileDispatcherImpl.pread0(FileDispatcherImpl.java:Unknown line)
sun.nio.ch.FileDispatcherImpl.pread(FileDispatcherImpl.java:52)
sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:220)
sun.nio.ch.IOUtil.read(IOUtil.java:197)
TRACE 301630:
sun.nio.ch.EPollArrayWrapper.epollCtl(EPollArrayWrapper.java:Unknown line)
sun.nio.ch.EPollArrayWrapper.updateRegistrations(EPollArrayWrapper.java:299)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:268)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
TRACE 301259:
sun.misc.Unsafe.unpark(Unsafe.java:Unknown line)
java.util.concurrent.locks.LockSupport.unpark(LockSupport.java:141)
java.util.concurrent.locks.AbstractQueuedSynchronizer.unparkSuccessor(AbstractQueuedSynchronizer.java:662)
java.util.concurrent.locks.AbstractQueuedSynchronizer.release(AbstractQueuedSynchronizer.java:1264)
TRACE 301559:
sun.nio.ch.FileDispatcherImpl.read0(FileDispatcherImpl.java:Unknown line)
sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
sun.nio.ch.IOUtil.read(IOUtil.java:197)
TRACE 300590:
java.lang.ClassLoader.defineClass1(ClassLoader.java:Unknown line)
java.lang.ClassLoader.defineClass(ClassLoader.java:763)
java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
TRACE 301643:
scala.Tuple2.equals(Tuple2.scala:20)
java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:940)
kafka.utils.Pool.get(Pool.scala:69)
kafka.server.ReplicaManager.getPartition(ReplicaManager.scala:280)
TRACE 300592:
java.util.zip.ZipFile.read(ZipFile.java:Unknown line)
java.util.zip.ZipFile.access$1400(ZipFile.java:60)
java.util.zip.ZipFile$ZipFileInputStream.read(ZipFile.java:717)
java.util.zip.ZipFile$ZipFileInflaterInputStream.fill(ZipFile.java:419)
TRACE 301018:
kafka.utils.CoreUtils$.crc32(CoreUtils.scala:148)
kafka.message.Message.computeChecksum(Message.scala:216)
kafka.message.Message.isValid(Message.scala:226)
kafka.message.Message.ensureValid(Message.scala:232)
TRACE 301561:
java.io.FileDescriptor.sync(FileDescriptor.java:Unknown line)
kafka.server.OffsetCheckpoint.liftedTree1$1(OffsetCheckpoint.scala:62)
kafka.server.OffsetCheckpoint.write(OffsetCheckpoint.scala:49)
kafka.server.ReplicaManager$$anonfun$checkpointHighWatermarks$2.apply(ReplicaManager.scala:945)
TRACE 301422:
java.util.Arrays.copyOf(Arrays.java:3332)
java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:137)
java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:121)
java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:421)
CPU SAMPLES BEGIN (total = 699214) Thu Mar 23 12:41:17 2017
rank self accum count trace method
1 86.46% 86.46% 604544 300920 sun.nio.ch.EPollArrayWrapper.epollWait
2 12.62% 99.08% 88254 300518 java.net.PlainSocketImpl.socketAccept
3 0.11% 99.19% 759 300940 sun.nio.ch.FileDispatcherImpl.write0
4 0.04% 99.23% 253 301003
org.xerial.snappy.SnappyNative.rawUncompress
5 0.03% 99.26% 231 300979 sun.nio.ch.FileDispatcherImpl.pread0
6 0.03% 99.29% 220 301630 sun.nio.ch.EPollArrayWrapper.epollCtl
7 0.03% 99.32% 219 301259 sun.misc.Unsafe.unpark
8 0.02% 99.34% 145 301559 sun.nio.ch.FileDispatcherImpl.read0
9 0.01% 99.36% 89 300590 java.lang.ClassLoader.defineClass1
10 0.01% 99.37% 87 301643 scala.Tuple2.equals
11 0.01% 99.38% 79 300592 java.util.zip.ZipFile.read
12 0.01% 99.39% 79 301018 kafka.utils.CoreUtils$.crc32
13 0.01% 99.40% 78 301561 java.io.FileDescriptor.sync
14 0.01% 99.41% 72 301422 java.util.Arrays.copyOf
CPU SAMPLES END
It seems like the constant disconnects is far bigger then the 10 minutes
default. I suspect this has something to do with double connects, which I'm
not sure to get around.
On Thu, Mar 23, 2017 at 11:46 AM, Manikumar <manikumar.re...@gmail.com>
wrote:
1. may be you can monitor thread wise cpu usage and correlate with thread
dump
to identify the bottleneck
2. Broker config property connections.max.idle.ms is used to close
idle connections.
default is 10min.
On Thu, Mar 23, 2017 at 3:55 PM, Paul van der Linden <p...@sportr.co.uk>
wrote:
Hi,
I deployed Kafka about a week ago, but there are a few problems with how
Kafka behaves.
The first is the surprisingly high resource usage, one this the memory
(1.5-2 GB for each broker, 3 brokers) although this might be normal. The
other one is the cpu usage, which starts with 20% minimum on each broker,
which I find strange with the current throughput (which is < 1 msg/s).
This might has something to do with something else which I find strange,
Kafka disconnects clients about every 10-20 minutes per broker. It might
have something to do with the configuration: Deployed in kubernetes,
bootstrapping with a single dns name (which is backed by all alive kafka
brokers), and then every broker has a separate dns address which is used
after the bootstrap. This means that a client is connected twice to one
of
the brokers. The reason for the bootstrap dns name is to make sure I
don't
have to update all clients to include other brokers.
Any advice on how to solve these 2 problems?
Thanks,
Paul
On Tue, Mar 21, 2017 at 10:30 AM, Paul van der Linden <p...@sportr.co.uk
wrote:
Hi,
I deployed Kafka about a week ago, but there are a few problems with
how
Kafka behaves.
The first is the surprisingly high resource usage, one this the memory
(1.5-2 GB for each broker, 3 brokers) although this might be normal.
The
other one is the cpu usage, which starts with 20% minimum on each
broker,
which I find strange with the current throughput (which is < 1 msg/s).
This might has something to do with something else which I find
strange,
Kafka disconnects clients about every 10-20 minutes per broker. It
might
have something to do with the configuration: Deployed in kubernetes,
bootstrapping with a single dns name (which is backed by all alive
kafka
brokers), and then every broker has a separate dns address which is
used
after the bootstrap. This means that a client is connected twice to one
of
the brokers. The reason for the bootstrap dns name is to make sure I
don't
have to update all clients to include other brokers.
Any advice on how to solve these 2 problems?
Thanks,
Paul
