I'm using Kafka 0.8.1.1, have 10 nodes in a cluster, all are started with default command: ./bin/kafka-server-start.sh conf/server.properties
but yesterday, I found that two nodes kept full GC(every 1-3 seconds), although there was still enough memory: 99546.435: [GC [1 CMS-initial-mark: 532714K(699072K)] 538916K(1013632K), 0.0099270 secs] [Times: user=0.01 sys=0.00, real=0.01 secs] 99546.446: [CMS-concurrent-mark-start] 99546.520: [CMS-concurrent-mark: 0.074/0.074 secs] [Times: user=0.21 sys=0.03, real=0.08 secs] 99546.520: [CMS-concurrent-preclean-start] 99546.525: [CMS-concurrent-preclean: 0.005/0.005 secs] [Times: user=0.01 sys=0.00, real=0.00 secs] 99546.525: [CMS-concurrent-abortable-preclean-start] 99547.348: [CMS-concurrent-abortable-preclean: 0.822/0.823 secs] [Times: user=1.53 sys=0.31, real=0.83 secs] 99547.350: [GC[YG occupancy: 158101 K (314560 K)]2016-03-12T20:19:58.597+0800: 99547.350: [GC 99547.350: [ParNew: 158101K->5170K(314560K), 0.0189700 secs] 690816K->538498K(1013632K), 0.0190720 secs] [Times: user=0.11 sys=0.00, real=0.02 secs] 99547.369: [Rescan (parallel) , 0.0099240 secs]99547.379: [weak refs processing, 0.0000090 secs]99547.379: [class unloading, 0.0028860 secs]99547.382: [scrub symbol table, 0.0015400 secs]99547.383: [scrub string table, 0.0001640 secs] [1 CMS-remark: 533327K(699072K)] 538498K(1013632K), 0.0353890 secs] [Times: user=0.18 sys=0.00, real=0.03 secs] 99547.386: [CMS-concurrent-sweep-start] 99547.421: [CMS-concurrent-sweep: 0.035/0.035 secs] [Times: user=0.08 sys=0.01, real=0.04 secs] 99547.421: [CMS-concurrent-reset-start] 99547.426: [CMS-concurrent-reset: 0.005/0.005 secs] [Times: user=0.01 sys=0.01, real=0.00 secs] 99549.132: [GC 99549.132: [ParNew: 284786K->4184K(314560K), 0.0158450 secs] 816880K->536891K(1013632K), 0.0159570 secs] [Times: user=0.08 sys=0.00, real=0.01 secs] is this caused by small head(1G by default), or consumer lag? I tried to use jmap to check the heap, but failed to attach to the java process. how to find the root cause? -- Don't schedule every day, make them disorder.
