[
https://issues.apache.org/jira/browse/KAFKA-1024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13751584#comment-13751584
]
Jason Toffaletti commented on KAFKA-1024:
-----------------------------------------
Process has been up for 19 hours now without running out of memory at Xmx 128MB
and queue.buffering.max.messages 1000 so I think your diagnosis of "not enough
memory" because of the queue length was correct. My average message size is 2KB
so even at 10,000 messages I didn't expect it to use more than 20MB. I'd like
to understand what side effects reducing the queue length has and why 10000 is
the default. Does it mean that if the queue is full and queue.buffering.max.ms
is reached waiting to insert to the queue, I'll see dropped messages?
> possible memory leak in 0.8 beta1 producer with G1GC
> ----------------------------------------------------
>
> Key: KAFKA-1024
> URL: https://issues.apache.org/jira/browse/KAFKA-1024
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8
> Environment: java version "1.7.0_17"
> Java(TM) SE Runtime Environment (build 1.7.0_17-b02)
> Java HotSpot(TM) 64-Bit Server VM (build 23.7-b01, mixed mode)
> Reporter: Jason Toffaletti
>
> I have this in my pom.xml
> <dependency>
> <groupId>org.apache.kafka</groupId>
> <artifactId>kafka_2.9.2</artifactId>
> <version>0.8.0-beta1</version>
> </dependency>
> <dependency>
> <groupId>org.scala-lang</groupId>
> <artifactId>scala-library</artifactId>
> <version>2.9.2</version>
> </dependency>
> <dependency>
> <groupId>org.xerial.snappy</groupId>
> <artifactId>snappy-java</artifactId>
> <version>1.0.4.1</version>
> </dependency>
> I'm using snappy compression codec and producing about 7k msg/sec on average.
> I'm producing batches up to 10 messages per batch with null keys to a topic
> with 16 partitions, no replication. Xms and Xmx are 64MB and I'm using
> XX:+UseG1GC. After about 12 hours of operation heap usage hits right up
> against the 64MB limit and the producer drops to about 4k msg/sec because of
> the GC pressure. When I restart the process the heap usage goes back to
> normal (around 30MB) and the producer does 7k msg/sec again.
> What else can I provide to help debug this?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
