[
https://issues.apache.org/jira/browse/KAFKA-1024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13751723#comment-13751723
]
Jay Kreps commented on KAFKA-1024:
----------------------------------
Your serialized message size is 2KB, perhaps but I wonder if this is true of
the java objects themselves. If these are compound objects with lots of little
subobjects that can add a ton of overhead.
Yes when you reach the max queue length you can either block the sender or drop
the message. That timeout controls the time until data will be dropped (-1 is
never drop).
One downside of counting the buffer in terms of messages is that most people
don't really have a good idea of their message sizes. So it is hard to say if
10k is too big or too small. In a future version we may improve the client to
buffer messages already serialized so we can give the bound in terms of memory
space, which I think would be more intuitive.
> possible memory leak in 0.8 beta1 producer with G1GC
> ----------------------------------------------------
>
> Key: KAFKA-1024
> URL: https://issues.apache.org/jira/browse/KAFKA-1024
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.8
> Environment: java version "1.7.0_17"
> Java(TM) SE Runtime Environment (build 1.7.0_17-b02)
> Java HotSpot(TM) 64-Bit Server VM (build 23.7-b01, mixed mode)
> Reporter: Jason Toffaletti
>
> I have this in my pom.xml
> <dependency>
> <groupId>org.apache.kafka</groupId>
> <artifactId>kafka_2.9.2</artifactId>
> <version>0.8.0-beta1</version>
> </dependency>
> <dependency>
> <groupId>org.scala-lang</groupId>
> <artifactId>scala-library</artifactId>
> <version>2.9.2</version>
> </dependency>
> <dependency>
> <groupId>org.xerial.snappy</groupId>
> <artifactId>snappy-java</artifactId>
> <version>1.0.4.1</version>
> </dependency>
> I'm using snappy compression codec and producing about 7k msg/sec on average.
> I'm producing batches up to 10 messages per batch with null keys to a topic
> with 16 partitions, no replication. Xms and Xmx are 64MB and I'm using
> XX:+UseG1GC. After about 12 hours of operation heap usage hits right up
> against the 64MB limit and the producer drops to about 4k msg/sec because of
> the GC pressure. When I restart the process the heap usage goes back to
> normal (around 30MB) and the producer does 7k msg/sec again.
> What else can I provide to help debug this?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
