Retention on compacted topics

Hi, Does Kafka "log.retention.bytes" or "log.retention.ms" apply to compaction enabled topic? I'll be surprised it does not, since this means that a compacted topic will potentially grow unbounded if deletion is not happening quickly enough (if we do deletion at all). Thanks, David

Re: Retention on compacted topics

Ok, thanks for confirming. On Thu, Sep 1, 2016 at 12:21 PM Tom Crayford wrote: > It is not applied. An upcoming release will have the ability to combine the > two, but right now they are mutually exclusive. > > On Thu, Sep 1, 2016 at 6:18 PM, David Yu wrote: > > > H

Re: Understand producer metrics

Thx! On Fri, Aug 19, 2016 at 12:48 AM Manikumar Reddy wrote: > This doc link may help: > > http://kafka.apache.org/documentation.html#new_producer_monitoring > > On Fri, Aug 19, 2016 at 2:36 AM, David Yu wrote: > > > Kafka users, > > > > I want to resurfac

Debug Kafka performance

Hi, We have setup a 3-broker Kafka 0.9.0.0 cluster in Cloudera. The bokers are ec2 d2.xlarge instances. What we are trying to do is to run some producer perf tests on this cluster. >From the client host, we did a bandwidth test to the brokers using iperf3 and we are getting 900Mbps connections. F

Re: Understand producer metrics

Kafka users, I want to resurface this post since it becomes crucial for our team to understand our recent Samza throughput issues we are facing. Any help is appreciated. Thanks, David On Tue, Aug 2, 2016 at 10:30 PM David Yu wrote: > I'm having a hard time finding documentation ex

Re: Failed to start kafka cleaner thread

ot; issues. Some of them are only fixed in > 0.10.0.0 (https://issues.apache.org/jira/browse/KAFKA-3587), and we're > working on a patch for 0.10.1.0 or the next pertinent release. > > Thanks > > Tom Crayford > Heroku Kafka > > On Thu, Aug 11, 2016 at 4:28 PM, David Yu

Re: Failed to start kafka cleaner thread

BTW, we are using Kafka 0.9.0.0. And I found the following thread describing the same issue: http://grokbase.com/t/kafka/users/159jbe18en/log-cleaner-thread-stops So is this still an issue in 0.9.0.0? Thanks, David On Thu, Aug 11, 2016 at 9:46 AM David Yu wrote: > Our Kafka cleaner

Failed to start kafka cleaner thread

Our Kafka cleaner has stopped running log compactions. From the log, we found: 2016-08-03 20:57:19,240 ERROR kafka.log.LogCleaner: [kafka-log-cleaner-thread-0], Error due to java.lang.IllegalArgumentException: requirement failed: Last clean offset is 104326 but segment base offset is 0 for log se

Re: Verify log compaction

nt. Look at > these settings for the topic, "segment.bytes" and "segment.ms". Lower > them > to force quicker compaction. > > On 0.8.2.2, occasionally, the compaction thread would die and then the > __consumer_offets topic would grow out of control. Kafka logs the t

Compress before or after serialization?

We are using Avro as our format when writing to a Kafka topic. After we enabled snappy compression on the producer, we don't see a change in the compression ratio (still 1). I was wondering if we should compress the message before serialization. Thanks, David

Understand producer metrics

I'm having a hard time finding documentation explaining the set of producer metrics exposed by Kafka. Can anyone explain the following? - batch-size-avg - Is this the number of msgs or number of bytes? Does this only make sense for async producers? - incoming-byte-rate/outgoing-byte-rate

Verify log compaction

Hi, We are using Kafka 0.9.0.0. One of our topic is set to use log compaction. We have also set log.cleaner.enable. However, we suspected that the topic is not being compacted. What is the best way for us to verify the compaction is happening? Thanks, David

Re: Log retention not working

ns. > > On Thu, Jul 28, 2016 at 10:16 AM, David Yu wrote: > > > Those are all with default value. Will continue investigation. > > > > On Wed, Jul 27, 2016 at 11:38 PM Manikumar Reddy < > > manikumar.re...@gmail.com> > > wrote: > > > &

Tune Kafka cluster performance

Hi, Kafka, We have a 2 broker staging cluster and a 6 broker prod cluster (all running 0.8.2.1). I'm trying to make sense of the producer performances on both. Here are some perf test results: - Prod: kafka-producer-perf-test.sh --broker-list kafkaProd-1:9092,kafkaProd-2:9092,k

Re: Log retention not working

your log directory is in /tmp/ and your OS is > deleting > > that directory? I know it's happened to me before. > > > > - Samuel > > > > On Jul 27, 2016 13:43, "David Yu" wrote: > > > > > We are using Kafka 0.8.2.0 provided by CDH. O

Log retention not working

We are using Kafka 0.8.2.0 provided by CDH. Our Kafka retention is set to default 7 days. One problem we have with one of our topics is that, the logs are purged within two days. This topic does not override any default settings. Just wanna get some pointers as to how to debug this issue. Thanks,