Re: Consumer Offsets and Open FDs

We ran into this as well, and I ended up with the following that works for us. log.cleaner.dedupe.buffer.size=536870912 log.cleaner.io.buffer.size=2000 On 13/07/2016 14:01, "Lawrence Weikum" wrote: >Apologies. Here is the full trace from a broker: > >[2016-06-24 09:57:39,881] ERROR [kaf

Re: Kafka without Storm/Spark

Definitely. You can read off kafka using the samples shown in KafkaConsumer javadoc, transform if necessary and publish to the destination topic. On 01/07/2016 03:24, "numan goceri" wrote: >Hello everyone, >I've a quick question:I'm using Apache Kafka producer to write the messages >into a

Re: How to work around log compaction error (0.8.2.2)

og-cleaner-thread-stops > > >On Thu, Apr 28, 2016 at 1:44 AM, Rakesh Vidyadharan < >rvidyadha...@gracenote.com> wrote: > >> Hello, >> >> We enabled log compaction on a few topics, as we want to preserve >> permanently the latest versions of messages pu

How to work around log compaction error (0.8.2.2)

Hello, We enabled log compaction on a few topics, as we want to preserve permanently the latest versions of messages published to specific topics. After enabling compaction, the log cleaner thread dies with the same error for the topics we tried it on. It looks like kafka has starting offset

Re: Log cleaner error

Found the issue. My publisher was not assigning a key to all messages. On 07/03/2016 14:40, "Rakesh Vidyadharan" wrote: >Hello, > >We are using Kafka 0.8.2.2 and have modified most of our topics to use log >compaction and a shorter retention.ms equivalent to 24

Log cleaner error

Hello, We are using Kafka 0.8.2.2 and have modified most of our topics to use log compaction and a shorter retention.ms equivalent to 24 hours for those topics. We did not see any change in the disk usage, and on looking at the log cleaner log file, noticed the following message (repeated on e

Re: Migrating to clustered environment

Found the issue. Our cluster is on AWS, and the third node had not set the advertised.host.name property which is required on AWS. Set that and replication completed successfully. On 05/02/2016 10:17, "Rakesh Vidyadharan" wrote: >Hello, > >After almost a year of runnin

Migrating to clustered environment

Hello, After almost a year of running kafka on a single node, we are in the process of migrating to a 3 node cluster. To test the process we followed the following process: * Stop our current kafka instance, copy the entire data directory and zookeeper data directories to one of the new

Re: Unable to set log.cleanup.policy on topic

on property (see >http://kafka.apache.org/documentation.html#topic-config ). Since you are >configuring particular topic, you need to use second one. > >Kind regards, >Stevo Slavic. > >On Mon, Dec 7, 2015 at 8:59 PM, Rakesh Vidyadharan < >rvidyadha...@gracenote.com> wrote: > >&

Unable to set log.cleanup.policy on topic

Hello, I upgraded to the latest 0.8 release 0.8.2.2, and tried to set the log cleanup policy on some of our topics (I got same error with 0.8.1, but thought it may have been some issue with that release). /opt/kafka/bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic metamorphosis.t

Re: Failed attempt to delete topic

Thanks Mayuresh. I was able to use the shell to delete the entries and things are working fine now. On 03/12/2015 18:22, "Mayuresh Gharat" wrote: >you can use the zookeeper shell inside the bin directory for that. > >Thanks, > >Mayuresh > >On Thu, Dec 3, 2015

Re: Failed attempt to delete topic

ntroller, when you deleted the topic? >> >> Thanks, >> >> Mayuresh >> >> On Thu, Dec 3, 2015 at 2:30 PM, Rakesh Vidyadharan < >> rvidyadha...@gracenote.com> wrote: >> >> > Hello, >> > >> > We are on an older kafka (0.8.

Re: Failed attempt to delete topic

fka.server.KafkaApis) Thanks Rakesh On 03/12/2015 17:33, "Mayuresh Gharat" wrote: >Can you paste some logs from the controller, when you deleted the topic? > >Thanks, > >Mayuresh > >On Thu, Dec 3, 2015 at 2:30 PM, Rakesh Vidyadharan < >rvidyadha...@graceno

Failed attempt to delete topic

Hello, We are on an older kafka (0.8.1) version. While a number of consumers were running, we attempted to delete a few topics using the kafka-topics.sh file (basically want to remove all messages in that topic and restart, since our entities went through some incompatible changes). We notice

Re: Avro messages from Kafka topic

We publish messages to kafka in Thrift format. We use the old simple consumer and just retrieve the message bytes, transform back to object model using Thrift API and do whatever our application needs with it. On 20/10/2015 11:08, "Buntu Dev" wrote: >I got a Kafka topic with messages in Avro fo

Re: Launch Kafka/Zookeeper servers without using command line

Hello, I tried using the sources for unit tests, but have been unsuccessful in getting it to work with the low level API (kafka version 0.8.2.2). Our code is modelled on the Simple Consumer API code, and the error always occurs in the findLeader code. SEVERE: Error communicating with broker:loca

replication factor: 1 larger than available brokers: 0 issue

Hello, We have been using Kafka 0.8.1 for a few months and this morning we started seeing a lot of these types of errors. We started trying to publish messages to a few additional topics, but topic creation (automatic creation mode) always fails with the following type of error: kafka.common.