Re: Altered retention.ms not working

Added the Kafka version and updated the command used. Regards, Saurav From: "Datta, Saurav" mailto:sda...@paypal.com>> Date: Sunday, March 27, 2016 at 10:22 AM To: "users@kafka.apache.org<mailto:users@kafka.apache.org>" mailto:users@kafka.apache.org>> Sub

Altered retention.ms not working

Hi all, We had configured our topics with an incorrect retention.ms of 864. With this, we observed the consumers are unable to receive any data after approximate 2.4 hours; this ties in to 864/1000/60/60. Later we altered the retention.ms to 8640 (an extra 0), but the consumers ar

ZookeeperConsumerConnector error

Hello, I am getting the below error message: kafka.common.MessageStreamsExistException: ZookeeperConsumerConnector can create message streams at most once at kafka.javaapi.consumer.ZookeeperConsumerConnector.createMessageStreams(ZookeeperConsumerConnector.scala:79) at kafka.javaapi.consumer

Failed toString() invocation in producer.send()

Hi all, I am getting the below message in producer.send() : SLF4J: Failed toString() invocation on an object of type [org.apache.kafka.common.Cluster] java.lang.NullPointerException at org.apache.kafka.common.PartitionInfo.toString(PartitionInfo.java:72) at java.lang.String.valueOf(String.j

Re: Using multiple consumers for same topic

Basically, if a source record is already consumed by Consumer1, will it also get consumed by Consumer2 and Consumer3 all subscribing to the same topic ? Regards, Saurav Datta Data Engineer| Desk - (408)967-7360| Cell - (408)666-1722 From: , Saurav Datta mailto:sda...@paypal.com>> Date: Tuesda

Using multiple consumers for same topic

Hello, If I use multiple consumers , each running on a different machine, for a single topic, will the output get duplicated ? Regards, Saurav

Publish large number of messages for load testing

Hello, I am load testing my streaming application. Currently, I am calling the producer.send a large number of times to queue up the messages, then running the consumer to check the throughput. The producer code looks like this: for (long nEvents = 0; nEvents < loopCntrMax; nEvents++) { long

Performance testing

Hello, Is there some documentation on how to run performance tests ? This link https://gist.github.com/jkreps/c7ddb4041ef62a900e6c lists the commands, but some documentation explaining the usage would be useful. Regards, Saurav

Sending multiple large HDFS files through Kafka

Hello, I am trying to publish multiple large HDFS files to Kafka. While it is possible to loop through the lines of each file and publish to Kafka, it will be very inefficient. Any suggestions for achieving this optimally ? Regards, Saurav

Start consuming from latest message

Hello, I have a Spark source producing to Kafka, and a Java consumer. The consumer was shutdown for around a week while the producer was active. Now when the consumer is active, it is fetching messages from 7 days back. I would like to ignore the older messages, and start from the latest one. H