Re: kafka.tools.ConsumerOffsetChecker fails for one topic

It might be there never was/currently isn't a consumer with the group jopgroup consuming from the twitter topic. I only used it for the new consumer(offsets in broker), and then the group needs to be 'active' in order to get the offsets. On Mon, May 30, 2016 at 2:37 PM Diego Woitasen wrote: > Hi

Re: Kafka Consumer rebalancing frequently

Hi, I still face the same issue sometimes. My kafka consumer is giving this exception after failing to claim any partition. java.nio.channels.ClosedChannelException at kafka.network.BlockingChannel.send(BlockingChannel.scala:110) at kafka.producer.SyncProducer.liftedTree1$1(SyncPr

about Kafka Client rebalance

hello I read the Kafka Client source code .Client do rebalance when receive JoinGroupResponse if it is the client leader.if it is not a leader,it don't rebalance? when has many Kafka Client and them same groupId,how to do rebalance? thanks all. org.apache.kafka.clients.consumer.internals.A

Re: Ping kafka broker for healthcheck

you can use kafka java metrics to monitor some events about kafka healthy. as to kafka listen port, just check it. 2016-05-30 13:04 GMT+08:00 Joe San : > Is there any such API on the consumer or the producer that I can use to > check for the underlying connection to the kafka brokers from my pro

Re: soft failure for kakfa 0.8.2.2

thanks for your reply. yes, there are more than one controller. the msg of "soft failure" is reported by the old controller. 2016-05-31 11:42 GMT+08:00 Muqtafi Akhmad : > hello Fredo, > > My guess is that there was partition leader election that was might be > triggered by detection of offline p

Re: soft failure for kakfa 0.8.2.2

hello Fredo, My guess is that there was partition leader election that was might be triggered by detection of offline partition in Kafka cluster. Somehow the broker try to trigger leader election while previous election has been completed hence this log : > [2016-05-28 12:33:31,511] ERROR Control

Re: Kafka encryption

If you are looking to do encryprion in the serializer, you could use this as a refernce - https://github.com/rogers/change-data-capture/tree/master/kafka FS encryprion is easier, but if you cannot do that, this is a good alternative. It also offloads the encryption work off the broker. On May 26,

Re: Avro deserialization

That was exactly the problem, I found the example here to be very helpful - https://github.com/confluentinc/examples/blob/master/kafka-clients/specific-avro-consumer/src/main/java/io/confluent/examples/consumer/AvroClicksSessionizer.java

Re: No more message scenario question

Also check out writing a custom partitioner. You could figure out the number of partitions and then use a custom partitioner to write a stop message to each. I would consider looking into not needing this functionality at all if possible. Dave > On May 29, 2016, at 15:18, Tauzell, Dave wro

Re: WARN Property [sasl-stuff] is not valid

Hi Stefano, For the console consumer, you need to pass --new-consumer for it to work. For the producer, it seems like you are using configs for Kafka 0.10 with a 0.9 producer, but the warning is benign. Are you sure the producer is not working? Ismael On 30 May 2016 11:50, "Stefano Baghino" wro

Re: Compatibility between different versions of Kafka server and kafka-clients library

Hi Mikael, This is expected. Older clients work with newer brokers, but newer clients don't work with older brokers. Ismael On 30 May 2016 17:29, "Mikael Ståldal" wrote: > I am experiencing compatibility issues between different versions of Kafka > server and kafka-clients library. > > kafka-cl

Compatibility between different versions of Kafka server and kafka-clients library

I am experiencing compatibility issues between different versions of Kafka server and kafka-clients library. kafka-client 0.8.2.2 to Kafka server 0.9.0.1 works fine, but when using kafka-client 0.9.0.1 to Kafka server 0.8.2.2 I get timeout when waiting for the future returned by Producer.send(). T

Re: Kafka Streams / Processor

2) I am not sure if I understand correctly * punctuation is independent from committing (ie, you cannot use it to flush) * if you need to align writes with commits you can either use a KStream/KTable or need to register a state (see StateStore.java) 5) The application goes down -- neither pr

Re: Topics, partitions and keys

Thank you Todd for great response. On Sun, May 29, 2016 at 8:52 PM, Todd Palino wrote: > Answers are in-line below. > > -Todd > > On Sun, May 29, 2016 at 3:00 PM, Igor Kravzov > wrote: > > > Please help me with the subject. > > In Kafka documentations I found the following: > > > > *Kafka only

kafka.tools.ConsumerOffsetChecker fails for one topic

Hi, I have these topic in Kafka: # /opt/kafka/bin/kafka-topics.sh --zookeeper localhost:2181 --list__consumer_offsets facebook twitter # bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group jopgroup --zookeeper localhost:2181 --topic twitter Exiting due to: org.apache.zookeeper.Keep

WARN Property [sasl-stuff] is not valid

Hello everybody, I'm trying to run a simple Flink that both read from and writes to Kafka, however at the moment I'm having a hard time at running both the console consumer and producer to test the effectiveness of my test. It is extremely important that I run these tests in a kerberized environm

Re: Avro deserialization

Rick, Is your code really importing the correct ConsumerConfig objects? It should be: import kafka.consumer.ConsumerConfig; If you are using your IDE's auto-import feature, you might however end up with the following import, which will give you the "ConsumerConfig is not a public class" co

Re: Question about applicability of Kafka Streams

Kim, yes, Kafka Stream is very suitable for this kind of application. The code example that Tobias linked to should be a good starting point for you (thanks, Tobias!). Best, Michael On Fri, May 27, 2016 at 4:35 AM, BYEONG-GI KIM wrote: > Thank you very much for the information! > I'll look

Re: soft failure for kakfa 0.8.2.2

my server.log lots of error msg: [2016-05-28 12:12:31,132] WARN [ReplicaFetcherThread-0-1007], Replica 1008 for partition [consup-03,16] reset its fetch offset from 13985537 to current leader 1007's latest offset 13985537 (kafka.server.ReplicaFetcherThread) [2016-05-28 1

Re: soft failure for kakfa 0.8.2.2

My state-change.log>>> [2016-05-28 12:33:31,510] ERROR Controller 1008 epoch 13 aborted leader election for partition [consup-43,78] since the LeaderAndIsr path was already written by another controll er. This probably means that the current controller 1008 went through a soft fai

Re: offset out of range for kafka follower when leader changed.

some of my consumer always get out of range exception, and i can find that the new leader who is follower before trunked its log file. 2016-05-30 15:05 GMT+08:00 Fredo Lee : > my kafka cluster has twenty kafka brokers. and my producer sets > request.required.acks = -1, my brokers set min.insync.r

offset out of range for kafka follower when leader changed.

my kafka cluster has twenty kafka brokers. and my producer sets request.required.acks = -1, my brokers set min.insync.replicas=2, and unclear leader is enable. i think under this configuration, i think when one leader of the topic is changed, there should are no any log trunked partition for the f