Re: Consumer reconsuming all kafka messages

I checked the __consumer_offsets topic and here is an extraction from this log for the same consumer group, a specific topic (users) and specific partition (15): [storm_kafka_topology,users,15]::[OffsetMetadata[8327,{topic-partition=users-15, offset=8327, numFails=0, thread='Thread-11-kafkaSpout-e

exception in multi threaded env for manual topic assignment

Hi , I am getting below exception while reading topic partitions individual. Exception: Subscription to topics, partitions and pattern are mutually exclusivejava.lang.IllegalStateException I am passing new KafkaConsumer for each thread and topic's partition number and same consumer group for enti

Pinning clients to specific brokers

Hi folks, Is it possible to pin kafka clients to use only specific brokers throughout their lifetime and not just for the initial bootstrapping ? Thanks, Mohit

Getting TimeoutException instead of TopicAuthorizationException while using security SASL_PLAINTEXT

Hi All, I am using 0.11 Kafka version and trying out an SASL_PLAINTEXT mechanism for Authentication and Authorization. I have configured Broker and Zookeeper as well Client with necessary configurations with help of following links https://kafka.apache.org/documentation/#security_sasl_plain

Kafka with kubernetes

I'm not sure this is the correct place to post this question, but anyway... When running kafka in kubernetes, the kafka config contains this: listeners = PLAINTEXT://:tcp://10.0.0.186:9092 Which is leading to this error: No security protocol defined for listener PLAINTEXT://:TCP Here is the sec

Re: Kafka with kubernetes

Not too familiar with that error, but I do have Kafka working on Kubernetes. I'll share my files here in case that helps: Zookeeper: https://gist.github.com/aliakhtar/812974c35cf2658022fca55cc83f4b1d Kafka: https://gist.github.com/aliakhtar/724fbee6910dec7263ab70332386af33 Essentially I have 3 k

Re: Kafka with kubernetes

This one works fine, and has statefull sets built in. https://github.com/Yolean/kubernetes-kafka/ Ben On Tue, Aug 22, 2017 at 4:41 PM, Ali Akhtar wrote: > Not too familiar with that error, but I do have Kafka working on > Kubernetes. I'll share my files here in case that helps: > > Zookeeper:

Re: Pinning clients to specific brokers

Doing that doesn't really make sense in a Kafka cluster because the topic partitions and their replicas are spread out across many brokers in the cluster. That's what enables the parallel processing and fault tolerance features of Kafka. -hans > On Aug 22, 2017, at 3:14 AM, Mohit Chawla > wr

Re: Getting TimeoutException instead of TopicAuthorizationException while using security SASL_PLAINTEXT

JIRA for this issue: https://issues.apache.org/jira/browse/KAFKA-5547 On Tue, Aug 22, 2017 at 8:57 PM, sukumar.np wrote: > Hi All, > > > > I am using 0.11 Kafka version and trying out an SASL_PLAINTEXT mechanism > for Authentication and Authorization. I have configured Broker and > Zookeeper as

kafka straming progress had be done a few minutes later

Hi all: When I find kafka stream api.I use it to consumer one topic and filter data and transform to another topic . I foud that the progress will be done in a few minutes later? 发送自 Windows 10 版邮件应用

unable to connect to manual partitions

Hi, I am unable to connect to kafka individual partitions from different server, while I am able to retrieve and connect to kafka and get partition size etc from other server. I am able to connect to each partition from same server where kafka is hosted. Keep learning keep moving .

Re: unable to connect to manual partitions

​ I am using below properties : props.put("bootstrap.servers", DataAppConfig.getInstance().getValue("bootstrap.servers")); props.put("zookeeper.connect","zookeeper_host:2181"); props.put("enable.auto.commit", DataAppConfig.getInstance().getValue("enable.auto.commit")); //props.put("auto.commit.in

Kafka streams application (v 0.10.0.1) stuck at close

Hi, I have the following three question regarding Apache Kafka streams. 1. I am in a position to use v 0.10.0.1 of Apache Kafka though so many bugs related to streams are fixed in the later versions. My application consists of low level processors that run for more than an hour for larger files (

Re: Consumer reconsuming all kafka messages

Is your storm consumer set to auto.offset.reset="earliest"? On 22 August 2017 at 10:05, Elyahou Ittah wrote: > I checked the __consumer_offsets topic and here is an extraction from this > log for the same consumer group, a specific topic (users) and specific > partition (15): > > [storm_kafka_to

Re: Consumer reconsuming all kafka messages

I have question , if my service went down and i want to consume back whichever was not consumed from a given consumer group , how to handle that ? its either earliest or latest message , how about consuming all un-commited messages ? Keep learning keep moving . On Wed, Aug 23, 2017 at 10:

Re: Consumer reconsuming all kafka messages

It use the default one, which is uncomitted earliest On Wed, Aug 23, 2017 at 7:34 AM, Dan Markhasin wrote: > Is your storm consumer set to auto.offset.reset="earliest"? > > On 22 August 2017 at 10:05, Elyahou Ittah wrote: > > > I checked the __consumer_offsets topic and here is an extraction fr