Re: Kafka rebalancing message lost

For very large number of consumers, you can manually manage the offsets and/or assign partitions yourself per consumer to avoid rebalancing. On Dec 18, 2018 9:58 AM, "Ryanne Dolan" wrote: > Parth, I am skeptical that you actually need 500+ consumers. A well tuned > consumer can process hundreds

Re: How to know the begin offset of a partition

Not the best idea but you can open another listener that is not SASL enabled until they patch is released. On Tue, Jul 31, 2018 at 8:17 AM, Pierre Coquentin < pierre.coquen...@gmail.com> wrote: > thanks, too bad :( > > On Tue, Jul 31, 2018 at 4:44 PM Gabriele Paggi > wrote: > > > Hi Pierre, > >

Re: What is the performance impact of setting max.poll.records=1

You can always add more partitions/consumer threads each fetching a few more records than 1 but manually commit asynchronously one at a time, not the best but better than doing max.poll.records=1 which fetches one record from remote server at a time. On Fri, May 4, 2018 at 4:19 AM, Mads Tandrup <

Ideas to achieve no loss in a 0.10 Publisher?

We are looking for a way to guarantee no "publisher" loss where waiting until the cluster/network health is resolved is fine. What we are doing right now is to at avoid this loss in favor of at least once guarantee by doing a *future.get() on say 1000th record or 1s (whatever comes first)* and if

Re: Cater to processing longer than max.poll.interval.ms

Think, new versions have better ways of doing this. In 0.10.2, because poll() ensure liveness, you can disable auto commits and use consumer pause() to avoid calling poll() (so brokers may ignore max.poll.interval.ms) so those partitions are not assigned to other consumers and also handle ConsumerR

Re: Intermittent NoLeaderForPartition exceptions

For us, it was always network blips between Kafka and ZK. On Tue, Jan 16, 2018 at 11:00 AM, Atul Mohan wrote: > Hello, > We have 5 Kafka brokers and have a service that continuously send events to > partitions across these 5 brokers. The configuration works fine but every > 90 minutes ~ 120 minu

Re: Error for partition [__consumer_offsets,15] to broker

This is a known issue for us in 0.10 due to network related problems with ZK causing no leader exception and restarting quickly fixed it. You can increase time out to alleviate the problem a bit. On Dec 8, 2017 8:20 PM, "Abhit Kalsotra" wrote: > Guys can I get any reply of help on the same.. thi

Re: Kafka cluster Error

"Unable to connect" Try pinging and running ZK cli commands on one of the ZKs from the Kafka Broker that is failing to come up. On Tue, Oct 10, 2017 at 3:35 PM, Kannappan, Saravanan (Contractor) < saravanan_kannap...@comcast.com> wrote: > Hello, Someone can you help me kafka server not starting

Re: Adding/Removing Brokers to Kafka, while data is flowing into Kafka topics

So, mirrormaker let's you use one data center available for live traffic and the other for repartitioning? On Tue, Jun 13, 2017 at 3:14 PM, Mohammed Manna wrote: > You should plan this ahead and make sure that your topic partition and > replication factors are updated correctly using kafka topic

Re: org.apache.kafka.common.errors.TimeoutException

Are you able to publish any messages at all? If it is one off, then it is possible that the broker is busy and the client busy that it could not publish that batch of messages in that partition 0 within 1732 ms in which case you should increase the message timeouts and retries. Search the timeout e

Re: Call to consumer.poll(1000) hangs

Running consumer with full DEBUG/TRACE level logging will show you why. On Thu, Mar 2, 2017 at 2:13 AM, Dhirendra Suman < dhirendra.su...@globallogic.com.invalid> wrote: > Hi, > > http://stackoverflow.com/questions/42551704/call-to- > consumerrecordsstring-string-records-consumer-poll1000-hangs-a

Re: error in kafka producer

It means that your request has timed out or could not be sent within request timeout ms, also check max block ms. You will have to tune/increase throughout (search threads) or the timeouts. On Feb 28, 2017 2:22 PM, "shyla deshpande" wrote: > org.apache.kafka.common.errors.TimeoutException: Expir

Kafka 0.10 and ZK 3.4.8 dependency upgrade

We tested Kafka 0.9 with zookeeper_3.4.5+dfsg-1_all.deb on Ubuntu 14.04 Trusty and want to use stable release of kafka_2.11-0.10.1.1 (hope, this has no critical issues) to avoid known producer issues, but, apparently this needs an upgraded ZK 3.4.8+ https://kafka.apache.org/ documentation/#zkversio

Re: Reg: Reading consumer groups and lag

You can run the same class executed in the scripts. On Feb 8, 2017 8:50 AM, "Sumit Maheshwari" wrote: > Hi, > > Currently in 0.10 we can get the information about the consumer groups and > respective lag using the kafka-consumer-groups.sh. > Is there a way to achieve the same programatically in j

Re: Fault tolerance not working in Kafka

What is the exact exception you see? With 4 partitions, consumers should not have a problem if one goes down, do you see any broker ISR for your topic? On Sun, Feb 5, 2017 at 8:15 PM, Nitin Shende wrote: > Hi Team, > > I am using Apache kafka with 6 brokers. I m having topic with 4 partition > a

Re: Consumer Rebalancing Question

"From what I understand, there's currently no way to prevent this type of shuffling of partitions from worker to worker while the consumers are under maintenance. I'm also not sure if this an issue I don't need to worry about." If you don't want rebalance, consumers can also manually subscribe to

Re: Questions about single consumer per partition approach

Newbie here, Q2) Think, there is no rebalance if you go with manual partition assignment (assign(List) ), but were you looking for subscribe(List, ConsumerRebalanceListener)

Re: sliding ktable?

Yes, thanks.

Re: Kafka streaming changelog topic max.message.bytes exception

Are you talking about total messages and therefore size or per message payload size. On Tue, Nov 8, 2016 at 10:00 AM, Sachin Mittal wrote: > Message size itself increases over the time. > > Message is something like > key=[list on objects] > > This increases with time and then at a point kafka i

Re: sliding ktable?

There is a problem with tombstoning old entries based on a new entry, that, the keys which have no new entries will remain there forever. On Mon, Nov 7, 2016 at 9:38 AM, Matthias J. Sax wrote: > -BEGIN PGP SIGNED MESSAGE- > Hash: SHA512 > > John, > > your thinking is on the right track!

Re: Kafka client circuit breaker

, Andrey Dyachkov wrote: > R Krishna, > > We have default timeout for producer and consumer it is 30 secs. > If Kafka is not available we will wait 30 secs on each request. > Decreasing timeout won't help much because we have a lot requests > but circuiting them will help

Re: Kafka client circuit breaker

What are the issues and problems you want to solve though? On Oct 29, 2016 3:59 AM, "Andrey Dyachkov" wrote: > Hi, > > We use Kafka for around half a year as a main backend system for storing > events from big variety of internal microservices. In general It works > quite well and we experience i

Re: Question: Data Loss and Data Duplication in Kafka

Experimenting with kafka myself, and found timeouts/batch expiry (valid and invalid configurations), and max retries also can drop messages unless you handle and log them gracefully. There are also a bunch of org.apache.kafka.common.KafkaException hierarchy exceptions some of which are thrown for v

Re: Batch Expired

Are any requests at all making it? That is a pretty big timeout. However, I noticed if there is no connections made to broker, you can still get batch expiry. On Fri, Aug 26, 2016 at 6:32 AM, Ghosh, Achintya (Contractor) < achintya_gh...@comcast.com> wrote: > Hi there, > > What is the recommend

Re: Recommendation for producer batch size

Don't think it always fills up buffer size before sending and then there is linger, that can add to the delay. From what I read, it depends on your use case. This link talks about http://webcache.googleusercontent.com/search?q=cache:http://ingest.tips/2015/07/19/tips-for-improving-performance-of-k

Re: Kafka topic-partition won't serve if prefered election fails for a partition.

After this producing/consuming from this partition fails. > > On Sunday 14 August 2016, R Krishna wrote: > > > But, isn't that assuming your topics are replicated to all brokers, in > your > > case there are no offsets probably for this reason too. > >

Re: kafka-consumer-groups.sh delete group with new-consumer

Guozhang, don't we just need latest partition offsets per consumer? what is the idea behind keeping all offsets "default" to forever? On Fri, Aug 12, 2016 at 12:05 PM, Guozhang Wang wrote: > Hi Yuanjia, > > New consumer's group registry information is stored on the Kafka brokers, > not ZK any mo

Re: Kafka topic-partition won't serve if prefered election fails for a partition.

But, isn't that assuming your topics are replicated to all brokers, in your case there are no offsets probably for this reason too. On Fri, Aug 12, 2016 at 2:23 PM, Zakee wrote: > Typically "preferred leader election” would fail if/when one or more > brokers still did not come back online after

Kafka compaction that can aggregate/count messages by key?

Is it possible to use Kafka to track counts instead of deletion on compaction? I know we can aggregate ourself and add it to a different topic but that won't make sense if the time window is more than few seconds. Say, I can then, use it to count based on a key containing minute, hour, day. https:

Re: Kafka java consumer processes duplicate messages

rformance deteriorates > drastically. > > What may be I need to try, please correct me if I have got it wrong > completely, is to try async commit mode and see how it performs. > > Also, as I mentioned there was a bug reported of same kind with > kafka-python, can it be same here

Re: Kafka java consumer processes duplicate messages

gt; -Dave > ____ > From: R Krishna > Sent: Monday, August 1, 2016 1:32 PM > To: users@kafka.apache.org > Subject: Re: Kafka java consumer processes duplicate messages > > Remember reading about these options for higher consumer guarantees: >

Re: Kafka java consumer processes duplicate messages

Remember reading about these options for higher consumer guarantees: Unclean.leader.election = false Auto.offset.commit = falseconsumer side Commit after processingsyncCommit() regularly What about your producer, does it wait until it reaches all replicas in ISR, i.e., ack=all or none? Not

Re: Performance of producer sending to many vs to few topics

We did similar testing recently, newbie here, assuming you did async publisher, did you also test with multiple partitions (1-1000) per topic as well. More topics, implies more metadata per topic exchanged every minute, more batches maintained and flushed per topic+partition per producer so higher

Re: consumer timeout not being respected when kafka broker is not available

Just started testing consumers and can reproduce this, did you check JIRA or the forum. May be as this is consumer side, there isn't much you can do but wait right especially for commit Sync? On Thu, Jul 7, 2016 at 1:21 PM, Fumo, Vincent wrote: > KafkaConsumer v0.9:: > > I have a consumer set up

Re: Kafka 0.9 API connection refused exception not raised to client code

Hmm, as a newbie with Kafka v0.9 and having tested this scenario, this is what is happening: You can reproduce failure by setting just BOOTSTRAP_SERVERS_CONFIG, by not passsing custom timeouts, it should then invoke your callback in producer.send(record, new ProducerCallback(record)); with "org.ap

Re: kafka unable to send records - scala / spark

Try increasing timeouts. On Jul 12, 2016 7:40 AM, "Sumit Khanna" wrote: > Hello Guys. > > Have tried a lot, from kafka.javaapi. etc to Producer to KafkaProducer, and > am working with 0.9.0.0 > This is the error I am getting : > > org.apache.kafka.common.errors.TimeoutException: Failed to update

Re: Kafka - offset preservation

Which version is this? I am a new user myself, are you using the same consumer group name? In V0.9, once connected, broker stores consumer offsets per consumer group, and continues from where it left off, the earliest/latest are for the first time only. On Wed, Jul 6, 2016 at 6:19 AM, Pawel Huszcz

Re: Configuring client-side timeouts in the Java Producer so that send() doesn't block

Any luck trying to figure out this problem? On Wed, May 18, 2016 at 10:53 AM, Samuel Chase wrote: > Hello Ismael, > > On Wed, May 18, 2016 at 5:54 PM, Ismael Juma wrote: > > Your second example should work as well. Can you please include the code > > you are using to test the scenario and what

Re: Kafka broker slow down when consumer try to fetch large messages from topic

Prateek, hope you looked at compression? On Thu, Jun 2, 2016 at 10:26 AM, Tom Crayford wrote: > The article says ideal is about 10KB, which holds up well with what we've > seen in practice as well. > > On Thu, Jun 2, 2016 at 6:25 PM, prateek arora > wrote: > > > Hi > > Thanks for the informatio

Re: async producer retry behavior - at least once guarantee

. On Mon, Jun 13, 2016 at 8:32 PM, R Krishna wrote: > As part of testing v0.9 Kafka at least once guarantees, we tried > disconnecting Producer network and found that retries=1000 are not > happening. We get a > > WARN kafka-producer-network-thre

Re: 10MB message

There are options to compress on the wire and in the topic. On Tue, May 31, 2016 at 8:35 AM, Igor Kravzov wrote: > In our system some data can be as big as 10MB. > Is it OK to send 10 MB message through Kafka? What configuration > parameters should I check/set? > It is going to be one topic wit

async producer retry behavior - at least once guarantee

As part of testing v0.9 Kafka at least once guarantees, we tried disconnecting Producer network and found that retries=1000 are not happening. We get a WARN kafka-producer-network-thread | producer-1 [.kafka.clients.producer.internals.Sender] - Got error produce response with correlation id

Re: Java Client for Kafka Consumer(0.9) not polling the records from brokers instantly

Don't think you cannot attach diagrams in these mailling lists. On Thu, May 26, 2016 at 7:35 PM, Navneet Kumar wrote: > Hi > We are facing a issue where our Consumer component is not instantly > logging the records in DB polled from the Brokers. We have following > theOur architecture diagram is

Suggestions with hostnames in a Kafka cluster

As a newbie, just setup my first Kafka 3 node, each in its own host cluster and its own ZK. Everything went fine, I could see three brokers registered in all three ZK /brokers/ids until I created a topic with this exception: ~/kafka/bin/kafka-topics.sh --create --zookeeper localhost:2181 --replica

Re: Kafka 9 version offset storage mechanism changes

And where is the documentation for this topic: "__consumers_offsets" On Tue, May 10, 2016 at 1:16 AM, Spico Florin wrote: > Hi! > Yes both are possible. The new versions 0.9 and above store the offsets in > a special Kafka topic named __consumers_offsets. > Regards, > florin > > On Tue, May 10

Re: KafkaProducer NullPointerException

Do you mind sharing your log4j2 xml and if you can run it with your version separately as a simple standalone client? On Wed, Apr 20, 2016 at 4:26 AM, Prem Panchami wrote: > Hi, > We have a Kafka producer app that participates in the larger system. It > worked fine sending messages. We just adde

Re: Not able to run multiple consumer instances

Not sure about your config, but I read somewhere (also a newbie) that if number of consumers is more than number of partitions on the topic, some will not get any messages. Search consumer parallelism. On Fri, Apr 29, 2016 at 4:11 AM, Marko Bonaći wrote: > Yes, a partition can be accessed by onl

Re: Kafka Newbie question

Also, a newbie, using 0.9.0.1, I think you meant auto.offset.reset=earliest, did the OP have an intention to use his own commit strategy/management by setting enable.auto.commit=false? With the auto.offset.reset=earliest, a "new" consumer will get the earliest partition offsets and commit them and

Re: Help understanding a failure please.

Sorry, if this sounds lame, but can you ping or telnet? On Wed, Apr 13, 2016 at 9:55 AM, Chris Neal wrote: > Hi all. > > I'm running a two node cluster that has been rock solid for almost a year > and a half. We experienced an outage of one of the two brokers this > morning, and from the logs,

Re: Multiple streaming jobs on the same topic

Then, can you specify a size/percentage of cache per consumer group? On Apr 1, 2016 9:09 AM, "Cees de Groot" wrote: > One of Kafka's design ideas is to keep data in the JVM to a minimum, > offloading caching to the OS. So on the Kafka level, there's pretty much > not much you can do - the old dat

RE: Error while publishing messages

Can you elaborate on this. "Is broker able to connect to ZK properly ? May be this can be checked via ZK CLI tools to check if the broker registrations happen properly as soon as you start Kafka broker" On Mar 26, 2016 8:18 AM, "Muthukumaran K" wrote: > Hi Bhargo, > > I ran into similar issues m

how do we get v0.9+ "new consumer" group current offsets when they are down?

We are evaluating Kafka starting with 0.9.0.1 and among v0.9 new consumer clients, we can monitor/describe them when they are actually running but how do you get the last committed offsets of all groups once they are down or have stopped consuming? This command only works when "new consumers are r

Number of messages in a kafka topic

Shouldn't this be difference of earliest and latest offset per partition sum? bash-4.3# $KAFKA_HOME/bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list 10.30.26.98:32774 --topic test-topic --time -1 | awk -F ":" '{sum += $3} END {print sum}' 13818663 bash-4.3# $KAFKA_HOME/bin/kafka-r

Starting v0.9.0.1 consumers from beginning only first time does not work auto.offset.reset=earliest

We are just evaluating Kafka starting with the latest v0.9.0.1. A very basic use case is to start initial consumers to begin from the earliest entry only for the first time, the second restarts will start from an offset wherever they left off. From what I read, you can set auto.offset.reset = earli