Re: Notable failure scenarios in high-level/new consumer

Hello Folks, Request your expertise on these doubts of mine Thanks, Prabhjot On Thu, Nov 26, 2015 at 11:28 PM, Prabhjot Bharaj wrote: > Hi, > > Request your expertise on these doubts of mine > > Thanks, > Prabhjot > > On Thu, Nov 26, 2015 at 12:09 PM, Prabhjot Bharaj > wrote: > >> Hello Folks

Re: 0.9.0 Client - Streaming vs Polling

Hello Erik, Are you referring to the java doc for KafkaConsumer ( http://kafka.apache.org/090/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html )? Currently the description of the timeout value is "timeout - The time, in milliseconds, spent waiting in poll if data is not ava

Re: producer-consumer issues during deployments

Hi, Thanks for your reply We have 4 phases of deploys and in each phase, we can take down few machines These releases happen every 2 weeks, because on all machines, there are a bunch of other micro services running along with the core system - Kafka in this case My only concern is that during run

How to tell if producer is producing messages to server?

Hi Using 0.8.2 I'm making a service status check and I want to include a kafka producer check. I was trying to use the kafka producer metrics https://github.com/apache/kafka/blob/0.8.2/clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java#L74 and I was planning to schedule a

Re: Increasing replication factor reliable?

Alright, thank you all. Appreciate it. Cheers On Wed, Nov 25, 2015 at 10:50 PM, Gaurav Agarwal wrote: > So u have two nodes running where you want to increase the replication > factor 2 because of fault tolerance. That won't be a problem > On Nov 25, 2015 6:26 AM, "Dillian Murphey" > wrote: >

Re: producer-consumer issues during deployments

Hi Prabhjot I may have slightly misunderstood your question so apologies if that’s the case. The general approach to releases is to use a rolling upgrade where you take one machine offline at a time, restart it, wait for it to come online (you can monitor this via JMX) then move onto the next.

Re: Notable failure scenarios in high-level/new consumer

Hi, Request your expertise on these doubts of mine Thanks, Prabhjot On Thu, Nov 26, 2015 at 12:09 PM, Prabhjot Bharaj wrote: > Hello Folks, > > I am trying to build fault tolerance on the consumer side, so as to make > sure that all failure scenarios are handled. > On Data integrity side, ther

Re: producer-consumer issues during deployments

Hi, Request your expertise on these doubts of mine Thanks, Prabhjot On Thu, Nov 26, 2015 at 4:43 PM, Prabhjot Bharaj wrote: > Hi, > > We arrange our kafka machines in groups and deploy these phases. > > For kafka, we’ll have to map groups with phases. During each phase of the > release, all th

Re: 0.9.0 Client - Streaming vs Polling

Hi Jay, Thanks for the quick response. That was not the behavior I expected when doing a poll(100), but I understand how it works now. I'm not sure if updating the documentation would have prevented my confusion, but maybe adding the "return the instant there are new messages, but if no messages c

Re: 0.9.0 Client - Streaming vs Polling

Actually this may be something we can improve in the documentation. Calling poll(1000) doesn't mean "check for new messages every second" but rather "return the instant there are new messages, but if no messages come return after a one second timeout passes". So in that respect both the old and ne

0.9.0 Client - Streaming vs Polling

Hi all, We're using Apache Kafka 0.9, and in our 0.8.2 Kafka, we had a consumer which used a ConsumerConnector to stream all the messages to be processed. The new KafkaConsumer seems to use a polling mechanism instead of streaming, and I was wondering: what is the 'right' way to go forward? Shoul

producer-consumer issues during deployments

Hi, We arrange our kafka machines in groups and deploy these phases. For kafka, we’ll have to map groups with phases. During each phase of the release, all the machines in that group can go down. When this happens, there are a couple of cases:- 1. All replicas are residing in a group of mach