New Java Producer Client handling case where Kafka is unreachable

Hello Everyone, In the the new Java Producer API, the Callback code in KafkaProducer.send is run after there is a response from the Kafka server. This can be used if some error handling needs to be done based on the response. When using the new Java Kafka Producer, I've noticed that when the Kafk

Re: New Java Producer Client handling case where Kafka is unreachable

You can set producer property retries not equal to 0. Details can be found here http://kafka.apache.org/documentation.html#newproducerconfigs On Fri, Mar 20, 2015 at 3:01 PM, Samuel Chase wrote: > Hello Everyone, > > In the the new Java Producer API, the Callback code in > KafkaProducer.send is

Re: New Java Producer Client handling case where Kafka is unreachable

Hello Tao, On Fri, Mar 20, 2015 at 12:39 PM, tao xiao wrote: > You can set producer property retries not equal to 0. Details can be found > here > http://kafka.apache.org/documentation.html#newproducerconfigs Thanks! I shall try that. Samuel

Re: New Java Producer Client handling case where Kafka is unreachable

@Tao, On Fri, Mar 20, 2015 at 12:39 PM, tao xiao wrote: > You can set producer property retries not equal to 0. Details can be found > here > http://kafka.apache.org/documentation.html#newproducerconfigs I set "retries" to "1", but send is still blocking until the Kafka Server is reachable again

Re: New Java Producer Client handling case where Kafka is unreachable

The underlining send runs in a different thread and doesn't block producer.send(). One way I can think of to detect this is to set block.on.buffer.full=false and catch BufferExhaustedException then check if the broker is reachable. But this is an hacky way as BufferExhaustedException may indicate o

Re: New Java Producer Client handling case where Kafka is unreachable

On Fri, Mar 20, 2015 at 3:24 PM, tao xiao wrote: > The underlining send runs in a different thread and doesn't block > producer.send(). One way I can think of to detect this is to set > block.on.buffer.full=false and catch BufferExhaustedException then check if > the broker is reachable. But this

Re: New Java Producer Client handling case where Kafka is unreachable

I think KafkaProducer.send method blocks until it fetches partition metadata for configured time using "metadata.fetch.timeout.ms", once time out it throws TimeoutException. You might be experiencing TimeoutException ? ref: KafkaProducer.java(waitOnMetadata) On Fri, Mar 20, 2015 at 2:42 PM, Samue

Re: New Java Producer Client handling case where Kafka is unreachable

@Sunil On Fri, Mar 20, 2015 at 3:36 PM, sunil kalva wrote: > I think KafkaProducer.send method blocks until it fetches partition > metadata for configured time using "metadata.fetch.timeout.ms", once time > out it throws TimeoutException. You might be experiencing TimeoutException ? My co-worker

Re: New Java Producer Client handling case where Kafka is unreachable

@Samuel My point was The else branch of the code will be executed when metadata is not available, and metadata is not available when kafka cluster is not rachable. please correct me if i am wrong.. On Fri, Mar 20, 2015 at 3:43 PM, Samuel Chase wrote: > @Sunil > > On Fri, Mar 20, 2015 at 3:36 PM

Re: New Java Producer Client handling case where Kafka is unreachable

@Sunil The else branch will be executed if `metadata.fetch().partitionsForTopic(topic)` returns non NULL value. I assume that when Kafka is unreachable, it will return NULL. `waitOnMetadata()` then returns; we never enter the else branch when Kafka is unreachable. @Everyone: Is this explanation c

Re: Kafka 0.9 consumer API

Thanks! On Thursday, March 19, 2015, Jay Kreps wrote: > Err, here: > > http://kafka.apache.org/083/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html > > -Jay > > On Thu, Mar 19, 2015 at 9:40 PM, Jay Kreps > wrote: > > > The current work in progress is documented here: > >

New Java Producer Client handling unreachable Kafka

Hello Everyone, In the the new Java Producer API, the Callback code in KafkaProducer.send is run after there is a response from the Kafka server. This can be used if some error handling needs to be done based on the response. When using the new Java Kafka Producer, I've noticed that when the Kafk

Re: Consumer offsets in offsets topic 0.8.2

Hi Vamsi, The ConsumerOffsetChecker.scala or kafka-consumer-offset-checker.sh still works. You can use them to check the offsets. If you need to check the offsets programmatically, you can send OffsetsFetcheRequest to broker using simple consumer. You may refer the ConsumerOffsetChecker.scala to s

Re: New Java Producer Client handling case where Kafka is unreachable

This is correct when you send to a topic for the first time. After that the metadata will be cached, the metadata cache has an age and after it expires, metadata will be refreshed. So the time a producer found a broker is not reachable is the minimum value of the following times: 1. Linger.ms + ret

Re: New Java Producer Client handling case where Kafka is unreachable

Even if you have metadata cached, if the broker isn't available then messages can get stuck in the producer indefinitely. Currently the new producer doesn't have any client-side timeouts, which is a bug. See https://issues.apache.org/jira/browse/KAFKA-1788 for more details. On Fri, Mar 20, 2015 a

Post on running Kafka at LinkedIn

For those who are interested in detail on how we've got Kafka set up at LinkedIn, I have just published a new posted to our Engineering blog titled "Running Kafka at Scale" https://engineering.linkedin.com/kafka/running-kafka-scale It's a general overview of our current Kafka install, tiered

Re: New Java Producer Client handling case where Kafka is unreachable

I think if the leader is down, it a leader is down, the producer would issue a metadata request and get the new leader and start producing to it. But if the entire kafka cluster is down, it would try for some configured number of retries and would return back an error in future. This is my understa

Re: kafka mirrormaker cross datacenter replication

Hey Kane, When mirrormakers loose offsets on catastrophic failure, you generally have two options. You can keep auto.offset.reset set to "latest" and handle the loss of messages, or you can have it set to "earliest" and handle the duplication of messages. Although we try to avoid duplicate mes

Re: kafka mirrormaker cross datacenter replication

Not sure if transactional messaging will help in this case, as at least for now it is still targeted within a single DC, i.e. a "transaction" is only defined within a Kafka cluster, not across clusters. Guozhang On Fri, Mar 20, 2015 at 10:08 AM, Jon Bringhurst < jbringhu...@linkedin.com.invalid>

Re: Updates To cwiki For Producer

For the new java producer, its java doc can be found here: http://kafka.apache.org/083/javadoc/index.html?org/apache/kafka/clients/producer/KafkaProducer.html We can update the wiki if there are some examples that are still missing from this java doc. Guozhang On Thu, Mar 19, 2015 at 4:37 PM, P

Re: Updates To cwiki For Producer

Thanks that's helpful. I am working on an example producer using the new API, if I have any helpful notes or examples I'll share that. I was basically trying to be lazy and poach some example code as a starting point for our internal tests :) Cheers, -pete On 03/20/15 10:59, Guozhang Wang wrote

Re: Kafka 0.9 consumer API

I am trying to understand the semantics of the timeout specified in the poll method in http://kafka.apache.org/083/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html. Is this timeout a measure of how long the fetch request will be parked on the broker waiting for a reply or is

Re: Post on running Kafka at LinkedIn

For those who missed it: The Kafka Audit tool was also presented at the 1/27 Kafka meetup: http://www.meetup.com/http-kafka-apache-org/events/219626780/ Recorded video is here, starting around the 40 minute mark: http://www.ustream.tv/recorded/58109076 Slides are here: http://www.ustream.tv/reco

Re: Post on running Kafka at LinkedIn

Hi, James, Thank you for sharing it! The links of videos and slides are the same. Could you check the link of slides? Xiao Li On Mar 20, 2015, at 11:30 AM, James Cheng wrote: > For those who missed it: > > The Kafka Audit tool was also presented at the 1/27 Kafka meetup: > http://www.mee

Re: Post on running Kafka at LinkedIn

here is the slide http://www.slideshare.net/JonBringhurst/kafka-audit-kafka-meetup-january-27th-2015 On Sat, Mar 21, 2015 at 2:36 AM, Xiao wrote: > Hi, James, > > Thank you for sharing it! > > The links of videos and slides are the same. Could you check the link of > slides? > > Xiao Li > > On

Re: Post on running Kafka at LinkedIn

Oops. Thank you Tao Xiao! -James > On Mar 20, 2015, at 11:49 AM, tao xiao wrote: > > here is the slide > > http://www.slideshare.net/JonBringhurst/kafka-audit-kafka-meetup-january-27th-2015 > > On Sat, Mar 21, 2015 at 2:36 AM, Xiao wrote: > >> Hi, James, >> >> Thank you for sharing it! >>

Re: 'roundrobin' partition assignment strategy restrictions

Jiangle, The error messages I got (and the config doc) do clearly state that the number of threads per consumer must match also I'm not convinced that an easy to understand algorithm would work fine with a heterogeneous set of selected topics between consumers. Jason On Thu, Mar 19, 2015 at

Re: Kafka 0.9 consumer API

The timeout in the poll call is more or less the timeout used by the selector. So each call to poll will do socket activity on any ready sockets, waiting for up to that time for a socket to be ready. There is no longer any background threads involved in the consumer, all activity is driven by the a

Kafka-Storm: troubleshooting low R/W throughput

Kafka on test cluster: 2 Kafka nodes, 2GB, 2CPUs3 Zookeeper nodes, 2GB, 2CPUs Storm:3 nodes, 3CPUs each, on the same Zookeeper cluster as Kafka. 1 topic, 5 partitions, replication x2 Whether I use 1 slot for the Kafka Spout or 5 slots (=#partitions), the throughput seems about the same. I can't se

Re: kafka mirrormaker cross datacenter replication

Also, as I understand we either have to mark all messages with unique IDs and then deduplicate them, or, if we want just store last message processed per partition we will need exactly the same partitions number in both clusters? On Fri, Mar 20, 2015 at 10:19 AM, Guozhang Wang wrote: > Not sure

Re: Kafka 0.9 consumer API

Awesome - can't wait for this version to be out! On Fri, Mar 20, 2015 at 12:22 PM, Jay Kreps wrote: > The timeout in the poll call is more or less the timeout used by the > selector. So each call to poll will do socket activity on any ready > sockets, waiting for up to that time for a socket to

No broker partitions consumed by consumer thread

We are running 0.8.1.1, am experiencing following issue, any clue I have 7 topics in my Kafka cluster of 5 nodes. I pushed  1 million messages to each topic using perf producer tool (no compression, no acknowledgement) . I saw old messages about a bug in 0.8 Consumer code and it is fixed but I am

Re: Post on running Kafka at LinkedIn

Amazing growth numbers. At the meetup on 1/27, Clark Haskins presented their Kafka usage at the time. It was: Bytes in: 120 TB Messages In: 585 million Bytes out: 540 TB Total brokers: 704 In Todd's post, the current numbers: Bytes in: 175 TB (45% growth) Messages In: 800 billion (36% growth)

Re: Post on running Kafka at LinkedIn

Yep! We are growing :) -Clark Sent from my iPhone > On Mar 20, 2015, at 2:14 PM, James Cheng wrote: > > Amazing growth numbers. > > At the meetup on 1/27, Clark Haskins presented their Kafka usage at the time. > It was: > > Bytes in: 120 TB > Messages In: 585 million > Bytes out: 540 TB > T

RE: Post on running Kafka at LinkedIn

800B messages / day = 9.26M messages / sec over 1100 brokers = ~8400 message / broker / sec Do I get this right? Trying to benchmark my own test cluster and that's what I see with 2 brokers...Just wondering if my numbers are good or bad... > Subject: Re: Post on running Kafka at LinkedIn > From

Re: Post on running Kafka at LinkedIn

Keep in mind that these brokers aren't really stressed too much at any given time -- we need to stay ahead of the capacity curve. Your message throughput will really just depend on what hardware you're using. However, in the past, we've benchmarked at 400,000 to more than 800,000 messages / bro

RE: Post on running Kafka at LinkedIn

This is why I'm confused because I'm tryign to benchmark and I see numbers that seem pretty low to me...8000 events/sec on 2 brokers with 3CPU each and 5 partitions should be way faster than this and I don't know where to start to debug... the kafka-consumer-perf-test script gives me ridiculousl

ConsumerConnector.shutdown() behavior

Hi, I am using Kafka 0.8.1.1 I’ve noticed that the call to ConsumerConnector.shutdown.shutdown() takes approximately 30 seconds. Coincidentally, that’s the same as *socket.timeout.ms* I also see this message in the logs right before the call completes: 13:21:52.708 WA

Re: kafka mirrormaker cross datacenter replication

I think 1) will work, but not sure if about 2), since messages replicated at two clusters may be out of order as well, hence you may get message 1,2,3,4 in one cluster and 1,3,4,2 in another. If you remember that your latest message processed in the first cluster is 2, when you fail over to the oth

Re: 'roundrobin' partition assignment strategy restrictions

Hi Jason, The motivation behind round robin is to better balance the consumers¹ load. Imagine you have two topics each with two partitions. These topics are consumed by two consumers each with two consumer threads. The range assignment gives: T1-P1 -> C1-Thr1 T1-P2 -> C1-Thr2 T2-P1 -> C1-Thr1 T2

Re: No broker partitions consumed by consumer thread

I think the issue is with " --from-latest" - this means consumers will consume only data that arrives AFTER they start. If you do that, first start consumers, leave them running, and then start producers. If you want to run producers first and only start consuming when producers are done, remove

Re: Updates To cwiki For Producer

We have a patch with examples: https://issues.apache.org/jira/browse/KAFKA-1982 Unfortunately, its not committed yet. Gwen On Fri, Mar 20, 2015 at 11:24 AM, Pete Wright wrote: > Thanks that's helpful. I am working on an example producer using the > new API, if I have any helpful notes or exam