Re: how to control assign policy for consumers

Does your case requires analyzing connection graphs? I think you just need to make sure that a producer would connect to as few brokers as possible, and to achieve this you just needs to know the partition assignment map, which exist on any broker's metadata cache. Guozhang On Thu, May 29, 2014

Re: Message lost during leader fail-over

On Thu, May 29, 2014 at 08:39:47PM -0700, Jun Rao wrote: > Have you looked at request.required.acks in > http://kafka.apache.org/documentation.html#producerconfigs ? Thanks Jun, that's make sense to me. -- Thanks, Chengwei > > Thanks, > > Jun

Re: running on scala 2.11

It is possible that when I tried this ticket initially there was something environmental in my build process causing it to appear to work. I reopened the ticket. I will go back through it again on a few different environments and in more detail and post results either way. /*

Re: Exception kafka.common.NotLeaderForPartitionException

Hi Guozhang Wang, I restarted the Broker yesterday, and I am still seeing this exception on consumer and producer sides. Thanks, Bhavesh On Thu, May 29, 2014 at 8:14 PM, Guozhang Wang wrote: > Bhavesh, > > This exception is usually normal and transient upon broker restarts. Do you > still see

Re: Message lost during leader fail-over

Have you looked at request.required.acks in http://kafka.apache.org/documentation.html#producerconfigs ? Thanks, Jun On Thu, May 29, 2014 at 6:43 PM, Chengwei Yang wrote: > Hi Jun, > > I think I was scared by these exceptions, the producer and consumer are > still alive, seems reconnect to an

Re: producer & consumer fail if the leader failed-over

That's just a warning and the producer will retry. Was there an error (which means all retries failed)? Thanks, Jun On Thu, May 29, 2014 at 6:06 PM, Chengwei Yang wrote: > Hi Jun, > > Thanks for your reply. > > On Thu, May 29, 2014 at 07:52:11AM -0700, Jun Rao wrote: > > You need to pass in m

Re: mBean to monitor message per partitions in topic

If you open up jmx (e.g. jconsole) in a consumer instance, you will see the jmx of name *-ConsumerLag. Thanks, Jun On Thu, May 29, 2014 at 8:53 AM, Рябков Алексей Николаевич < a.ryab...@ntc-vulkan.ru> wrote: > Do you mean getOffsetLag ? ( > https://cwiki.apache.org/confluence/display/KAFKA/Ope

Re: Exception kafka.common.NotLeaderForPartitionException

Bhavesh, This exception is usually normal and transient upon broker restarts. Do you still see them after restarting the brokers for a while? Guozhang On Thu, May 29, 2014 at 6:02 PM, Bhavesh Mistry wrote: > Hi Kafka User Group, > > We have recently changed the “message.max.bytes” to 2MB and

Re: another question about Synchronous producer

Libo, No there will be no ACKs sent back, messages may failed to reach the broker but producers will not be notified in this case. Guozhang On Thu, May 29, 2014 at 5:24 PM, Libo Yu wrote: > Hi, > > Assume I am using a synchronous producer and request.required.acks is set > to 0 (default). > W

Re: question about synchronous producer

Libo, That is correct. You may want to increase the retry.backoff.ms in this case. In practice, if the brokers are down for too long, then data loss is usually inevitable. Guozhang On Thu, May 29, 2014 at 2:55 PM, Libo Yu wrote: > Hi team, > > Assume I am using a synchronous producer and it h

Re: Get a list of consumer groups (for ConsumerOffsetChecker)

Ryan, We do not have this support in zk tools yet, could you file a JIRA for this? Guozhang On Thu, May 29, 2014 at 7:22 PM, Ryan Williams wrote: > I'm doing some initial exploration of kafka and the system tools. With the > console producer and consumers running, I'd like to run the > Consu

Get a list of consumer groups (for ConsumerOffsetChecker)

I'm doing some initial exploration of kafka and the system tools. With the console producer and consumers running, I'd like to run the ConsumerOffsetChecker, but it requires a group parameter. I don't know what to put there. Is there a tool I can run to get a list of consumer groups? I see a hint

Message lost during leader fail-over

Hi Jun, I think I was scared by these exceptions, the producer and consumer are still alive, seems reconnect to another broker though it through a lot of exceptions. However I find another issue seems if the consumer write a message just after the leader broker killed, the message will lost, mean

Re: producer & consumer fail if the leader failed-over

Hi Jun, Thanks for your reply. On Thu, May 29, 2014 at 07:52:11AM -0700, Jun Rao wrote: > You need to pass in multiple brokers in --broker-list in the producer. To > obtain the new leaders, the producer needs to talk to a live broker to get > the new metadata. Then I did tried with with 3 broker

Exception kafka.common.NotLeaderForPartitionException

Hi Kafka User Group, We have recently changed the “message.max.bytes” to 2MB and rebooted all the brokers at once. After this, we are getting “kafka.common.NotLeaderForPartitionException” on both Producer side and consumer side. How do I fix this, I have restarted producer and consumer appli

Re: Producer Side Metric Details

We have 4 producers (for 4 topics) within one JVM and all of them configured with same broker list. Each one of the producer has client.id configured as topic name. When I see the JMX console, I see that " *ProducerRequestMetric*" JMX have issue, it takes the last client.id config and report the

another question about Synchronous producer

Hi, Assume I am using a synchronous producer and request.required.acks is set to 0 (default). When java api Producer.send(message) returns, is the message acknowledged? Is there any chance of message loss in the case? Thanks. Libo

question about synchronous producer

Hi team, Assume I am using a synchronous producer and it has the following default properties: message.send.max.retries 3 retry.backoff.ms 100 I use java api Producer.send(message) to send a message. While send() is being called, if the brokers are shutdown, what happens? send() wi

HA: how to control assign policy for consumers

Well may be yes and may no I think that in design we must point out that co-ordinator also must analize connection graph and must map consumers to broker with minimal network overhead... -Исходное сообщение- От: Guozhang Wang [mailto:wangg...@gmail.com] Отправлено: Thursday, May

Re: how to control assign policy for consumers

In the new consumer design the partitioning algorithm can be customized by the user: https://cwiki.apache.org/confluence/display/KAFKA/Kafka+0.9+Consumer+Rewrite+Design Would that fit your use case? Guozhang On Thu, May 29, 2014 at 8:59 AM, Рябков Алексей Николаевич < a.ryab...@ntc-vulkan.ru>

Re: Job opportunity for someone learning Kafka

Please use job boards /mailing lists for opportunities and let us keep this forum for technical issues. 2014-05-29 9:09 GMT-07:00 Jason Gordon : > Hi All, > > > > I’m looking to connect with someone who’s got about 6 months of Kafka > experience & looking to build on it with a contract opportun

Job opportunity for someone learning Kafka

Hi All, I'm looking to connect with someone who's got about 6 months of Kafka experience & looking to build on it with a contract opportunity I have. Please let me know if you'd be interested. Thanks, Jason Gordon Technical Recruiter Red Oak Technologies, Inc. 2105 S. Bascom Ave;

HA: how to control assign policy for consumers

May be right place to add such optimization within https://cwiki.apache.org/confluence/display/KAFKA/Hierarchical+Topics ? -Исходное сообщение- От: Guozhang Wang [mailto:wangg...@gmail.com] Отправлено: Wednesday, May 28, 2014 8:57 PM Кому: users@kafka.apache.org Тема: Re: how to control

HA: mBean to monitor message per partitions in topic

Do you mean getOffsetLag ? (https://cwiki.apache.org/confluence/display/KAFKA/Operations) But its all about offset not message... So for example to create "real" load balance between broker I can: 1. Calculate average unread message in topic per broker: - Sall=sum(kafka.BrokerAllTopicSta

Re: How to get last message

There is no direct way. You can try getting the last offset (the offset of the next to be appended message) using the getOffsetBefore api and then using that offset - 1 to fetch. Thanks, Jun On Wed, May 28, 2014 at 5:24 PM, hsy...@gmail.com wrote: > Is there a way to get the last message of a

Re: producer & consumer fail if the leader failed-over

You need to pass in multiple brokers in --broker-list in the producer. To obtain the new leaders, the producer needs to talk to a live broker to get the new metadata. Thanks, Jun On Thu, May 29, 2014 at 2:38 AM, Chengwei Yang wrote: > Hi List, > > I'm new to kafka, feel sorry if this has been

Re: Geeting Payload back from Kafka message

Have you looked at https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example ? Thanks, Jun On Wed, May 28, 2014 at 10:19 PM, Kumar Pradeep wrote: > Hi, > I am having trouble in getting my payload message back from the Kafka > Message in my consumer. > My producer is passing a b

Re: producer & consumer fail if the leader failed-over

Hi Chengwei, What do you see in the Kafka server logs? This may help you diagnose the failure. Cheers, Robert On Thu, May 29, 2014 at 2:38 AM, Chengwei Yang wrote: > Hi List, > > I'm new to kafka, feel sorry if this has been asked, I didn't figure out > my answer by googling, so asked here.

Re: running on scala 2.11

just simply tried artifact kafka_2.10:0.8.1.1 as suggested in https://issues.apache.org/jira/browse/KAFKA-1454 but got the error written above now I've cloned the repo and compiled with ./gradlew jar => success ./gradlew -PscalaVersion=2.10.0 jar => success ./gradlew -PscalaVersion=2.11.0 jar =>

producer & consumer fail if the leader failed-over

Hi List, I'm new to kafka, feel sorry if this has been asked, I didn't figure out my answer by googling, so asked here. Thanks in advance! I'm following the kafka quick start http://kafka.apache.org/documentation.html#quickstart and managed setup a kafka cluster with two brokers, which connected