kafka 0.8 producer when zookeeper 3.4.6 is shut down

,zk1b:2181,zk1c:2181". Let's say that the consumer is connected to zk1a, what happens when zk1a is suddenly unavailable? Will the zookeeper client ver 3.4.6 bundled with kafka 0.8 automatically retry and attempt to connect to one of the other zookeepers in the connect string? Let's keep

Re: Consumer api 0.10 backwards compatible with Kafka 0.8

Thanks so much to both of you. Vadim On Wed, Sep 7, 2016 at 2:07 AM, Ismael Juma wrote: > Hi Vadim, > > You have to upgrade the brokers first and then the clients. So, you can use > 0.8 clients with 0.10 brokers, but _not_ 0.10 clients with 0.8 brokers. > > On Wed, Sep 7, 2016 at 9:19 AM, Vadim

Re: Consumer api 0.10 backwards compatible with Kafka 0.8

Hi Vadim, You have to upgrade the brokers first and then the clients. So, you can use 0.8 clients with 0.10 brokers, but _not_ 0.10 clients with 0.8 brokers. On Wed, Sep 7, 2016 at 9:19 AM, Vadim Keylis wrote: > Hello. We are running kafka server 0.8. We are considering upgrading to the > lates

Re: Consumer api 0.10 backwards compatible with Kafka 0.8

I believe there were some incompatibilities between 0.8.x and 0.9.x for the clients. If you are upgrading from 0.8.x to 0.10.x, you should check out the upgrade guide available here: http://kafka.apache.org/documentation.html#upgrade_10 Cheers, Francis On 7/09/2016 6:19 PM, Vadim Keylis wrot

Consumer api 0.10 backwards compatible with Kafka 0.8

Hello. We are running kafka server 0.8. We are considering upgrading to the latest version of kafka. Does latest consumer 0.10 backward compatible with older version of Kafka? What is the right approach of upgrading Kafka servers and consumers? Thanks in advance.

Re: Kafka 0.8 old high level consumer client gets inactive after few hours of inactivity

- To understand what the stuck consumer is doing, it would be useful to collect the logs and a thread dump. I'd try to find out what the fetcher threads are doing. What about the handler/application/stream threads? - are the offsets committed? After a restart, could it be that the consumer is just

Kafka 0.8 old high level consumer client gets inactive after few hours of inactivity

Hi, I'm using kafka version 0.8.2.1 with old high level consumer api. I'm following this gude: https://cwiki.apache.org/confluence/display/KAFKA/Consumer+Group+Example The client is working fine but after few hours of inactivity it gets inactive and stops receiving messages. but if I restart my cl

Re: kafka 0.8 producer issue

For setting up logs for kafka, we need to export KAFKA_LOG4J_OPS env variable before starting the kafka servers. if log4j.properties is presented in /etc/kafka/ directory then it will be directly picks otherwise it uses tools-log4j.properties from /etc/kafka. You can set the location of server log

Re: kafka 0.8 producer issue

Can u share the code that you are using to publish the message. Also can u whether a small message is.published On Nov 25, 2015 9:25 PM, "Kudumula, Surender" wrote: > Hi all > I am trying to get the producer working. It was working before but now > getting the following issue. I have created a ne

Re: kafka 0.8 producer issue

Hi, >From the information that you've provided, I think your callback is the culprit here. It is seen from the stacktrace:- at com.hpe.ssmamp.kafka.KafkaPDFAProducer$1.onCompletion( KafkaPDFAProducer.java:62) Please provide more information like a code snippet etc, so that we can tell more

RE: kafka 0.8 producer issue

Message- From: Prabhjot Bharaj [mailto:prabhbha...@gmail.com] Sent: 25 November 2015 17:35 To: users@kafka.apache.org Cc: d...@kafka.apache.org Subject: Re: kafka 0.8 producer issue Hi, From the information that you've provided, I think your callback is the culprit here. It is seen fro

kafka 0.8 producer issue

Hi all I am trying to get the producer working. It was working before but now getting the following issue. I have created a new topic as well just in case if it was the issue with topic but still no luck. I have increased the message size in broker as iam trying to send atleast 3mb message here

Re: kafka 0.8 consumer polling topic

You can find the Java doc with some examples under "KafkaConsumer" here: http://people.apache.org/~nehanarkhede/kafka-0.9-consumer-javadoc/doc/ Guozhang On Thu, Oct 22, 2015 at 1:28 PM, Kudumula, Surender < surender.kudum...@hpe.com> wrote: > Hi all > > General question does the current kafka

kafka 0.8 consumer polling topic

Hi all General question does the current kafka consumer needs to be written in java threads in order to poll the topic continuously and how should it be written any ideas please? Thanks Regards Surender Kudumula Big Data Consultant - EMEA Analytics & Data Management surender.kudum...@hpe.com

RE: Is there a complete Kafka 0.8.* replication design document

: Jun Rao [mailto:j...@confluent.io] Sent: April 09, 2015 4:19 To: Jason Guo (jguo2) Cc: users@kafka.apache.org; d...@kafka.apache.org Subject: Re: Is there a complete Kafka 0.8.* replication design document #2. /brokers/topics/[topic] stores the replica assignment for all partitions in the topic

Re: Is there a complete Kafka 0.8.* replication design document

in > *green* > > > -Original Message- > From: Jun Rao [mailto:j...@confluent.io ] > Sent: April 08, 2015 5:04 > To: users@kafka.apache.org > Cc: d...@kafka.apache.org > Subject: Re: Is there a complete Kafka 0.8.* replication design document > > Yes, the wiki

RE: Is there a complete Kafka 0.8.* replication design document

Thanks for your response and I have some other questions as below in green -Original Message- From: Jun Rao [mailto:j...@confluent.io] Sent: April 08, 2015 5:04 To: users@kafka.apache.org Cc: d...@kafka.apache.org Subject: Re: Is there a complete Kafka 0.8.* replication design document

Re: Is there a complete Kafka 0.8.* replication design document

nager. Thanks, Jun On Mon, Apr 6, 2015 at 7:51 PM, Jason Guo (jguo2) wrote: > Hi, > > These days I have been focus on Kafka 0.8 replication design and > found three replication design proposals from the wiki (according to the > document, the V3 version is used

Is there a complete Kafka 0.8.* replication design document

Hi, These days I have been focus on Kafka 0.8 replication design and found three replication design proposals from the wiki (according to the document, the V3 version is used in Kafka 0.8 release). But the v3 proposal is not complete and is inconsistent with the release. Is

Re: Is there a complete Kafka 0.8.* replication design document

nager. Thanks, Jun On Mon, Apr 6, 2015 at 6:12 AM, Jason Guo (jguo2) wrote: > Hi, > > These days I have been focus on Kafka 0.8 replication design and > found three replication design proposals from the wiki (according to the > document, the V3 version is used

Is there a complete Kafka 0.8.* replication design document

Hi, These days I have been focus on Kafka 0.8 replication design and found three replication design proposals from the wiki (according to the document, the V3 version is used in Kafka 0.8 release). But the v3 proposal is not complete and is inconsistent with the release. Is

Re: how to do disaster recovery for kafka 0.8 cluster with consumers that uses high-level consumer api?

at 8:09 PM, Jun Rao wrote: > > > Did you set the replication factor to be more than 1? > > > > Thanks, > > > > Jun > > > > On Sat, Oct 18, 2014 at 2:32 AM, Yu Yang wrote: > > > > > Hi all, > > > > > > We have a kafka 0.8.1 c

Re: how to do disaster recovery for kafka 0.8 cluster with consumers that uses high-level consumer api?

gt; > We have a kafka 0.8.1 cluster. We implemented a consumers for the topics > on > > the Kafka 0.8 cluster using high-level consumer api. We observed that if > > the Kafka cluster was down and got rebooted and the consumer was running, > > the consumer will fail to read a

Re: how to do disaster recovery for kafka 0.8 cluster with consumers that uses high-level consumer api?

Did you set the replication factor to be more than 1? Thanks, Jun On Sat, Oct 18, 2014 at 2:32 AM, Yu Yang wrote: > Hi all, > > We have a kafka 0.8.1 cluster. We implemented a consumers for the topics on > the Kafka 0.8 cluster using high-level consumer api. We observed that if

how to do disaster recovery for kafka 0.8 cluster with consumers that uses high-level consumer api?

Hi all, We have a kafka 0.8.1 cluster. We implemented a consumers for the topics on the Kafka 0.8 cluster using high-level consumer api. We observed that if the Kafka cluster was down and got rebooted and the consumer was running, the consumer will fail to read a few topic partitions due to

Php client for Kafka 0.8

Hi folks, One of things stopping us from upgrading version of Kafka from 0.7 to 0.8 is the absence of a native PHP client. A search points to this: https://github.com/salebab/phpkafka. Does anyone have experience with it ? We require a PHP client due to high throughput, low latency existing syste

Re: Announce: Capillary, a monitor for Kafka 0.8 spout topologies -- and the upgrade to 0.8

e was that Kafka 0.8 is basically a whole new system. Since the only real way to upgrade is to set up a new cluster things were obviously harder than install & reboot. :) - We had to read a lot of the Storm Spout code to understand how things work with leadership and replication. We've n

Re: Announce: Capillary, a monitor for Kafka 0.8 spout topologies -- and the upgrade to 0.8

e/display/KAFKA/Ecosystem > >Philip > >  >  >- >http://www.philipotoole.com > > > >On Wednesday, August 20, 2014 9:32 AM, Cory Watson wrote: > > > >Hello all! > >If you use Storm and Kafka 0.8 together I may have an interesting project >for you. 

Re: Announce: Capillary, a monitor for Kafka 0.8 spout topologies

s://cwiki.apache.org/confluence/display/KAFKA/Ecosystem > > Philip > > > > - > http://www.philipotoole.com > > > On Wednesday, August 20, 2014 9:32 AM, Cory Watson wrote: > > > > Hello all! > > If you use Storm a

Re: Announce: Capillary, a monitor for Kafka 0.8 spout topologies

ay, August 20, 2014 9:32 AM, Cory Watson wrote: Hello all! If you use Storm and Kafka 0.8 together I may have an interesting project for you.  Recently at Keen IO <https://keen.io/> we upgraded from Kafka 0.7 to 0.8 and needed to replace the features of stormkafkamon <https:/

Announce: Capillary, a monitor for Kafka 0.8 spout topologies

Hello all! If you use Storm and Kafka 0.8 together I may have an interesting project for you. Recently at Keen IO <https://keen.io/> we upgraded from Kafka 0.7 to 0.8 and needed to replace the features of stormkafkamon <https://github.com/otoolep/stormkafkamon> for monitoring spout o

Re: Kafka 0.8 automatically changing leadership

rez > > > > > > On Fri, Aug 1, 2014 at 4:53 PM, Joel Koshy wrote: > > > > > > Leadership moves automatically for at least a few of the topics, > which > > > > never happens when we run them on our prod, non-AWS hardware. This > > > causes > > &g

Re: Kafka 0.8 automatically changing leadership

ailures) leadership > > should not move. Leader changes occur when brokers fail - due to GC, > > controlled shutdowns/bounces, or network partitioning. > > > > Do you see any "Broker failure callback" message on the controller > > log? Can you also check your broker lo

Re: Kafka 0.8 automatically changing leadership

r failures) leadership > should not move. Leader changes occur when brokers fail - due to GC, > controlled shutdowns/bounces, or network partitioning. > > Do you see any "Broker failure callback" message on the controller > log? Can you also check your broker logs to see if

Re: Kafka 0.8 automatically changing leadership

Hi, > > We have a Kafka 0.8 cluster in a test environment (in this case, on AWS EC2 > nodes). Even though we've tried to run very little load on this test > cluster, it seems like the instances can't even keep up with that. > Leadership moves automatically for at least a f

Kafka 0.8 automatically changing leadership

Hi, We have a Kafka 0.8 cluster in a test environment (in this case, on AWS EC2 nodes). Even though we've tried to run very little load on this test cluster, it seems like the instances can't even keep up with that. Leadership moves automatically for at least a few of the topics, w

Re: Syslog Collector to Kafka 0.8 -- in Go

Awesome, thanks Philip! I updated the ecosystem page https://cwiki.apache.org/confluence/display/KAFKA/Ecosystem /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop

Syslog Collector to Kafka 0.8 -- in Go

I went looking for a Syslog Collector, written in Go, which would stream to Kafka. I couldn't find any, so put one together myself -- others might be interested. It optionally performs basic parsing of an RFC5424 header too, before streaming the messages to Kafka. As always, YMMV. https://gith

Re: Kafka 0.8.x failover with multiple data centers

get an older offset to start with, and dedup messages at the application level. Guozhang On Mon, Jul 7, 2014 at 7:54 AM, Janos Mucza wrote: > Dear Kafka Users, > > I would like to use Kafka 0.8.x in a multi-cluster environment so that > when my primary cluster fails, producers a

Kafka 0.8.x failover with multiple data centers

Dear Kafka Users, I would like to use Kafka 0.8.x in a multi-cluster environment so that when my primary cluster fails, producers and consumers could switch to the secondary cluster. Clusters would be hosted in different data centers. A possibility would be mirroring topics (similar to Kafka

Re: Rolling update Kafka 0.8 -> Kafka 0.8.1.1 in detail

uot;A", "B", "C"]. But brokers 2 and 3 didn't have a chance to fetch > > "B", > > > so their log looks like ["A", "C"]. Will Kafka try to fill the gap in > the > > > background once broker 1 started over? > > > > >

Re: Rolling update Kafka 0.8 -> Kafka 0.8.1.1 in detail

;]. Will Kafka try to fill the gap in the > > background once broker 1 started over? > > > > > > 2014-06-18 19:59 GMT+04:00 Neha Narkhede : > > > > > You don't gain much by running #4 between broker bounces. Running it > > after > > > the

Re: Rolling update Kafka 0.8 -> Kafka 0.8.1.1 in detail

Neha Narkhede : > > > You don't gain much by running #4 between broker bounces. Running it > after > > the cluster is upgraded will be sufficient. > > > > Thanks, > > Neha > > > > > > On Wed, Jun 18, 2014 at 8:33 AM, Yury Ruchin > >

Re: Rolling update Kafka 0.8 -> Kafka 0.8.1.1 in detail

e gap in the background once broker 1 started over? 2014-06-18 19:59 GMT+04:00 Neha Narkhede : > You don't gain much by running #4 between broker bounces. Running it after > the cluster is upgraded will be sufficient. > > Thanks, > Neha > > > On Wed, Jun 18, 2014 at 8

Re: Kafka 0.8's VerifyConsumerRebalance reports an error

This is a bug in the tool. Please file a bug and attach these error/info logs to it. Thanks, Neha On Thu, Jun 26, 2014 at 5:24 AM, Yury Ruchin wrote: > I have set log level to DEBUG and saw something strange in the output. For > each topic partition, I see the following pattern: > > [2014-06-2

Re: Kafka 0.8's VerifyConsumerRebalance reports an error

I have set log level to DEBUG and saw something strange in the output. For each topic partition, I see the following pattern: [2014-06-26 16:00:24,467] ERROR No owner for partition [,0] (kafka.tools.VerifyConsumerRebalance$) ... [2014-06-26 16:00:24,469] INFO Owner of partition [,0] is _-1403049

Re: Kafka 0.8/VIP/SSL

is possible in your setup. Thanks, Neha On Tue, Jun 24, 2014 at 3:35 PM, Reiner Stach wrote: > I'm looking for advice on running Kafka 0.8 behind VIPs. The goal is to > support SSL traffic, with encryption and decryption being performed by > back-to-back VIPs at the client and i

Kafka 0.8/VIP/SSL

I'm looking for advice on running Kafka 0.8 behind VIPs. The goal is to support SSL traffic, with encryption and decryption being performed by back-to-back VIPs at the client and in front of the broker. That is: Kafka client --> vip1a.myco.com:8080 (SSL encrypt) --- WAN --->

Re: Kafka 0.8's VerifyConsumerRebalance reports an error

I would turn on DEBUG on the tool to see which url it reads and doesn't find the owners. On Tue, Jun 24, 2014 at 11:28 AM, Yury Ruchin wrote: > I've just double-checked. The URL is correct, the same one is used by Kafka > clients. > > > 2014-06-24 22:21 GMT+04:00 Neha Narkhede : > > > Is it p

Re: Kafka 0.8's VerifyConsumerRebalance reports an error

I've just double-checked. The URL is correct, the same one is used by Kafka clients. 2014-06-24 22:21 GMT+04:00 Neha Narkhede : > Is it possible that maybe the zookeeper url used for the > VerifyConsumerRebalance tool is incorrect? > > > On Tue, Jun 24, 2014 at 12:02 AM, Yury Ruchin > wrote: >

Re: Kafka 0.8's VerifyConsumerRebalance reports an error

Is it possible that maybe the zookeeper url used for the VerifyConsumerRebalance tool is incorrect? On Tue, Jun 24, 2014 at 12:02 AM, Yury Ruchin wrote: > Hi, > > I've run into the following problem. I try to read from a 50-partition > Kafka topic using high level consumer with 8 streams. I'm u

Kafka 0.8's VerifyConsumerRebalance reports an error

Hi, I've run into the following problem. I try to read from a 50-partition Kafka topic using high level consumer with 8 streams. I'm using 8-thread pool, each thread handling one stream. After a short time, the threads reading from the stream stop reading. Lag between topic latest offset and the c

Re: Rolling update Kafka 0.8 -> Kafka 0.8.1.1 in detail

You don't gain much by running #4 between broker bounces. Running it after the cluster is upgraded will be sufficient. Thanks, Neha On Wed, Jun 18, 2014 at 8:33 AM, Yury Ruchin wrote: > Hi folks, > > In my project, we want to perform to update our active Kafka 0.8 cluster to &

Rolling update Kafka 0.8 -> Kafka 0.8.1.1 in detail

Hi folks, In my project, we want to perform to update our active Kafka 0.8 cluster to Kafka 0.8.1.1 without downtime and losing any data. The process (after reading http://kafka.apache.org/documentation.html#upgrade) looks to me like this. For each broker in turn: 1. Bring the broker down. 2

Re: kafka-storm-starter released: code examples that integrate Kafka 0.8 and Storm 0.9

Cool. Thanks for sharing. I added it to our ecosystem wiki. -Neha On Fri, May 23, 2014 at 8:02 AM, Michael G. Noll < michael+ka...@michael-noll.com> wrote: > Hi everyone, > > to sweeten the upcoming long weekend I have released code examples that > show how to integrate Kaf

kafka-storm-starter released: code examples that integrate Kafka 0.8 and Storm 0.9

Hi everyone, to sweeten the upcoming long weekend I have released code examples that show how to integrate Kafka 0.8+ with Storm 0.9+, while using Apache Avro as the data serialization format. https://github.com/miguno/kafka-storm-starter Since the integration of the latest Kafka and Storm

Re: Kafka 0.8's ConsumerConnector.close() hangs if ZK is unavailable and autocommit is enabled

by design that to make consistent offset maintenance, ZK should be up all the time. Guozhang On Mon, May 12, 2014 at 9:59 AM, Yury Ruchin wrote: > Hi all, > > I'm using Kafka 0.8 and I've ran into an issue with ConsumerConnector. > Steps to reproduce: > > 1. Start

Re: Kafka 0.8's ConsumerConnector.close() hangs if ZK is unavailable and autocommit is enabled

In 0.8, ZK is critical for managing the brokers. So, we do expect that ZK service to be always available. Thanks, Jun On Mon, May 12, 2014 at 9:59 AM, Yury Ruchin wrote: > Hi all, > > I'm using Kafka 0.8 and I've ran into an issue with ConsumerConnector. > Steps to r

Kafka 0.8's ConsumerConnector.close() hangs if ZK is unavailable and autocommit is enabled

Hi all, I'm using Kafka 0.8 and I've ran into an issue with ConsumerConnector. Steps to reproduce: 1. Start single-broker Kafka cluster with auto.create.topic.enable set to "true" 2. Start ConsumerConnector on topic (which does not yet exist) with auto.offset.reset set to &q

Re: Migration Tool consumer rebalance error when attempting to migrate from Kafka 0.7 to Kafka 0.8

from Kafka 0.7 to 0.8, and the first stage of this > is to use the Migration Tool. > > I have setup a local Kafka 0.8 cluster consisting of 3 nodes on my local > Vagrant (along with a zookeeper running locally). The Migration Tool > Producer will connect to these three nodes in order

Re: Migration Tool consumer rebalance error when attempting to migrate from Kafka 0.7 to Kafka 0.8

erConnector.scala:432) > at > > kafka.consumer.ZookeeperConsumerConnector.kafka$consumer$ZookeeperConsumerConnector$$reinitializeConsumer(ZookeeperConsumerConnector.scala:722) > at > > kafka.consumer.ZookeeperConsumerConnector$WildcardStreamsHandler.(ZookeeperConsumerConnector.scal

Re: Migration Tool consumer rebalance error when attempting to migrate from Kafka 0.7 to Kafka 0.8

avaapi.consumer.ZookeeperConsumerConnector.createMessageStreamsByFilter(ZookeeperConsumerConnector.scala:100) ... 5 more [2014-03-19 17:35:40,783] DEBUG Got ping response for sessionid: 0x3448da1f8f0644b after 7ms (org.apache.zookeeper.ClientCnxn) [2014-03-19 17:35:42,786] DEBUG Got ping response

Re: Migration Tool consumer rebalance error when attempting to migrate from Kafka 0.7 to Kafka 0.8

; Jun > > > On Wed, Mar 19, 2014 at 7:07 AM, Mo Firouz > wrote: > > > Hello. > > > > I am trying to migrate from Kafka 0.7 to 0.8, and the first stage of this > > is to use the Migration Tool. > > > > I have setup a local Kafka 0.8 cluster consisti

Re: Migration Tool consumer rebalance error when attempting to migrate from Kafka 0.7 to Kafka 0.8

Tool. > > I have setup a local Kafka 0.8 cluster consisting of 3 nodes on my local > Vagrant (along with a zookeeper running locally). The Migration Tool > Producer will connect to these three nodes in order to replicate the data. > The Migration Tool Consumer(s) will connect to

Migration Tool consumer rebalance error when attempting to migrate from Kafka 0.7 to Kafka 0.8

Hello. I am trying to migrate from Kafka 0.7 to 0.8, and the first stage of this is to use the Migration Tool. I have setup a local Kafka 0.8 cluster consisting of 3 nodes on my local Vagrant (along with a zookeeper running locally). The Migration Tool Producer will connect to these three nodes

Re: Kafka 0.8: delete topic with data

ease but I think it will be stable only in 0.8.2. Thanks, Neha On Thu, Feb 27, 2014 at 1:16 AM, Yury Ruchin wrote: > Hi, > > I'm using Kafka 0.8 which does not have a command to delete topic. However, > I need the functionality and I'm trying to adopt this approach: > >

Re: Kafka-0.8 Log4j Appender

e a JIRA? > > > > > > On Tue, Feb 25, 2014 at 9:51 PM, 김동경 wrote: > > > > > Dear all. > > > > > > Are there anyone who tried running Kafka-0.8 Log4j Appender? > > > I want to send my application log into Kafka via Log4j Appender. > >

Kafka 0.8: delete topic with data

Hi, I'm using Kafka 0.8 which does not have a command to delete topic. However, I need the functionality and I'm trying to adopt this approach: https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/admin/DeleteTopicCommand.scala. I see it simply deletes the topic node f

Re: Kafka-0.8 Log4j Appender

essage send logs an error that in turn tries to resend the message and > gets into this infinite loop. Could you file a JIRA? > > > On Tue, Feb 25, 2014 at 9:51 PM, 김동경 wrote: > > > Dear all. > > > > Are there anyone who tried running Kafka-0.8 Log4j Appender? >

Re: Kafka-0.8 Log4j Appender

who tried running Kafka-0.8 Log4j Appender? > I want to send my application log into Kafka via Log4j Appender. > > > Here is my log4j.properties. > I couldn`t find any proper encoder, so I just configure it to use default > encoder.

Kafka-0.8 Log4j Appender

Dear all. Are there anyone who tried running Kafka-0.8 Log4j Appender? I want to send my application log into Kafka via Log4j Appender. Here is my log4j.properties. I couldn`t find any proper encoder, so I just configure it to use default encoder. (e.g I commented the line

Re: Delay in fetching messages - High level consumer -Kafka 0.8

i, I am using kafka 0.8. I have 3 brokers on three systems and 3 zookeepers running. I am using the high level consumer which is in examples folder of kafka. I am able to push the messages into the queue, but retriving the messages is taking some time. Is there any way i can tune this. I get this inf

Re: Delay in fetching messages - High level consumer -Kafka 0.8

ks Arjun Narasimha kota On Tuesday 25 February 2014 03:45 PM, Arjun wrote: Hi, I am using kafka 0.8. I have 3 brokers on three systems and 3 zookeepers running. I am using the high level consumer which is in examples folder of kafka. I am able to push the messages into the queue, but retriving the

Re: Delay in fetching messages - High level consumer -Kafka 0.8

.server.AbstractFetcherThread.doWork( > AbstractFetcherThread.scala:86) > at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51) > > > Does it effect anything. I havent looked at it as it was just a warning. > Should i be worried about this? > > > Thanks > Arj

Re: Delay in fetching messages - High level consumer -Kafka 0.8

>>> SimpleConsumer.scala:109) >>> at kafka.consumer.SimpleConsumer$$anonfun$fetch$1.apply( >>> SimpleConsumer.scala:109) >>> at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33) >>> at kafka.consumer.SimpleConsumer.fetch(SimpleConsumer.scala:108) &

Re: Delay in fetching messages - High level consumer -Kafka 0.8

worried about this? Thanks Arjun Narasimha kota On Tuesday 25 February 2014 03:45 PM, Arjun wrote: Hi, I am using kafka 0.8. I have 3 brokers on three systems and 3 zookeepers running. I am using the high level consumer which is in examples folder of kafka. I am able to push the messages i

Re: Delay in fetching messages - High level consumer -Kafka 0.8

it as it was just a warning. Should i be worried about this? Thanks Arjun Narasimha kota On Tuesday 25 February 2014 03:45 PM, Arjun wrote: Hi, I am using kafka 0.8. I have 3 brokers on three systems and 3 zookeepers running. I am using the high level consumer which is in examples folder of

Re: Delay in fetching messages - High level consumer -Kafka 0.8

at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:51) Does it effect anything. I havent looked at it as it was just a warning. Should i be worried about this? Thanks Arjun Narasimha kota On Tuesday 25 February 2014 03:45 PM, Arjun wrote: Hi, I am using kafka 0.8. I have 3 brokers on three systems an

Delay in fetching messages - High level consumer -Kafka 0.8

Hi, I am using kafka 0.8. I have 3 brokers on three systems and 3 zookeepers running. I am using the high level consumer which is in examples folder of kafka. I am able to push the messages into the queue, but retriving the messages is taking some time. Is there any way i can tune this. I

Re: offset asked for broker is more than what broker has when does this happen - High level consumer -kafka 0.8

Not sure what happened. It could be that the broker received messages with offset 5 to 10 at one time, but lost them later during unclean leader election. If this is the case, you will see sth like "No broker in ISR is alive for %s. Elect leader %d from live brokers %s. There's potential data loss.

offset asked for broker is more than what broker has when does this happen - High level consumer -kafka 0.8

Hi, I have set up a clean kafka set up on three ec2 nodes. I have pushed in two messages into the set up. But i invoke the high level consumer, i get this error in the kafka broker. I ran the Consumer offset tool. It says the lag to be -6. This set up is fresh set up and it has three kafka br

Re: Kafka : 0.8 beta --> 0.8 release upgrade

Yes, you can do an in-place upgrade. We probably made a minor protocol change btw the controller and the brokers (can't remember for sure though). So, it's safer to upgrade the controller the last. Thanks, Jun On Sat, Feb 8, 2014 at 10:24 AM, Banerjee, Aparup wrote: > I understand from other p

Kafka : 0.8 beta --> 0.8 release upgrade

I understand from other posts , that is can be an in-place upgrade. Is there some documentation describing the same? Thanks, Aparup

Re: only one ProducerSendThread thread when running with multiple brokers (kafka 0.8)

ges to multiple brokers, the send thread will do > > that > > > > serially, which will reduce the throughput. We plan to improve that > in > > > 0.9 > > > > through client rewrites. For now, you can improve the throughput by > > > either > &g

Re: only one ProducerSendThread thread when running with multiple brokers (kafka 0.8)

ughput. We plan to improve that in > > 0.9 > > > through client rewrites. For now, you can improve the throughput by > > either > > > using a larger batch size or using more producer instances. > > > > > > As for degraded performance with more partitions,

Re: only one ProducerSendThread thread when running with multiple brokers (kafka 0.8)

e plan to improve that in > 0.9 > > through client rewrites. For now, you can improve the throughput by > either > > using a larger batch size or using more producer instances. > > > > As for degraded performance with more partitions, are you specifying a > key >

Re: only one ProducerSendThread thread when running with multiple brokers (kafka 0.8)

u specifying a key > for each message? > > Thanks, > > Jun > > On Wed, Jan 1, 2014 at 4:17 AM, yosi botzer wrote: > > > Hi, > > > > I am using kafka 0.8. I have 3 machines each running kafka broker. > > > > I am using async mode o

Re: only one ProducerSendThread thread when running with multiple brokers (kafka 0.8)

Hi, > > I am using kafka 0.8. I have 3 machines each running kafka broker. > > I am using async mode of my Producer. I expected to see 3 different threads > with names starting with ProducerSendThread- (according to this article: > http://engineering.gnip.com/kafka-async-producer/)

Re: only one ProducerSendThread thread when running with multiple brokers (kafka 0.8)

er (I am using > the > > > > async > > > > > > > mode)? Why do I get a better performance sending a message > that > > > has > > > > > > fewer > > > > > > > partitions? > > > > > > > > > > > >

Re: only one ProducerSendThread thread when running with multiple brokers (kafka 0.8)

get a better performance sending a message that > > has > > > > > fewer > > > > > > partitions? > > > > > > > > > > > > > > > > > > On Wed, Jan 1, 2014 at 2:22 PM, Gerrit Jansen van Vuuren < > > > > > > gerrit...@gmail.com&g

Re: only one ProducerSendThread thread when running with multiple brokers (kafka 0.8)

Jansen van Vuuren < > > > > > gerrit...@gmail.com> wrote: > > > > > > > > > > > The producer is heavily synchronized (i.e. all the code in the > send > > > > > method > > > > > > is encapsulated in one huge synchro

Re: only one ProducerSendThread thread when running with multiple brokers (kafka 0.8)

ly synchronized (i.e. all the code in the send > > > > method > > > > > is encapsulated in one huge synchronized block). > > > > > Try creating multiple producers and round robin send over them. > > > > > > > > > > e.g. > >

Re: only one ProducerSendThread thread when running with multiple brokers (kafka 0.8)

> > > > > > e.g. > > > > > > > > p = producers[ n++ % producers.length ] > > > > > > > > p.send msg > > > > This will give you one thread per producer instance. > > > > > > > > I'm working on an async mu

Re: only one ProducerSendThread thread when running with multiple brokers (kafka 0.8)

This will give you one thread per producer instance. > > > > > > I'm working on an async multi threaded producer for kafka, but its > > nothing > > > near complete yet. > > > https://github.com/gerritjvv/kafka-fast > > > > > > > > > Regard

Re: only one ProducerSendThread thread when running with multiple brokers (kafka 0.8)

r for kafka, but its > nothing > > near complete yet. > > https://github.com/gerritjvv/kafka-fast > > > > > > Regards, > > Gerrit > > > > > > On Wed, Jan 1, 2014 at 1:17 PM, yosi botzer > wrote: > > > > > Hi, > > >

Re: only one ProducerSendThread thread when running with multiple brokers (kafka 0.8)

stance. > > I'm working on an async multi threaded producer for kafka, but its nothing > near complete yet. > https://github.com/gerritjvv/kafka-fast > > > Regards, > Gerrit > > > On Wed, Jan 1, 2014 at 1:17 PM, yosi botzer wrote: > > > Hi, >

Re: only one ProducerSendThread thread when running with multiple brokers (kafka 0.8)

. I'm working on an async multi threaded producer for kafka, but its nothing near complete yet. https://github.com/gerritjvv/kafka-fast Regards, Gerrit On Wed, Jan 1, 2014 at 1:17 PM, yosi botzer wrote: > Hi, > > I am using kafka 0.8. I have 3 machines each running kafka br

only one ProducerSendThread thread when running with multiple brokers (kafka 0.8)

Hi, I am using kafka 0.8. I have 3 machines each running kafka broker. I am using async mode of my Producer. I expected to see 3 different threads with names starting with ProducerSendThread- (according to this article: http://engineering.gnip.com/kafka-async-producer/) However I can see only

Re: Kafka 0.8 High-Level Consumer: fetcher threads

Hi Jun, Thank you for the quick feedback! here is the issue https://issues.apache.org/jira/browse/KAFKA-1184 -- Best Regards, Evelina On 12/16/2013 08:18 PM, Jun Rao wrote: It's possible to support multi-fetchers per broker in the consumer. Could you file a jira to track that? However, typical

Re: Kafka 0.8 High-Level Consumer: fetcher threads

It's possible to support multi-fetchers per broker in the consumer. Could you file a jira to track that? However, typically the bottleneck is not in the fetcher, but in decompression, decoding and application logic during the consumption. Thanks, Jun On Mon, Dec 16, 2013 at 7:33 AM, Evelina wr

  1   2   3   >