Re: Consumer starts to consume after changing group id

Hello, It looks like it consumes from Kafka. here is the request log : https://gist.github.com/gurkanoluc/39d2762ade4e2148a20c On Thu, Jan 16, 2014 at 7:29 AM, Jun Rao wrote: > It seems that it's using SimpleConsumer. Could you check Kafka broker's > request log and see if the consumer is ma

Re: How to force producer and high level consumer to use different Ethernet cards on the kafka broker node

It seems the answer may lie on the broker and zookeeper sides. Can these guys listen on 2 ports, simultaneously? I believe you can listen on a filtered ip:port, so they could go to 2 NICs on those boxes and then have different props for the producer and consumers to use the different local NIC

Re: compile error sbt / log4j , cannot exclude log4j

try "org.apache.kafka" % "kafka_2.10" % "0.8.0" intransitive() /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop *

compile error sbt / log4j , cannot exclude log4j

Using : SBT 13 Scala 2.10.3 Kafka 0.8.0 Getting a long cryptic compile error ending if I exclude log4j: [error] uncaught exception during compilation: scala.reflect.internal.Types$TypeError [trace] Stack trace suppressed: run last shops-integrationlayer/compile:compile for the full output. [erro

Re: How to force producer and high level consumer to use different Ethernet cards on the kafka broker node

Hi Jun. Just to be clear. Its not possible on single node that has two NIC cards(each NIC card assign its own ip address) to route traffic for producer through one NIC card and consumer through the other one while having kafka listening on the same port for both consumer and producer, correct? Tha

Re: How to force producer and high level consumer to use different Ethernet cards on the kafka broker node

Currently, Kafka broker only supports one port for both the produce and the consumer clients. Thanks, Jun On Wed, Jan 15, 2014 at 4:15 PM, Vadim Keylis wrote: > Hi. We have kafka setup on server that has two Ethernet cards. Each > Ethernet port has its own ip address. We would like to separat

Re: Consumer starts to consume after changing group id

It seems that it's using SimpleConsumer. Could you check Kafka broker's request log and see if the consumer is making the right fetch request? Thanks, Jun On Wed, Jan 15, 2014 at 10:58 AM, Gürkan Oluç wrote: > Hello, > > We are using Kafka 0.8 with Storm. To consume from Kafka, we are using >

Re: python and kafka - how to use as a queue

Hi Arthur, I'm running into a very similar issue even with the latest version ( kafka-python @ V. 0.8.1_1 used with kafka_2.8.0-0.8.0.tar.gz). I have created a topic 'my-topic' with two partitions and 1-replication (across a set of 3 kafka brokers). I've published 100 messages to the topic (see

Re: Kafka-0.7.2 - Launching Topic Terminates Zookeeper Server

Did you build from source or are these downloaded binaries? fyi, can't see the png through the user lists. /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop

RE: Kafka-0.7.2 - Launching Topic Terminates Zookeeper Server

All the zookeeper logs are spitting out binary. After calling: bin/kafka-console-producer.sh --zookeeper localhost:2181 --topic test [cid:image001.png@01CF1222.27E74650] The Zookeeper Terminal says: [cid:image002.png@01CF1222.27E74650] -Original Message- From: Joe Stein [mail

Re: kafka-create-topic.sh missing in Kafka-0.7.2

Kafka 0.7 has no notion of an admin tool to create topics. Topics are created when the first message for that topic is sent to a broker. Similarly, topic can be deleted by shutting down the broker, deleting the log files and restarting the broker. Thanks, Neha On Wed, Jan 15, 2014 at 4:29 PM, w

Re: Kafka-0.7.2 - Launching Topic Terminates Zookeeper Server

Not sure what you mean by launching a Topic. Do you have logs from the error? /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop **

Kafka-0.7.2 - Launching Topic Terminates Zookeeper Server

Can't figure out why this keeps happening. Any Ideas? Joseph Simmons Accenture LLC TGP-CMT Analyst San Francisco, CA B: 415.537.4703 C: 951.941.1125 This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confid

RE: Kafka Authenication and Monitoring Tools

Thank You Otis, I'll check this out. -Original Message- From: Otis Gospodnetic [mailto:otis.gospodne...@gmail.com] Sent: Tuesday, January 14, 2014 9:12 PM To: users@kafka.apache.org Subject: Re: Kafka Authenication and Monitoring Tools Hi Joseph, For the monitoring piece there is SPM - s

kafka-create-topic.sh missing in Kafka-0.7.2

Where is this file? I've can't create any topics. [cid:image001.png@01CF120E.F85E85A0] This message is for the designated recipient only and may contain privileged, proprietary, or otherwise confidential information. If you have received it in error, please n

How to force producer and high level consumer to use different Ethernet cards on the kafka broker node

Hi. We have kafka setup on server that has two Ethernet cards. Each Ethernet port has its own ip address. We would like to separate traffic and setup producers to communicate through one card and consumers that use high level client through the other card. Would you advise if that configuration pos

Re: Killing broker leader

Hello Hanish, Did you see "failed to send messages after 10 tries" in your producer log? Guozhang On Wed, Jan 15, 2014 at 8:38 AM, Hanish Bansal < hanish.bansal.agar...@gmail.com> wrote: > Hi Francois, > > Probably Kafka-1193 is not due to any misconfiguration, there may be > something else is

Consumer starts to consume after changing group id

Hello, We are using Kafka 0.8 with Storm. To consume from Kafka, we are using https://github.com/wurstmeister/storm-kafka-0.8-plus this spout. We are creating spout with this configuration : https://gist.github.com/gurkanoluc/0cab49b8ddcfe15df8e9 Our problem is, consumers stops after working some

Re: Killing broker leader

Hi Francois, Probably Kafka-1193 is not due to any misconfiguration, there may be something else is missing. I also tried both (max retries 10 and producer acks -1) together, that was also causing data loss. On Wed, Jan 15, 2014 at 9:40 PM, François Langelier wrote: > Yeah, that's what i found.

Re: Killing broker leader

Yeah, that's what i found... sorry about that... Well, i tested it out! At first, my command was : bin/kafka-console-producer.sh --broker-list localhost:9092 --topic my-replicated-topic --message-send-max-retries 10 --retry-backoff-ms 1000 Then i noticed that only the second message i sent got

Re: Killing broker leader

Those are actually producer side configs. Thanks, Jun On Wed, Jan 15, 2014 at 6:51 AM, François Langelier wrote: > Nope, it's the "3 tries" messages... maybe I did something wrong... > > i put > > message.send.max.retries=10 > retry.backoff.ms=1000 > > in my server.properties of each broker >

Re: kafka performance test

The results there are obtained using our ProducerPerformance tool. Thanks, Jun On Wed, Jan 15, 2014 at 1:14 AM, 陈小军 wrote: > I was just reading this: > https://cwiki.apache.org/confluence/display/KAFKA/Performance+testingHasthis > been implemented yet? I can't seem tofind it in the source.

Re: Killing broker leader

Nope, it's the "3 tries" messages... maybe I did something wrong... i put message.send.max.retries=10 retry.backoff.ms=1000 in my server.properties of each broker I'm checking it right now! François Langelier Étudiant en génie Logiciel - École de Technologie Supérieure

RE: purging a topic

Thanks, Jun. I checked ImportZkOffsets. It is not very handy and cannot automate all steps for purging purpose. Regards, Libo -Original Message- From: Jun Rao [mailto:jun...@gmail.com] Sent: Tuesday, January 14, 2014 11:43 PM To: users@kafka.apache.org Subject: Re: purging a topic You

kafka performance test

I was just reading this:https://cwiki.apache.org/confluence/display/KAFKA/Performance+testingHas this been implemented yet? I can't seem tofind it in the source. Best Regards Jerry