Re: Hardware planning

Ray, We are, for SPM . On c1.medium instances, I believe, we have: * Jetty receiving tens of thousands of metrics per second (in batches, so the rate of HTTP requests is lower than that number_ * Kafka brokers * ZK instances So far we have not had issues with this. Knock

Re: Migration Tool consumer rebalance error when attempting to migrate from Kafka 0.7 to Kafka 0.8

The following is the issue. It seems that the 0.7 jar that you used in MigrationTool is not the same as the one used in 0.7 broker. So, the consumer can't parse the format of the broker registration in ZK. [mfirouz-vagrant-kafkamigration_precise64-1395250529941-562df2ce], exception during rebalanc

Re: sources jar for 0.8.1?

Hello Ian, just a workaround I used in the interim: I downloaded the tar’ed sources from here https://www.apache.org/dyn/closer.cgi?path=/kafka/0.8.1/kafka-0.8.1-src.tgz (I use Intellij) When I try to open the definition of a Kafka object that it doesn’t have sources for, there is a button tha

Re: kafka/zk errors

Hello Alex, Error 1,3 and 4 can be transient and expected if on broker startup/bouncing/shutdown, error 2 indicates the request type is incorrect: currently we only have up to type 10, which consumer client do you use to consume data? I am not sure about the ZK error, but it seems indicating the

Re: Can a producer detect when a topic has no consumers?

Currently producers cannot detect if a topic is consumed by someone or not themselves. If you want to know globally if a specific topic is consumed by anyone or not, you can query the ZK for all the consumer groups, and search if they contain the topic name. Guozhang On Wed, Mar 19, 2014 at 1:01

Re: How produce to remote Kafka Cluster over ssh tunneling

Thanks for quick reply. advertised.host.name/advertised.port setting is this. I confirmed only producer setting, so I couldn't find these properties. I will try it. Thank you. 2014-03-20 7:50 GMT+09:00 Joe Stein : > With 0.8.1 update these two properties in the server.properties config of > t

Re: How produce to remote Kafka Cluster over ssh tunneling

With 0.8.1 update these two properties in the server.properties config of the broker to be 172.16.0.100 and 19092 respectively # Hostname the broker will advertise to producers and consumers. If not set, it uses the # value for "host.name" if configured. Otherwise, it will use the value returned

How produce to remote Kafka Cluster over ssh tunneling

Hi, all. I'm trying Log produce to Remote Kafka Cluster over SSH Tunneling. Configuration is below. - Kafka cluster in DataCenter(each machine has Grobal IP Address). Firewall allows only ssh port(22). ex) 192.168.100.100 - Kafka producer exists out of DataCenter. ex) 172.16.0.100 I tried produc

.8 .net/csharp API?

Hi- wondering if anyone knows of a working .net API for kafka. Last time I checked only .7 was available. At this point only publishing/no consumption required.

Apache Kafka

Hi, I was trying to understand why "Kafka" is called "Kafka". Any help would be high appreciated. Best Regards, Asif Abbasi

Re: Can a producer detect when a topic has no consumers?

Hey Todd, For my purposes, I wrote a Partitioner that reads all of the ConsumerOffsets for a particular group from Zookeeper, and then sends its message to the one with least lag (with some probabilistic weighting thrown in to prevent it from getting overwhelmed). I basically read and reimpleme

sources jar for 0.8.1?

Hey guys, I am wondering why there is no sources jar in maven for kafka 0.8.1? I am trying to migrate our project now and not having the sources easily available in my IDE is a real hindrance. I did some googling and found this: https://issues.apache.org/jira/browse/KAFKA-1248 Not sure why it wa

kafka/zk errors

we're getting hundreds of these errs with kafka 0.8 and topics become unavailable after running for a few days, any ideas how to proceed? kafka error 1 (had to recreate the topic as it became unavailable after a few days) [2014-03-18 16:34:56,403] ERROR [KafkaApi-3] Error while fetchingmetadata f

Re: 0.8.1 stability

-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Thanks for this follow-up, and +1 here. I was aware, for instance, that the newly introduced delete topic feature in 0.8.1 is not fully ready for prime time. But IIRC I learned about this by following the mailing list (It think it was actually a repl

Re: Adding replicas to existing topic

Ya, this is one tool that has been dragging for a bit too long and is not available in 0.8.1. There is a way to use the partition reassignment tool to achieve the same objective (by reassigning a partition from replicas r1

Re: Adding replicas to existing topic

done, https://issues.apache.org/jira/browse/KAFKA-1313 Can you expand on the hacky method? If could fix this before 0.8.2 is released, it'd be great. thanks marc On Wed, Mar 19, 2014 at 12:27 PM, Neha Narkhede wrote: > Ya, this is one tool that has been dragging for a bit too long and is not >

Re: Requesting advice about Producer & Consumer Design

Hi Jay, Thanks for taking time for the details ! Appreciate that. Just clarifying myself couple of things in the same line. 1) The new producer and consumer is being designed to take care of auto balancing between partitions. Right ? 2) with the current available producer and consumer, is my cu

Re: Migration Tool consumer rebalance error when attempting to migrate from Kafka 0.7 to Kafka 0.8

Hi Jun, Thanks for the reply. Please find below the DEBUG output of the log. *vagrant@precise64:~/Desktop$ ~/Desktop/kafka_2.9.2-0.8.1/bin/kafka-run-class.sh kafka.tools.KafkaMigrationTool --kafka.07.jar kafka-0.7.19.jar --zkclient.01.jar zkclient-0.2.0.jar --num.producers 16 --producer.config=/

Re: Apache Kafka

Many thanks Clark. That makes sense :) On Wed, Mar 19, 2014 at 5:10 PM, Clark Breyman wrote: > Asif - Kafka was a writer. > https://twitter.com/jaykreps/status/421065665160548352 > > > On Wed, Mar 19, 2014 at 10:05 AM, Muhammad Asif Abbasi < > asif.abb...@gmail.com> wrote: > > > Hi, > > > > I w

Re: Apache Kafka

Asif - Kafka was a writer. https://twitter.com/jaykreps/status/421065665160548352 On Wed, Mar 19, 2014 at 10:05 AM, Muhammad Asif Abbasi < asif.abb...@gmail.com> wrote: > Hi, > > I was trying to understand why "Kafka" is called "Kafka". > > Any help would be high appreciated. > > Best Regards, >

Re: Migration Tool consumer rebalance error when attempting to migrate from Kafka 0.7 to Kafka 0.8

Hi Mo, Is the broker id in your server properties really the same for all 3, or is that a typo? Steve On Wed, Mar 19, 2014 at 8:10 AM, Jun Rao wrote: > Were there any logging before that? There should be some logging that tells > you why the rebalance failed. > > Thanks, > > Jun > > > On Wed,

Can a producer detect when a topic has no consumers?

I'd like to avoid creating messages for a topic is the topic currently has no consumers. Is there a way a producer can subscribe to the number of consumers of a topic? Or if that's not possible, can a producer request the current number of consumers of a topic? The searchable hadoop kafka topi

Re: Kafka C++ Windows Client for 0.8 Releases

Hi Magnus, our applications are running under Windows and use many Windows features. Therefore Cygwin can not be used for runtime. However it would be interesting if we could generate the librdkafka libs for the client for the Windows environment. I am wondering if you have checked building t

Adding replicas to existing topic

Hi, I have a few topics that were created a while ago with default parameters with only 1 replica per partition. I am wondering if there is a way to add replicas to existing partitions of an existing topic. I didn't see anything obvious from the tools. marc

Re: Compute offset for messages compressed with Snappy inKakfa 0.8.0

Do you mean Camus didn't write any data to HDFS or you can't read the data on HDFS? SimpleConsumer works with compressed data. You just need to make sure that the fetch size is larger than the compressed set of messages. Also, have you asked the Camus mailing list? Thanks, Jun On Wed, Mar 19, 2

Re: Migration Tool consumer rebalance error when attempting to migrate from Kafka 0.7 to Kafka 0.8

Were there any logging before that? There should be some logging that tells you why the rebalance failed. Thanks, Jun On Wed, Mar 19, 2014 at 7:07 AM, Mo Firouz wrote: > Hello. > > I am trying to migrate from Kafka 0.7 to 0.8, and the first stage of this > is to use the Migration Tool. > > I

Re: Requesting advice about Producer & Consumer Design

Hey Krishna, Let me clarify the current state of things a little: 1. Kafka offers a single producer interface as well as two consumer interfaces: the low-level "simple consumer" which just directly makes network requests, and the higher level interface which handles fault-tolerance, partition assi

Re: Kafka C++ Windows Client for 0.8 Releases

That would be fantastic. How long will that take you? Would you put it up on github? On Wednesday, March 19, 2014, Tianning Zhang wrote: > Hi, > > we build our impl. using microsoft visual studio 2010 (migration to 2012 > is under way). Cygwin not used. > If useful I will separate generic part

Re: Kafka C++ Windows Client for 0.8 Releases

Hi, we build our impl. using microsoft visual studio 2010 (migration to 2012 is under way). Cygwin not used. If useful I will separate generic part from our proprietary resources/Apis and make it public, KR Tianning Dan Hoffman schrieb am Mi,

Migration Tool consumer rebalance error when attempting to migrate from Kafka 0.7 to Kafka 0.8

Hello. I am trying to migrate from Kafka 0.7 to 0.8, and the first stage of this is to use the Migration Tool. I have setup a local Kafka 0.8 cluster consisting of 3 nodes on my local Vagrant (along with a zookeeper running locally). The Migration Tool Producer will connect to these three nodes i

Compute offset for messages compressed with Snappy inKakfa 0.8.0

I am using Camus to get compressed Snappy messages into HDFS. However, I can't get records in HDFS even though CamusJob was completed successfully. Getting uncompressed AVRO messages into HDFS is fine. According to Neha's post, the consumer should be * agnostic to compression.* http://geekmantra.w

Re: Kafka C++ Windows Client for 0.8 Releases

If you are okay to use Cygwin then librdkafka might be of interest: https://github.com/edenhill/librdkafka/tree/c++_configure 2014-03-19 19:10 GMT+07:00 Tianning Zhang : > Dear all, > > I am writing to this list to check if there is any new progress concerning > 0.8 Windows C++ clients and if t

Re: Kafka C++ Windows Client for 0.8 Releases

Curious- does your impl require Cygwin? If not, I'd love to use it On Wednesday, March 19, 2014, Tianning Zhang wrote: > Dear all, > > I am writing to this list to check if there is any new progress concerning > 0.8 Windows C++ clients and if there are experiences we can share. > > My company (

Kafka C++ Windows Client for 0.8 Releases

Dear all, I am writing to this list to check if there is any new progress concerning 0.8 Windows C++ clients and if there are experiences we can share. My company (we are in the context of performance advertising networks) is using Kafka 0.8 as the backbone for the event infrastructure. As the

Requesting advice about Producer & Consumer Design

Hello Experts & Kafka Team, Its existing to learn and work on using Kafka. I have been going through lot of pages and Q&A. We are building an infra & topology using Kafka for events processing in our application. We need some advice about designing the Producer and Consumer. *Please find attach