Re: STOMP binding for Kafka

yes; unless something is available already. Thanks Mridul On Mon, Jan 14, 2013 at 11:09 AM, Jun Rao wrote: > Are you thinking of writing a JMS wrapper over Kafka client API? > > Thanks, > > Jun > > On Fri, Jan 11, 2013 at 9:43 PM, Mridul Jain > wrote: > > > I have to replace some existing mess

Re: STOMP binding for Kafka

I am not aware of anything like that available right now. Let us know if you hit any issue. Thanks, Jun On Mon, Jan 14, 2013 at 12:47 AM, Mridul Jain wrote: > yes; unless something is available already. > > Thanks > Mridul > > On Mon, Jan 14, 2013 at 11:09 AM, Jun Rao wrote: > > > Are you thi

Re: partitioning

On Fri, Jan 11, 2013 at 12:37 AM, Jun Rao wrote: > Our current partitioning strategy is to mod key by # of partitions, not # > brokers. For better balancing partitions over brokers, one simple strategy > is to over partition, i.e., you have a few times of more partitions than > brokers. That way,

Re: partitioning

I'm not sure what design doc you are looking at (v1 probably?, v3 is here: https://cwiki.apache.org/KAFKA/kafka-detailed-replication-design-v3.html ) but If I understand correctly, consistent hashing for partitioning is more about remapping as few keys as possible when adding/deleting partitions, w

kafka-0.7.2 sbt and remote installs help

Trying to install kafka on remote machines with no internet access. Not sure how to approach the sbt update && sbt package pieces. I tried using sbt (lib/sbt-launch.jar) on a local machine with internet access as follows: $ sudo su - kafka $ cd /usr/local/kafka-0.7.2-incubating-src $ cat ./sbt jav

Question about kafka consumer stream/partition

Hi I know if the number of kafka consumers is greater than the number of partitions in the kafka broker cluster, several kafka consumers will be idle. My question is, does the number of kafka consumers mean the number of kafka streams? For example, I have one broker with one partition. What if I

Re: hadoop-consumer code in contrib package

I think you may be misunderstanding the way Kafka works. A kafka broker is never supposed to clear messages just because a consumer read them. The kafka broker will instead clear messages after their retention period ends, though it will not delete the messages at the exact time when they expire.

Re: kafka-0.7.2 sbt and remote installs help

You need to do "./sbt update " and "./sbt package" on your local machine with internet access. Then, you can copy the whole dir to your remote machine. Thanks, Jun On Mon, Jan 14, 2013 at 10:23 AM, Joseph Crotty wrote: > Trying to install kafka on remote machines with no internet access. Not >

Re: Is this a good overview of kafka?

Hello, Your (non-question) statements seem mostly right to me. There is a bit of confusion regarding your statement about partitions, however. Partitions are primarily used to represent the smallest unit of parallelism. If you need to split consumption among a pool of processes, you need to have

Re: SyncProducer vs Producer

Producer is the high level api whereas SyncProducer is the lower level api. Producer takes one or more messages and converts them to a request which is sent by SyncProducer. Producer is actually the client api that everyone should be using. Thanks, Jun On Mon, Jan 14, 2013 at 10:26 AM, navneet s

Re: Question about kafka consumer stream/partition

> My question is, does the number of kafka consumers mean the number of > kafka streams? > Yes. To know the total number of consumers/streams in a group, you need to add up the number of streams on every consumer instance > For example, I have one broker with one partition. What if I create > th

Re: Is this a good overview of kafka?

Hi Felix, Would you mind elaborating on what you said regarding the ordering guaranteed; inlined below. Thanks, stan On Mon, Jan 14, 2013 at 6:08 PM, Felix GV wrote: > > For example if you partitioned using a User ID field within the messages, > you would be > guaranteed that all messages per

Re: Is this a good overview of kafka?

Sure, I'll try to give a better explanation :) Little disclaimer though: My knowledge is based on my reading of the Kafka design paper more than a year ago, so right off the bat, it's possible that I may be forgetting or assuming things which I shouldn't... Al

About kafka 0.8 producer auto detect broker

Hi, We know in Kafka 0.7, we can specify zk.connect. And with zookeeper, the producer can dynamically detect broker. But in Kafka 0.8, we can't specify zk.connect for producer. How does the producer in Kafka 0.8 auto detect broker? I have done two experiments. In first one, I configure the broke

Re: SyncProducer vs Producer

If that is the case: "Producer is actually the client api that everyone should be using." Then why contrib/hadoop-consumer is using SynProducer. Can i modify the code to use Producer? Will it have any impact on the system? Thanks, Navneet Sharma On Tue, Jan 15, 2013 at 5:16 AM, Jun Rao wrote:

Number of Partitions Per Broker

All, I was re-reading this: https://cwiki.apache.org/confluence/display/KAFKA/Operations and noticed that the number of partitions is 1. Is this accurate? In our environment we are currently running 20+ partitions per topic - with two brokers, the gut feel was this would speed up our ability to