Re: Kafka elastic no downtime scalability

2015-03-13 Thread Chi Hoang
Hi Stevo, I won't speak for Joe, but what we do is documented in the link that Joe provided: "Adding servers to a Kafka cluster is easy, just assign them a unique broker id and start up Kafka on your new servers. However these new servers will not automatically be assigned any data partitions, so u

Re: Kafka elastic no downtime scalability

2015-03-13 Thread Stevo Slavić
These features are all nice, if one adds new brokers to support additional topics, or to move existing partitions or whole topics to new brokers. Referenced sentence is in paragraph named scalability. When I read "expanded" I was thinking of scaling out, extending parallelization capabilities, and

Re: Kafka elastic no downtime scalability

2015-03-13 Thread Joe Stein
https://kafka.apache.org/documentation.html#basic_ops_cluster_expansion ~ Joe Stein - - - - - - - - - - - - - - - - - http://www.stealth.ly - - - - - - - - - - - - - - - - - On Fri, Mar 13, 2015 at 3:05 PM, sunil kalva wrote: > Joe > > "Well, I know it is semantic but right now it "can" be e

Re: Kafka elastic no downtime scalability

2015-03-13 Thread sunil kalva
Joe "Well, I know it is semantic but right now it "can" be elastically scaled without down time but you have to integrate into your environment for what that means it has been that way since 0.8.0 imho" here what do you mean "you have to integrate into your environment", how do i achieve elas

Re: Kafka elastic no downtime scalability

2015-03-13 Thread Joe Stein
Well, I know it is semantic but right now it "can" be elastically scaled without down time but you have to integrate into your environment for what that means it has been that way since 0.8.0 imho. My point was just another way to-do that out of the box... folks do this elastic scailing today

Re: Kafka elastic no downtime scalability

2015-03-13 Thread Stevo Slavić
OK, thanks for heads up. When reading Apache Kafka docs, and reading what Apache Kafka "can" I expect it to already be available in latest general availability release, not what's planned as part of some other project. Kind regards, Stevo Slavic. On Fri, Mar 13, 2015 at 2:32 PM, Joe Stein wrote

Re: Kafka elastic no downtime scalability

2015-03-13 Thread Joe Stein
Hey Stevo, "can be elastically and transparently expanded without downtime." is the goal of Kafka on Mesos https://github.com/mesos/kafka as Kafka as the ability (knobs/levers) to-do this but has to be made to-do this out of the box. e.g. in Kafka on Mesos when a broker fails, after the configurab

Kafka elastic no downtime scalability

2015-03-13 Thread Stevo Slavić
Hello Apache Kafka community, On Apache Kafka website home page http://kafka.apache.org/ it is stated that Kafka "can be elastically and transparently expanded without downtime." Is that really true? More specifically, can one just add one more broker, have another partition added for the topic, h