Re: abstracting ZooKeeper

2015-08-09 Thread Joe Stein
I have started writing a KIP about this topic https://cwiki.apache.org/confluence/display/KAFKA/KIP-30+-+Allow+for+brokers+to+have+plug-able+consensus+and+meta+data+storage+sub+systems with hopes to get more of it typed out and then circulate for discussion on the dev list over the next ~ week. I

Is decommissioning brokers supported in 0.8.2?

2015-08-09 Thread yuto sasaki
Hi, I found the document says that "we plan to add tooling support for decommissioning brokers in 0.8.2." in http://kafka.apache.org/082/documentation.html#basic_ops_decommissioning_brokers But in 0.8.2.1, I could not find the option. How can we decommission brokers? Thanks.

Re: abstracting ZooKeeper

2015-08-09 Thread Julio Castillo
The only reason for this request is because I may want to use alternatives like Consul. ** julio On 8/9/15, 3:40 PM, "Joe Lawson" wrote: >Inline responses below. > >Sincerely, > >Joe Lawson > >On Aug 9, 2015 1:52 PM, "Julio Castillo" >wrote: >> >> Thank for the lead. >> Does that mean that Kaf

Re: New producer in production

2015-08-09 Thread yuto sasaki
Hi, Is there any progress on this doc bug? This discourages us from using the new producer. Thanks On 2015/07/18 5:23, Sivananda Reddy wrote: Hi Jay, I found it here: http://kafka.apache.org/documentation.html#newproducerconfigs, the same link is reported by James. @Joel: Thanks a lot for t

Re: Why Apache is scalable?

2015-08-09 Thread Daniel Compton
Hi Lucky Your question at the moment is a bit vague, and not particularly well posed. These kinds of questions can normally be answered best by a quick google search and reading the docs, rather than asking the list. Kafka provides excellent docs at http://kafka.apache.org/documentation.html#intr

Re: configuring log compaction

2015-08-09 Thread David Judd
Ok. I think that actually works fine for my use case. Thanks! On Sun, Aug 9, 2015 at 4:26 PM, Jiangjie Qin wrote: > Actually Kafka only support two mutually exclusive log cleanup policy: 1) > delete logs after retention period passed. 2) compact the log to only keep > the last updated value of a

Re: configuring log compaction

2015-08-09 Thread Jiangjie Qin
Actually Kafka only support two mutually exclusive log cleanup policy: 1) delete logs after retention period passed. 2) compact the log to only keep the last updated value of a key. log.retention.hours is only used by (1). For log compaction, currently it is not compacting the logs by its age, but

Re: abstracting ZooKeeper

2015-08-09 Thread Joe Lawson
Inline responses below. Sincerely, Joe Lawson On Aug 9, 2015 1:52 PM, "Julio Castillo" wrote: > > Thank for the lead. > Does that mean that Kafka is/will be using Curator? I don't think so. > > Also, this appears to simplify the interaction with ZooKeeper, but if I > understand it correctly,

configuring log compaction

2015-08-09 Thread David Judd
Hi, I have a beginner's question. I'd like to confirm that I'm interpreting the docs correctly. My goal is a setup where, by default, logs are compacted after 7 days and the compacted records are retained indefinitely. Does the following combination of settings achieve this? log.retention.hours=

Re: abstracting ZooKeeper

2015-08-09 Thread Julio Castillo
Thank for the lead. Does that mean that Kafka is/will be using Curator? Also, this appears to simplify the interaction with ZooKeeper, but if I understand it correctly, it doesn¹t abstract the interface where could plug-in a different service. Thanks ** julio On 8/9/15, 10:21 AM, "Joe Lawson"

Re: abstracting ZooKeeper

2015-08-09 Thread Joe Lawson
Netflix contributed Curator (http://curator.apache.org/) to Apache which implements some generic zk recipes. On Aug 9, 2015 11:39 AM, "Julio Castillo" wrote: > Had there been any thought at abstracting the interface to ZooKeeper? > > The reason I'm asking is because I'm looking at Consul for serv

abstracting ZooKeeper

2015-08-09 Thread Julio Castillo
Had there been any thought at abstracting the interface to ZooKeeper? The reason I'm asking is because I'm looking at Consul for service discovery today, perhaps a different one tomorrow, but the point here is the ability to plug in any type of service discovery, K/V store service. Any thoughts

Why Apache is scalable?

2015-08-09 Thread lucky Johnson
Hi, I am new to Apache Kafka and i have two questions: Why do we say Apache Kafka is Scalable? How do I test its Scalability? need someone help please! Best Lucky