Wouldn't it make sense to move away from these rich binary broker
descriptors ({ host, port, proto })
(which require protocol churning on change), and simply use URIs instead?E.g.: kafka://<host>[:port]/ <-- cleantext proto on standard port 9092 kafkas://<host>[:port] <-- SSL enveloped proto on standard port 9093 kafkas://<user:pass>@<host>[:port]/ <-- SSL enveloped, with user authentication .. kafkafuturetech://.../#opts <-- six months from now. Trailing #fragment_ids could be used to hint the client on protocol versions, supported authentications, etc. This also makes error reporting more meaningful on the client, e.g compare: "Unsupported protocol 19 on broker foo:1234" to "Unsupported protocol kafkafturetech on broker foo:1234" A positive side effect would be a more generalized topic addressing in clients: kafkacat kafka://<bootstrap>/mytopic/3?offset=end <-- tail partition 3 of mytopic Just an idea, Magnus 2015-01-23 5:43 GMT+01:00 Jun Rao <j...@confluent.io>: > Reviewed the latest patch in KAFKA-1809 :). > > Thanks, > > Jun > > On Thu, Jan 22, 2015 at 12:38 PM, Gwen Shapira <gshap...@cloudera.com> > wrote: > > > Thanks for validating our ideas. Updated the KIP with the workflow. > > > > Now if you can nudge Jun to review the latest patch... ;) > > > > > > On Thu, Jan 22, 2015 at 11:44 AM, Jay Kreps <j...@confluent.io> wrote: > > > Oh yeah I think that is better, I hadn't thought of that approach! Any > > way > > > you could describe the usage in the KIP, just for completeness? > > > > > > -Jay > > > > > > On Thu, Jan 22, 2015 at 10:23 AM, Gwen Shapira <gshap...@cloudera.com> > > > wrote: > > > > > >> I think what you described was the original design, so no wonder you > > >> are confused :) > > >> > > >> Following suggestions from Jun, I changed it a bit. The current model > > is: > > >> > > >> - Clients (producers and consumers) need to know about the broker > > >> ports in advance. They don't need to know about all brokers, but they > > >> need to know at least one host:port pair that speaks the protocol they > > >> want to use. The change is that all host:port pairs in broker.list > > >> must be of the same protocol and match the security.protocol > > >> configuration parameter. > > >> > > >> - Client uses security.protocol configuration parameter to open a > > >> connection to one of the brokers and sends the good old > > >> MetadataRequest. The broker knows which port it got the connection on, > > >> therefore it knows which security protocol is expected (it needs to > > >> use the same protocol to accept the connection and respond), and > > >> therefore it can send a response that contains only the host:port > > >> pairs that are relevant to that protocol. > > >> > > >> - From the client side the MetadataResponse did not change - it > > >> contains a list of brokerId,host,port that the client can connect to. > > >> The fact that all those broker endpoints were chosen out of a larger > > >> collection to match the right protocol is irrelevant for the client. > > >> > > >> I really like the new design since it preserves a lot of the same > > >> configurations and APIs. > > >> > > >> Thoughts? > > >> > > >> Gwen > > >> > > >> On Thu, Jan 22, 2015 at 9:19 AM, Jay Kreps <jay.kr...@gmail.com> > wrote: > > >> > I think I am still confused. In addition to the > UpdateMetadataRequest > > >> don't > > >> > we have to change the MetadataResponse so that it's possible for > > clients > > >> to > > >> > discover the new ports? Or is that a second phase? I was imagining > it > > >> > worked by basically allowing the brokers to advertise multiple > ports, > > one > > >> > per security type, and then in the client you configure a protocol > > which > > >> > will implicitly choose the port from the options returned in > metadata > > to > > >> > you... > > >> > > > >> > Likewise in the ConsumerMetadataResponse we are currently giving > back > > >> full > > >> > broker information. I think we would have two options here: either > > change > > >> > the broker information included in that response to match the > > >> > metadataresponse or else remove the broker information entirely and > > just > > >> > return the node id (since in order to use that request you would > > already > > >> > have to have the cluster metadata). The second option may be cleaner > > >> since > > >> > it means we won't have to continue evolving those two in lockstep... > > >> > > > >> > -Jay > > >> > > > >> > On Wed, Jan 21, 2015 at 6:19 PM, Gwen Shapira < > gshap...@cloudera.com> > > >> wrote: > > >> > > > >> >> Good point :) > > >> >> > > >> >> I added the specifics of the new UpdateMetadataRequest, which is > the > > >> >> only protocol bump in this change. > > >> >> > > >> >> Highlighted the broker and producer/consumer configuration changes, > > >> >> added some example values and added the new zookeeper json. > > >> >> > > >> >> Hope this makes things clearer. > > >> >> > > >> >> On Wed, Jan 21, 2015 at 2:19 PM, Jay Kreps <jay.kr...@gmail.com> > > wrote: > > >> >> > Hey Gwen, > > >> >> > > > >> >> > Could we get the actual changes in that KIP? I.e. changes to > > metadata > > >> >> > request, changes to UpdateMetadataRequest, new configs and what > > will > > >> >> their > > >> >> > valid values be, etc. This kind of says that those things will > > change > > >> but > > >> >> > doesn't say what they will change to... > > >> >> > > > >> >> > -Jay > > >> >> > > > >> >> > On Mon, Jan 19, 2015 at 9:45 PM, Gwen Shapira < > > gshap...@cloudera.com> > > >> >> wrote: > > >> >> > > > >> >> >> I created a KIP for the multi-port broker change. > > >> >> >> > > >> >> >> > > >> >> > > >> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-2+-+Refactor+brokers+to+allow+listening+on+multiple+ports+and+IPs > > >> >> >> > > >> >> >> I'm not re-opening the discussion, since it was agreed on over a > > >> month > > >> >> >> ago and implementation is close to complete (I hope!). Lets > > consider > > >> >> >> this voted and accepted? > > >> >> >> > > >> >> >> Gwen > > >> >> >> > > >> >> >> On Sun, Jan 18, 2015 at 10:31 AM, Jay Kreps < > jay.kr...@gmail.com> > > >> >> wrote: > > >> >> >> > Great! Sounds like everyone is on the same page > > >> >> >> > > > >> >> >> > - I created a template page to make things easier. If you > do > > >> >> >> Tools->Copy > > >> >> >> > on this page you can just fill in the italic portions with > > your > > >> >> >> details. > > >> >> >> > - I retrofitted KIP-1 to match this formatting > > >> >> >> > - I added the metadata section people asked for (a link to > > the > > >> >> >> > discussion, the JIRA, and the current status). Let's make > > sure > > >> we > > >> >> >> remember > > >> >> >> > to update the current status as things are figured out. > > >> >> >> > - Let's keep the discussion on the mailing list rather than > > on > > >> the > > >> >> >> wiki > > >> >> >> > pages. It makes sense to do one or the other so all the > > comments > > >> >> are > > >> >> >> in one > > >> >> >> > place and I think prior experience is that the wiki > comments > > are > > >> >> the > > >> >> >> worse > > >> >> >> > way. > > >> >> >> > > > >> >> >> > I think it would be great do KIPs for some of the in-flight > > items > > >> >> folks > > >> >> >> > mentioned. > > >> >> >> > > > >> >> >> > -Jay > > >> >> >> > > > >> >> >> > On Sat, Jan 17, 2015 at 8:23 AM, Gwen Shapira < > > >> gshap...@cloudera.com> > > >> >> >> wrote: > > >> >> >> > > > >> >> >> >> +1 > > >> >> >> >> > > >> >> >> >> Will be happy to provide a KIP for the multiple-listeners > > patch. > > >> >> >> >> > > >> >> >> >> Gwen > > >> >> >> >> > > >> >> >> >> On Sat, Jan 17, 2015 at 8:10 AM, Joe Stein < > > joe.st...@stealth.ly> > > >> >> >> wrote: > > >> >> >> >> > +1 to everything we have been saying and where this (has > > settled > > >> >> >> to)/(is > > >> >> >> >> > settling to). > > >> >> >> >> > > > >> >> >> >> > I am sure other folks have some more feedback and think we > > >> should > > >> >> try > > >> >> >> to > > >> >> >> >> > keep this discussion going if need be. I am also a firm > > >> believer of > > >> >> >> form > > >> >> >> >> > following function so kicking the tires some to flesh out > the > > >> >> details > > >> >> >> of > > >> >> >> >> > this and have some organic growth with the process will be > > >> healthy > > >> >> and > > >> >> >> >> > beneficial to the community. > > >> >> >> >> > > > >> >> >> >> > For my part, what I will do is open a few KIP based on some > > of > > >> the > > >> >> >> work I > > >> >> >> >> > have been involved with for 0.8.3. Off the top of my head > > this > > >> >> would > > >> >> >> >> > include 1) changes to re-assignment of partitions 2) kafka > > cli > > >> 3) > > >> >> >> global > > >> >> >> >> > configs 4) security white list black list by ip 5) SSL 6) > We > > >> >> probably > > >> >> >> >> will > > >> >> >> >> > have lots of Security related KIPs and should treat them > all > > >> >> >> individually > > >> >> >> >> > when the time is appropriate 7) Kafka on Mesos. > > >> >> >> >> > > > >> >> >> >> > If someone else wants to jump in to start getting some of > the > > >> >> security > > >> >> >> >> KIP > > >> >> >> >> > that we are going to have in 0.8.3 I think that would be > > great > > >> >> (e.g. > > >> >> >> >> > Multiple Listeners for Kafka Brokers). There are also a few > > >> other > > >> >> >> >> tickets I > > >> >> >> >> > can think of that are important to have in the code in > 0.8.3 > > >> that > > >> >> >> should > > >> >> >> >> > have KIP also that I haven't really been involved in. I > will > > >> take a > > >> >> >> few > > >> >> >> >> > minutes and go through JIRA (one I can think of like auto > > >> assign id > > >> >> >> that > > >> >> >> >> is > > >> >> >> >> > already committed I think) and ask for a KIP if appropriate > > or > > >> if I > > >> >> >> feel > > >> >> >> >> > that I can write it up (both from a time and understanding > > >> >> >> perspective) > > >> >> >> >> do > > >> >> >> >> > so. > > >> >> >> >> > > > >> >> >> >> > long story short, I encourage folks to start moving ahead > > with > > >> the > > >> >> KIP > > >> >> >> >> for > > >> >> >> >> > 0.8.3 as how we operate. any objections? > > >> >> >> >> > > > >> >> >> >> > On Fri, Jan 16, 2015 at 2:40 PM, Guozhang Wang < > > >> wangg...@gmail.com > > >> >> > > > >> >> >> >> wrote: > > >> >> >> >> > > > >> >> >> >> >> +1 on the idea, and we could mutually link the KIP wiki > page > > >> with > > >> >> the > > >> >> >> >> the > > >> >> >> >> >> created JIRA ticket (i.e. include the JIRA number on the > > page > > >> and > > >> >> the > > >> >> >> >> KIP > > >> >> >> >> >> url on the ticket description). > > >> >> >> >> >> > > >> >> >> >> >> Regarding the KIP process, probably we do not need two > phase > > >> >> >> >> communication > > >> >> >> >> >> of a [DISCUSS] followed by [VOTE], as Jay said the voting > > >> should > > >> >> be > > >> >> >> >> clear > > >> >> >> >> >> while people discuss about that. > > >> >> >> >> >> > > >> >> >> >> >> About who should trigger the process, I think the only > > involved > > >> >> >> people > > >> >> >> >> >> would be 1) when the patch is submitted / or even the > > ticket is > > >> >> >> created, > > >> >> >> >> >> the assignee could choose to start the KIP process if she > > >> thought > > >> >> it > > >> >> >> is > > >> >> >> >> >> necessary; 2) the reviewer of the patch can also suggest > > >> starting > > >> >> KIP > > >> >> >> >> >> discussions. > > >> >> >> >> >> > > >> >> >> >> >> On Fri, Jan 16, 2015 at 10:49 AM, Gwen Shapira < > > >> >> >> gshap...@cloudera.com> > > >> >> >> >> >> wrote: > > >> >> >> >> >> > > >> >> >> >> >> > +1 to Ewen's suggestions: Deprecation, status and > version. > > >> >> >> >> >> > > > >> >> >> >> >> > Perhaps add the JIRA where the KIP was implemented to > the > > >> >> metadata. > > >> >> >> >> >> > This will help tie things together. > > >> >> >> >> >> > > > >> >> >> >> >> > On Fri, Jan 16, 2015 at 9:35 AM, Ewen Cheslack-Postava > > >> >> >> >> >> > <e...@confluent.io> wrote: > > >> >> >> >> >> > > I think adding a section about deprecation would be > > >> helpful. A > > >> >> >> good > > >> >> >> >> >> > > fraction of the time I would expect the goal of a KIP > > is to > > >> >> fix > > >> >> >> or > > >> >> >> >> >> > replace > > >> >> >> >> >> > > older functionality that needs continued support for > > >> >> >> compatibility, > > >> >> >> >> but > > >> >> >> >> >> > > should eventually be phased out. This helps Kafka devs > > >> >> understand > > >> >> >> >> how > > >> >> >> >> >> > long > > >> >> >> >> >> > > they'll end up supporting multiple versions of > features > > and > > >> >> helps > > >> >> >> >> users > > >> >> >> >> >> > > understand when they're going to have to make updates > to > > >> their > > >> >> >> >> >> > applications. > > >> >> >> >> >> > > > > >> >> >> >> >> > > Less important but useful -- having a bit of standard > > >> metadata > > >> >> >> like > > >> >> >> >> >> PEPs > > >> >> >> >> >> > > do. Two I think are important are status (if someone > > lands > > >> on > > >> >> the > > >> >> >> >> KIP > > >> >> >> >> >> > page, > > >> >> >> >> >> > > can they tell whether this KIP was ever completed?) > > and/or > > >> the > > >> >> >> >> version > > >> >> >> >> >> > the > > >> >> >> >> >> > > KIP was first released in. > > >> >> >> >> >> > > > > >> >> >> >> >> > > > > >> >> >> >> >> > > > > >> >> >> >> >> > > On Fri, Jan 16, 2015 at 9:20 AM, Joel Koshy < > > >> >> jjkosh...@gmail.com > > >> >> >> > > > >> >> >> >> >> wrote: > > >> >> >> >> >> > > > > >> >> >> >> >> > >> I'm definitely +1 on the KIP concept. As Joe > > mentioned, we > > >> >> are > > >> >> >> >> already > > >> >> >> >> >> > >> doing this in one form or the other. However, IMO it > is > > >> >> fairly > > >> >> >> ad > > >> >> >> >> hoc > > >> >> >> >> >> > >> - i.e., a combination of DISCUSS threads, jira > > comments, > > >> RB > > >> >> and > > >> >> >> >> code > > >> >> >> >> >> > >> comments, wikis and html documentation. In the past I > > have > > >> >> had > > >> >> >> to > > >> >> >> >> dig > > >> >> >> >> >> > >> into a bunch of these to try and figure out why > > something > > >> was > > >> >> >> >> >> > >> implemented a certain way. I think KIPs can help a > lot > > >> here > > >> >> >> first > > >> >> >> >> by > > >> >> >> >> >> > >> providing guidelines on what to think about > > >> (compatibility, > > >> >> new > > >> >> >> >> APIs, > > >> >> >> >> >> > >> etc.) when working through a major feature; and > second > > by > > >> >> >> becoming > > >> >> >> >> a > > >> >> >> >> >> > >> crisp source of truth documentation for new releases. > > >> E.g., > > >> >> for > > >> >> >> >> >> > >> feature X: see relevant KIPs: a, b, c, etc. > > >> >> >> >> >> > >> > > >> >> >> >> >> > >> On Thu, Jan 15, 2015 at 08:11:20PM -0800, Jay Kreps > > wrote: > > >> >> >> >> >> > >> > Hey Joe, > > >> >> >> >> >> > >> > > > >> >> >> >> >> > >> > Yeah I guess the question is what is the definition > > of > > >> >> major? > > >> >> >> I > > >> >> >> >> >> agree > > >> >> >> >> >> > we > > >> >> >> >> >> > >> > definitely don't want to generate a bunch of > > paperwork. > > >> We > > >> >> >> have > > >> >> >> >> >> enough > > >> >> >> >> >> > >> > problems just getting all the contributions > reviewed > > and > > >> >> >> checked > > >> >> >> >> in > > >> >> >> >> >> > in a > > >> >> >> >> >> > >> > timely fashion... > > >> >> >> >> >> > >> > > > >> >> >> >> >> > >> > So obviously bug fixes would not apply here. > > >> >> >> >> >> > >> > > > >> >> >> >> >> > >> > I think it is also pretty clear that big features > > should > > >> >> get > > >> >> >> >> >> reviewed > > >> >> >> >> >> > and > > >> >> >> >> >> > >> > discussed. To pick on myself, for example, the log > > >> >> compaction > > >> >> >> >> work > > >> >> >> >> >> was > > >> >> >> >> >> > >> done > > >> >> >> >> >> > >> > without enough public discussion about how it > worked > > and > > >> >> why > > >> >> >> >> >> (imho). I > > >> >> >> >> >> > >> > hope/claim that enough rigour in thinking about > > >> use-cases > > >> >> and > > >> >> >> >> >> > operations > > >> >> >> >> >> > >> > and so on was done that it turned out well, but the > > >> >> discussion > > >> >> >> >> was > > >> >> >> >> >> > just > > >> >> >> >> >> > >> > between a few people with no real public output. > This > > >> kind > > >> >> of > > >> >> >> >> >> feature > > >> >> >> >> >> > is > > >> >> >> >> >> > >> > clearly a big change and something we should > discuss. > > >> >> >> >> >> > >> > > > >> >> >> >> >> > >> > If we limit ourselves to just the public contracts > > the > > >> KIP > > >> >> >> >> >> introduces > > >> >> >> >> >> > the > > >> >> >> >> >> > >> > discussion would just be on the new configs and > > >> monitoring > > >> >> >> >> without > > >> >> >> >> >> > >> really a > > >> >> >> >> >> > >> > discussion of the design and how it works which is > > >> >> obviously > > >> >> >> >> closely > > >> >> >> >> >> > >> > related. > > >> >> >> >> >> > >> > > > >> >> >> >> >> > >> > I don't think this should be more work because in > > >> practice > > >> >> we > > >> >> >> are > > >> >> >> >> >> > making > > >> >> >> >> >> > >> > wiki pages for any big thing anyway. So this would > > just > > >> be > > >> >> a > > >> >> >> >> >> > consistent > > >> >> >> >> >> > >> way > > >> >> >> >> >> > >> > of numbering and structuring these pages. This > would > > >> also > > >> >> >> give a > > >> >> >> >> >> clear > > >> >> >> >> >> > >> call > > >> >> >> >> >> > >> > to action: "hey kafka people, come read my wiki and > > >> think > > >> >> this > > >> >> >> >> >> > through". > > >> >> >> >> >> > >> > > > >> >> >> >> >> > >> > I actually thinking the voting aspect is less > > >> important. I > > >> >> >> think > > >> >> >> >> it > > >> >> >> >> >> is > > >> >> >> >> >> > >> > generally clear when there is agreement on > something > > and > > >> >> not. > > >> >> >> So > > >> >> >> >> >> from > > >> >> >> >> >> > my > > >> >> >> >> >> > >> > point of view we could actually just eliminate that > > >> part if > > >> >> >> that > > >> >> >> >> is > > >> >> >> >> >> > too > > >> >> >> >> >> > >> > formal, it just seemed like a good way to formally > > adopt > > >> >> >> >> something. > > >> >> >> >> >> > >> > > > >> >> >> >> >> > >> > To address some of your comments from the wiki: > > >> >> >> >> >> > >> > > > >> >> >> >> >> > >> > 1. This doesn't inhibit someone coming along and > > putting > > >> >> up a > > >> >> >> >> patch. > > >> >> >> >> >> > It > > >> >> >> >> >> > >> is > > >> >> >> >> >> > >> > just that when they do if it is a big thing > > introducing > > >> new > > >> >> >> >> >> > functionality > > >> >> >> >> >> > >> > we would ask for a little discussion on the basic > > >> >> >> >> feature/contracts > > >> >> >> >> >> > prior > > >> >> >> >> >> > >> > to code review. > > >> >> >> >> >> > >> > > > >> >> >> >> >> > >> > 2. We definitely definitely don't want people > > >> generating a > > >> >> >> lot of > > >> >> >> >> >> > these > > >> >> >> >> >> > >> > things every time they have an idea that they > aren't > > >> going > > >> >> to > > >> >> >> >> >> > implement. > > >> >> >> >> >> > >> So > > >> >> >> >> >> > >> > this is only applicable to things you absolutely > will > > >> >> check in > > >> >> >> >> code > > >> >> >> >> >> > for. > > >> >> >> >> >> > >> We > > >> >> >> >> >> > >> > also don't want to be making proposals before > things > > are > > >> >> >> thought > > >> >> >> >> >> > through, > > >> >> >> >> >> > >> > which often requires writing the code. So I think > the > > >> right > > >> >> >> time > > >> >> >> >> >> for a > > >> >> >> >> >> > >> KIP > > >> >> >> >> >> > >> > is when you are far enough along that you know the > > >> issues > > >> >> and > > >> >> >> >> >> > tradeoffs > > >> >> >> >> >> > >> but > > >> >> >> >> >> > >> > not so far along that you are going to be totally > > >> opposed > > >> >> to > > >> >> >> any > > >> >> >> >> >> > change. > > >> >> >> >> >> > >> > Sometimes that is prior to writing any code and > > >> sometimes > > >> >> not > > >> >> >> >> until > > >> >> >> >> >> > you > > >> >> >> >> >> > >> are > > >> >> >> >> >> > >> > practically done. > > >> >> >> >> >> > >> > > > >> >> >> >> >> > >> > The key problem I see this fixing is that there is > > >> enough > > >> >> new > > >> >> >> >> >> > development > > >> >> >> >> >> > >> > happening that it is pretty hard for everyone to > > review > > >> >> every > > >> >> >> >> line > > >> >> >> >> >> of > > >> >> >> >> >> > >> every > > >> >> >> >> >> > >> > iteration of every patch. But all of us should be > > fully > > >> >> aware > > >> >> >> of > > >> >> >> >> new > > >> >> >> >> >> > >> > features, the ramifications, the new public > > interfaces, > > >> >> etc. > > >> >> >> If > > >> >> >> >> we > > >> >> >> >> >> > aren't > > >> >> >> >> >> > >> > aware of that we can't really build a holistic > system > > >> that > > >> >> is > > >> >> >> >> >> > beautiful > > >> >> >> >> >> > >> and > > >> >> >> >> >> > >> > consistent across. So the idea is that if you fully > > >> review > > >> >> the > > >> >> >> >> KIPs > > >> >> >> >> >> > you > > >> >> >> >> >> > >> can > > >> >> >> >> >> > >> > be sure that even if you don't know every new line > of > > >> code, > > >> >> >> you > > >> >> >> >> know > > >> >> >> >> >> > the > > >> >> >> >> >> > >> > major changes coming in. > > >> >> >> >> >> > >> > > > >> >> >> >> >> > >> > -Jay > > >> >> >> >> >> > >> > > > >> >> >> >> >> > >> > > > >> >> >> >> >> > >> > > > >> >> >> >> >> > >> > On Thu, Jan 15, 2015 at 12:18 PM, Joe Stein < > > >> >> >> >> joe.st...@stealth.ly> > > >> >> >> >> >> > >> wrote: > > >> >> >> >> >> > >> > > > >> >> >> >> >> > >> > > Thanks Jay for kicking this off! I think the > > >> confluence > > >> >> page > > >> >> >> >> you > > >> >> >> >> >> > wrote > > >> >> >> >> >> > >> up > > >> >> >> >> >> > >> > > is a great start. > > >> >> >> >> >> > >> > > > > >> >> >> >> >> > >> > > > > >> >> >> >> >> > >> > > The KIP makes sense to me (at a minimum) if there > > is > > >> >> going > > >> >> >> to > > >> >> >> >> be > > >> >> >> >> >> any > > >> >> >> >> >> > >> > > "breaking change". This way Kafka can continue to > > grow > > >> >> and > > >> >> >> >> blossom > > >> >> >> >> >> > and > > >> >> >> >> >> > >> we > > >> >> >> >> >> > >> > > have a process in place if we are going to > release > > a > > >> >> thorn > > >> >> >> ... > > >> >> >> >> and > > >> >> >> >> >> > >> when we > > >> >> >> >> >> > >> > > do it is *CLEAR* about what and why that is. We > can > > >> >> easily > > >> >> >> >> >> document > > >> >> >> >> >> > >> which > > >> >> >> >> >> > >> > > KIPs where involved with this release (which I > > think > > >> >> should > > >> >> >> get > > >> >> >> >> >> > >> committed > > >> >> >> >> >> > >> > > afterwards somewhere so no chance of edit after > > >> release). > > >> >> >> This > > >> >> >> >> >> > >> approach I > > >> >> >> >> >> > >> > > had been thinking about also allows changes to > > occur > > >> as > > >> >> >> they do > > >> >> >> >> >> now > > >> >> >> >> >> > as > > >> >> >> >> >> > >> long > > >> >> >> >> >> > >> > > as they are backwards compatible. Hopefully we > > never > > >> >> need a > > >> >> >> >> KIP > > >> >> >> >> >> but > > >> >> >> >> >> > >> when > > >> >> >> >> >> > >> > > we do the PMC can vote on it and folks can read > the > > >> >> release > > >> >> >> >> notes > > >> >> >> >> >> > with > > >> >> >> >> >> > >> > > *CLEAR* understanding what is going to break > their > > >> >> existing > > >> >> >> >> >> > setup... at > > >> >> >> >> >> > >> > > least that is how I have been thinking about it. > > >> >> >> >> >> > >> > > > > >> >> >> >> >> > >> > > > > >> >> >> >> >> > >> > > Let me know what you think about this base > minimum > > >> >> >> approach... > > >> >> >> >> I > > >> >> >> >> >> > hadn't > > >> >> >> >> >> > >> > > really thought of the KIP for *ANY* "major > change" > > >> and I > > >> >> >> have > > >> >> >> >> to > > >> >> >> >> >> > think > > >> >> >> >> >> > >> more > > >> >> >> >> >> > >> > > about that. I have some other comments for minor > > >> items in > > >> >> >> the > > >> >> >> >> >> > >> confluence > > >> >> >> >> >> > >> > > page I will make once I think more about how I > feel > > >> >> having a > > >> >> >> >> KIP > > >> >> >> >> >> for > > >> >> >> >> >> > >> more > > >> >> >> >> >> > >> > > than what I was thinking about already. > > >> >> >> >> >> > >> > > > > >> >> >> >> >> > >> > > > > >> >> >> >> >> > >> > > I do think we should have "major changes" go > > through > > >> >> >> >> confluence, > > >> >> >> >> >> > >> mailing > > >> >> >> >> >> > >> > > list discuss and JIRA but kind of feel we have > been > > >> doing > > >> >> >> that > > >> >> >> >> >> > already > > >> >> >> >> >> > >> ... > > >> >> >> >> >> > >> > > if there are cases where that isn't the case we > > should > > >> >> >> >> highlight > > >> >> >> >> >> and > > >> >> >> >> >> > >> learn > > >> >> >> >> >> > >> > > from them and formalize that more if need be. > > >> >> >> >> >> > >> > > > > >> >> >> >> >> > >> > > > > >> >> >> >> >> > >> > > /******************************************* > > >> >> >> >> >> > >> > > Joe Stein > > >> >> >> >> >> > >> > > Founder, Principal Consultant > > >> >> >> >> >> > >> > > Big Data Open Source Security LLC > > >> >> >> >> >> > >> > > http://www.stealth.ly > > >> >> >> >> >> > >> > > Twitter: @allthingshadoop < > > >> >> >> >> >> http://www.twitter.com/allthingshadoop> > > >> >> >> >> >> > >> > > ********************************************/ > > >> >> >> >> >> > >> > > > > >> >> >> >> >> > >> > > On Thu, Jan 15, 2015 at 1:42 PM, Jay Kreps < > > >> >> >> >> jay.kr...@gmail.com> > > >> >> >> >> >> > >> wrote: > > >> >> >> >> >> > >> > > > > >> >> >> >> >> > >> > > > The idea of KIPs came up in a previous > discussion > > >> but > > >> >> >> there > > >> >> >> >> was > > >> >> >> >> >> no > > >> >> >> >> >> > >> real > > >> >> >> >> >> > >> > > > crisp definition of what they were. Here is an > > >> attempt > > >> >> at > > >> >> >> >> >> > defining a > > >> >> >> >> >> > >> > > > process: > > >> >> >> >> >> > >> > > > > > >> >> >> >> >> > >> > > > > > >> >> >> >> >> > >> > > > > >> >> >> >> >> > >> > > >> >> >> >> >> > > > >> >> >> >> >> > > >> >> >> >> > > >> >> >> > > >> >> > > >> > > > https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals > > >> >> >> >> >> > >> > > > > > >> >> >> >> >> > >> > > > The trick here is to have something > light-weight > > >> enough > > >> >> >> that > > >> >> >> >> it > > >> >> >> >> >> > >> isn't a > > >> >> >> >> >> > >> > > > hassle for small changes, but enough so that > > changes > > >> >> get > > >> >> >> the > > >> >> >> >> >> > >> eyeballs of > > >> >> >> >> >> > >> > > > the committers and heavy users. > > >> >> >> >> >> > >> > > > > > >> >> >> >> >> > >> > > > Thoughts? > > >> >> >> >> >> > >> > > > > > >> >> >> >> >> > >> > > > -Jay > > >> >> >> >> >> > >> > > > > > >> >> >> >> >> > >> > > > > >> >> >> >> >> > >> > > >> >> >> >> >> > >> > > >> >> >> >> >> > > > > >> >> >> >> >> > > > > >> >> >> >> >> > > -- > > >> >> >> >> >> > > Thanks, > > >> >> >> >> >> > > Ewen > > >> >> >> >> >> > > > >> >> >> >> >> > > >> >> >> >> >> > > >> >> >> >> >> > > >> >> >> >> >> -- > > >> >> >> >> >> -- Guozhang > > >> >> >> >> >> > > >> >> >> >> > > >> >> >> > > >> >> > > >> > > >
