Sorry for late response, Magnus. See my comments inline:
On Fri, Jan 23, 2015 at 7:31 AM, Magnus Edenhill <mag...@edenhill.se> wrote:
> Wouldn't it make sense to move away from these rich binary broker
> descriptors ({ host, port, proto })
> (which require protocol churning on change), and simply use URIs instead?
We use different representations in different places, so I'm not sure
which one you mean...
Our clients will use host:port and --security.protocol in the
configuration to specify the protocol (this is to make it easier to
ensure one protocol per client)
ZK registration is in JSON
Internal objects are binary, since parsing the URI over and over will be a PITA
Wire protocol doesn't include a security protocol (since we don't
negotiate it), so its still host+port, as it always was.
>
> E.g.:
> kafka://<host>[:port]/ <-- cleantext proto on standard port 9092
> kafkas://<host>[:port] <-- SSL enveloped proto on standard port 9093
> kafkas://<user:pass>@<host>[:port]/ <-- SSL enveloped, with user
> authentication ..
> kafkafuturetech://.../#opts <-- six months from now.
>
> Trailing #fragment_ids could be used to hint the client on protocol
> versions, supported authentications, etc.
>
> This also makes error reporting more meaningful on the client, e.g compare:
> "Unsupported protocol 19 on broker foo:1234"
> to
> "Unsupported protocol kafkafturetech on broker foo:1234"
I agree that the second error is more readable, but I'm not sure why
you think its currently unfeasible on clients?
>
> A positive side effect would be a more generalized topic addressing in
> clients:
> kafkacat kafka://<bootstrap>/mytopic/3?offset=end <-- tail partition 3
> of mytopic
Clients can pretty much do what they want, right? As long as they call
Kafka with the right configuration, its up to them to decide how to
accept arguments.
> Just an idea,
> Magnus
>
>
> 2015-01-23 5:43 GMT+01:00 Jun Rao <j...@confluent.io>:
>
>> Reviewed the latest patch in KAFKA-1809 :).
>>
>> Thanks,
>>
>> Jun
>>
>> On Thu, Jan 22, 2015 at 12:38 PM, Gwen Shapira <gshap...@cloudera.com>
>> wrote:
>>
>> > Thanks for validating our ideas. Updated the KIP with the workflow.
>> >
>> > Now if you can nudge Jun to review the latest patch... ;)
>> >
>> >
>> > On Thu, Jan 22, 2015 at 11:44 AM, Jay Kreps <j...@confluent.io> wrote:
>> > > Oh yeah I think that is better, I hadn't thought of that approach! Any
>> > way
>> > > you could describe the usage in the KIP, just for completeness?
>> > >
>> > > -Jay
>> > >
>> > > On Thu, Jan 22, 2015 at 10:23 AM, Gwen Shapira <gshap...@cloudera.com>
>> > > wrote:
>> > >
>> > >> I think what you described was the original design, so no wonder you
>> > >> are confused :)
>> > >>
>> > >> Following suggestions from Jun, I changed it a bit. The current model
>> > is:
>> > >>
>> > >> - Clients (producers and consumers) need to know about the broker
>> > >> ports in advance. They don't need to know about all brokers, but they
>> > >> need to know at least one host:port pair that speaks the protocol they
>> > >> want to use. The change is that all host:port pairs in broker.list
>> > >> must be of the same protocol and match the security.protocol
>> > >> configuration parameter.
>> > >>
>> > >> - Client uses security.protocol configuration parameter to open a
>> > >> connection to one of the brokers and sends the good old
>> > >> MetadataRequest. The broker knows which port it got the connection on,
>> > >> therefore it knows which security protocol is expected (it needs to
>> > >> use the same protocol to accept the connection and respond), and
>> > >> therefore it can send a response that contains only the host:port
>> > >> pairs that are relevant to that protocol.
>> > >>
>> > >> - From the client side the MetadataResponse did not change - it
>> > >> contains a list of brokerId,host,port that the client can connect to.
>> > >> The fact that all those broker endpoints were chosen out of a larger
>> > >> collection to match the right protocol is irrelevant for the client.
>> > >>
>> > >> I really like the new design since it preserves a lot of the same
>> > >> configurations and APIs.
>> > >>
>> > >> Thoughts?
>> > >>
>> > >> Gwen
>> > >>
>> > >> On Thu, Jan 22, 2015 at 9:19 AM, Jay Kreps <jay.kr...@gmail.com>
>> wrote:
>> > >> > I think I am still confused. In addition to the
>> UpdateMetadataRequest
>> > >> don't
>> > >> > we have to change the MetadataResponse so that it's possible for
>> > clients
>> > >> to
>> > >> > discover the new ports? Or is that a second phase? I was imagining
>> it
>> > >> > worked by basically allowing the brokers to advertise multiple
>> ports,
>> > one
>> > >> > per security type, and then in the client you configure a protocol
>> > which
>> > >> > will implicitly choose the port from the options returned in
>> metadata
>> > to
>> > >> > you...
>> > >> >
>> > >> > Likewise in the ConsumerMetadataResponse we are currently giving
>> back
>> > >> full
>> > >> > broker information. I think we would have two options here: either
>> > change
>> > >> > the broker information included in that response to match the
>> > >> > metadataresponse or else remove the broker information entirely and
>> > just
>> > >> > return the node id (since in order to use that request you would
>> > already
>> > >> > have to have the cluster metadata). The second option may be cleaner
>> > >> since
>> > >> > it means we won't have to continue evolving those two in lockstep...
>> > >> >
>> > >> > -Jay
>> > >> >
>> > >> > On Wed, Jan 21, 2015 at 6:19 PM, Gwen Shapira <
>> gshap...@cloudera.com>
>> > >> wrote:
>> > >> >
>> > >> >> Good point :)
>> > >> >>
>> > >> >> I added the specifics of the new UpdateMetadataRequest, which is
>> the
>> > >> >> only protocol bump in this change.
>> > >> >>
>> > >> >> Highlighted the broker and producer/consumer configuration changes,
>> > >> >> added some example values and added the new zookeeper json.
>> > >> >>
>> > >> >> Hope this makes things clearer.
>> > >> >>
>> > >> >> On Wed, Jan 21, 2015 at 2:19 PM, Jay Kreps <jay.kr...@gmail.com>
>> > wrote:
>> > >> >> > Hey Gwen,
>> > >> >> >
>> > >> >> > Could we get the actual changes in that KIP? I.e. changes to
>> > metadata
>> > >> >> > request, changes to UpdateMetadataRequest, new configs and what
>> > will
>> > >> >> their
>> > >> >> > valid values be, etc. This kind of says that those things will
>> > change
>> > >> but
>> > >> >> > doesn't say what they will change to...
>> > >> >> >
>> > >> >> > -Jay
>> > >> >> >
>> > >> >> > On Mon, Jan 19, 2015 at 9:45 PM, Gwen Shapira <
>> > gshap...@cloudera.com>
>> > >> >> wrote:
>> > >> >> >
>> > >> >> >> I created a KIP for the multi-port broker change.
>> > >> >> >>
>> > >> >> >>
>> > >> >>
>> > >>
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-2+-+Refactor+brokers+to+allow+listening+on+multiple+ports+and+IPs
>> > >> >> >>
>> > >> >> >> I'm not re-opening the discussion, since it was agreed on over a
>> > >> month
>> > >> >> >> ago and implementation is close to complete (I hope!). Lets
>> > consider
>> > >> >> >> this voted and accepted?
>> > >> >> >>
>> > >> >> >> Gwen
>> > >> >> >>
>> > >> >> >> On Sun, Jan 18, 2015 at 10:31 AM, Jay Kreps <
>> jay.kr...@gmail.com>
>> > >> >> wrote:
>> > >> >> >> > Great! Sounds like everyone is on the same page
>> > >> >> >> >
>> > >> >> >> > - I created a template page to make things easier. If you
>> do
>> > >> >> >> Tools->Copy
>> > >> >> >> > on this page you can just fill in the italic portions with
>> > your
>> > >> >> >> details.
>> > >> >> >> > - I retrofitted KIP-1 to match this formatting
>> > >> >> >> > - I added the metadata section people asked for (a link to
>> > the
>> > >> >> >> > discussion, the JIRA, and the current status). Let's make
>> > sure
>> > >> we
>> > >> >> >> remember
>> > >> >> >> > to update the current status as things are figured out.
>> > >> >> >> > - Let's keep the discussion on the mailing list rather than
>> > on
>> > >> the
>> > >> >> >> wiki
>> > >> >> >> > pages. It makes sense to do one or the other so all the
>> > comments
>> > >> >> are
>> > >> >> >> in one
>> > >> >> >> > place and I think prior experience is that the wiki
>> comments
>> > are
>> > >> >> the
>> > >> >> >> worse
>> > >> >> >> > way.
>> > >> >> >> >
>> > >> >> >> > I think it would be great do KIPs for some of the in-flight
>> > items
>> > >> >> folks
>> > >> >> >> > mentioned.
>> > >> >> >> >
>> > >> >> >> > -Jay
>> > >> >> >> >
>> > >> >> >> > On Sat, Jan 17, 2015 at 8:23 AM, Gwen Shapira <
>> > >> gshap...@cloudera.com>
>> > >> >> >> wrote:
>> > >> >> >> >
>> > >> >> >> >> +1
>> > >> >> >> >>
>> > >> >> >> >> Will be happy to provide a KIP for the multiple-listeners
>> > patch.
>> > >> >> >> >>
>> > >> >> >> >> Gwen
>> > >> >> >> >>
>> > >> >> >> >> On Sat, Jan 17, 2015 at 8:10 AM, Joe Stein <
>> > joe.st...@stealth.ly>
>> > >> >> >> wrote:
>> > >> >> >> >> > +1 to everything we have been saying and where this (has
>> > settled
>> > >> >> >> to)/(is
>> > >> >> >> >> > settling to).
>> > >> >> >> >> >
>> > >> >> >> >> > I am sure other folks have some more feedback and think we
>> > >> should
>> > >> >> try
>> > >> >> >> to
>> > >> >> >> >> > keep this discussion going if need be. I am also a firm
>> > >> believer of
>> > >> >> >> form
>> > >> >> >> >> > following function so kicking the tires some to flesh out
>> the
>> > >> >> details
>> > >> >> >> of
>> > >> >> >> >> > this and have some organic growth with the process will be
>> > >> healthy
>> > >> >> and
>> > >> >> >> >> > beneficial to the community.
>> > >> >> >> >> >
>> > >> >> >> >> > For my part, what I will do is open a few KIP based on some
>> > of
>> > >> the
>> > >> >> >> work I
>> > >> >> >> >> > have been involved with for 0.8.3. Off the top of my head
>> > this
>> > >> >> would
>> > >> >> >> >> > include 1) changes to re-assignment of partitions 2) kafka
>> > cli
>> > >> 3)
>> > >> >> >> global
>> > >> >> >> >> > configs 4) security white list black list by ip 5) SSL 6)
>> We
>> > >> >> probably
>> > >> >> >> >> will
>> > >> >> >> >> > have lots of Security related KIPs and should treat them
>> all
>> > >> >> >> individually
>> > >> >> >> >> > when the time is appropriate 7) Kafka on Mesos.
>> > >> >> >> >> >
>> > >> >> >> >> > If someone else wants to jump in to start getting some of
>> the
>> > >> >> security
>> > >> >> >> >> KIP
>> > >> >> >> >> > that we are going to have in 0.8.3 I think that would be
>> > great
>> > >> >> (e.g.
>> > >> >> >> >> > Multiple Listeners for Kafka Brokers). There are also a few
>> > >> other
>> > >> >> >> >> tickets I
>> > >> >> >> >> > can think of that are important to have in the code in
>> 0.8.3
>> > >> that
>> > >> >> >> should
>> > >> >> >> >> > have KIP also that I haven't really been involved in. I
>> will
>> > >> take a
>> > >> >> >> few
>> > >> >> >> >> > minutes and go through JIRA (one I can think of like auto
>> > >> assign id
>> > >> >> >> that
>> > >> >> >> >> is
>> > >> >> >> >> > already committed I think) and ask for a KIP if appropriate
>> > or
>> > >> if I
>> > >> >> >> feel
>> > >> >> >> >> > that I can write it up (both from a time and understanding
>> > >> >> >> perspective)
>> > >> >> >> >> do
>> > >> >> >> >> > so.
>> > >> >> >> >> >
>> > >> >> >> >> > long story short, I encourage folks to start moving ahead
>> > with
>> > >> the
>> > >> >> KIP
>> > >> >> >> >> for
>> > >> >> >> >> > 0.8.3 as how we operate. any objections?
>> > >> >> >> >> >
>> > >> >> >> >> > On Fri, Jan 16, 2015 at 2:40 PM, Guozhang Wang <
>> > >> wangg...@gmail.com
>> > >> >> >
>> > >> >> >> >> wrote:
>> > >> >> >> >> >
>> > >> >> >> >> >> +1 on the idea, and we could mutually link the KIP wiki
>> page
>> > >> with
>> > >> >> the
>> > >> >> >> >> the
>> > >> >> >> >> >> created JIRA ticket (i.e. include the JIRA number on the
>> > page
>> > >> and
>> > >> >> the
>> > >> >> >> >> KIP
>> > >> >> >> >> >> url on the ticket description).
>> > >> >> >> >> >>
>> > >> >> >> >> >> Regarding the KIP process, probably we do not need two
>> phase
>> > >> >> >> >> communication
>> > >> >> >> >> >> of a [DISCUSS] followed by [VOTE], as Jay said the voting
>> > >> should
>> > >> >> be
>> > >> >> >> >> clear
>> > >> >> >> >> >> while people discuss about that.
>> > >> >> >> >> >>
>> > >> >> >> >> >> About who should trigger the process, I think the only
>> > involved
>> > >> >> >> people
>> > >> >> >> >> >> would be 1) when the patch is submitted / or even the
>> > ticket is
>> > >> >> >> created,
>> > >> >> >> >> >> the assignee could choose to start the KIP process if she
>> > >> thought
>> > >> >> it
>> > >> >> >> is
>> > >> >> >> >> >> necessary; 2) the reviewer of the patch can also suggest
>> > >> starting
>> > >> >> KIP
>> > >> >> >> >> >> discussions.
>> > >> >> >> >> >>
>> > >> >> >> >> >> On Fri, Jan 16, 2015 at 10:49 AM, Gwen Shapira <
>> > >> >> >> gshap...@cloudera.com>
>> > >> >> >> >> >> wrote:
>> > >> >> >> >> >>
>> > >> >> >> >> >> > +1 to Ewen's suggestions: Deprecation, status and
>> version.
>> > >> >> >> >> >> >
>> > >> >> >> >> >> > Perhaps add the JIRA where the KIP was implemented to
>> the
>> > >> >> metadata.
>> > >> >> >> >> >> > This will help tie things together.
>> > >> >> >> >> >> >
>> > >> >> >> >> >> > On Fri, Jan 16, 2015 at 9:35 AM, Ewen Cheslack-Postava
>> > >> >> >> >> >> > <e...@confluent.io> wrote:
>> > >> >> >> >> >> > > I think adding a section about deprecation would be
>> > >> helpful. A
>> > >> >> >> good
>> > >> >> >> >> >> > > fraction of the time I would expect the goal of a KIP
>> > is to
>> > >> >> fix
>> > >> >> >> or
>> > >> >> >> >> >> > replace
>> > >> >> >> >> >> > > older functionality that needs continued support for
>> > >> >> >> compatibility,
>> > >> >> >> >> but
>> > >> >> >> >> >> > > should eventually be phased out. This helps Kafka devs
>> > >> >> understand
>> > >> >> >> >> how
>> > >> >> >> >> >> > long
>> > >> >> >> >> >> > > they'll end up supporting multiple versions of
>> features
>> > and
>> > >> >> helps
>> > >> >> >> >> users
>> > >> >> >> >> >> > > understand when they're going to have to make updates
>> to
>> > >> their
>> > >> >> >> >> >> > applications.
>> > >> >> >> >> >> > >
>> > >> >> >> >> >> > > Less important but useful -- having a bit of standard
>> > >> metadata
>> > >> >> >> like
>> > >> >> >> >> >> PEPs
>> > >> >> >> >> >> > > do. Two I think are important are status (if someone
>> > lands
>> > >> on
>> > >> >> the
>> > >> >> >> >> KIP
>> > >> >> >> >> >> > page,
>> > >> >> >> >> >> > > can they tell whether this KIP was ever completed?)
>> > and/or
>> > >> the
>> > >> >> >> >> version
>> > >> >> >> >> >> > the
>> > >> >> >> >> >> > > KIP was first released in.
>> > >> >> >> >> >> > >
>> > >> >> >> >> >> > >
>> > >> >> >> >> >> > >
>> > >> >> >> >> >> > > On Fri, Jan 16, 2015 at 9:20 AM, Joel Koshy <
>> > >> >> jjkosh...@gmail.com
>> > >> >> >> >
>> > >> >> >> >> >> wrote:
>> > >> >> >> >> >> > >
>> > >> >> >> >> >> > >> I'm definitely +1 on the KIP concept. As Joe
>> > mentioned, we
>> > >> >> are
>> > >> >> >> >> already
>> > >> >> >> >> >> > >> doing this in one form or the other. However, IMO it
>> is
>> > >> >> fairly
>> > >> >> >> ad
>> > >> >> >> >> hoc
>> > >> >> >> >> >> > >> - i.e., a combination of DISCUSS threads, jira
>> > comments,
>> > >> RB
>> > >> >> and
>> > >> >> >> >> code
>> > >> >> >> >> >> > >> comments, wikis and html documentation. In the past I
>> > have
>> > >> >> had
>> > >> >> >> to
>> > >> >> >> >> dig
>> > >> >> >> >> >> > >> into a bunch of these to try and figure out why
>> > something
>> > >> was
>> > >> >> >> >> >> > >> implemented a certain way. I think KIPs can help a
>> lot
>> > >> here
>> > >> >> >> first
>> > >> >> >> >> by
>> > >> >> >> >> >> > >> providing guidelines on what to think about
>> > >> (compatibility,
>> > >> >> new
>> > >> >> >> >> APIs,
>> > >> >> >> >> >> > >> etc.) when working through a major feature; and
>> second
>> > by
>> > >> >> >> becoming
>> > >> >> >> >> a
>> > >> >> >> >> >> > >> crisp source of truth documentation for new releases.
>> > >> E.g.,
>> > >> >> for
>> > >> >> >> >> >> > >> feature X: see relevant KIPs: a, b, c, etc.
>> > >> >> >> >> >> > >>
>> > >> >> >> >> >> > >> On Thu, Jan 15, 2015 at 08:11:20PM -0800, Jay Kreps
>> > wrote:
>> > >> >> >> >> >> > >> > Hey Joe,
>> > >> >> >> >> >> > >> >
>> > >> >> >> >> >> > >> > Yeah I guess the question is what is the definition
>> > of
>> > >> >> major?
>> > >> >> >> I
>> > >> >> >> >> >> agree
>> > >> >> >> >> >> > we
>> > >> >> >> >> >> > >> > definitely don't want to generate a bunch of
>> > paperwork.
>> > >> We
>> > >> >> >> have
>> > >> >> >> >> >> enough
>> > >> >> >> >> >> > >> > problems just getting all the contributions
>> reviewed
>> > and
>> > >> >> >> checked
>> > >> >> >> >> in
>> > >> >> >> >> >> > in a
>> > >> >> >> >> >> > >> > timely fashion...
>> > >> >> >> >> >> > >> >
>> > >> >> >> >> >> > >> > So obviously bug fixes would not apply here.
>> > >> >> >> >> >> > >> >
>> > >> >> >> >> >> > >> > I think it is also pretty clear that big features
>> > should
>> > >> >> get
>> > >> >> >> >> >> reviewed
>> > >> >> >> >> >> > and
>> > >> >> >> >> >> > >> > discussed. To pick on myself, for example, the log
>> > >> >> compaction
>> > >> >> >> >> work
>> > >> >> >> >> >> was
>> > >> >> >> >> >> > >> done
>> > >> >> >> >> >> > >> > without enough public discussion about how it
>> worked
>> > and
>> > >> >> why
>> > >> >> >> >> >> (imho). I
>> > >> >> >> >> >> > >> > hope/claim that enough rigour in thinking about
>> > >> use-cases
>> > >> >> and
>> > >> >> >> >> >> > operations
>> > >> >> >> >> >> > >> > and so on was done that it turned out well, but the
>> > >> >> discussion
>> > >> >> >> >> was
>> > >> >> >> >> >> > just
>> > >> >> >> >> >> > >> > between a few people with no real public output.
>> This
>> > >> kind
>> > >> >> of
>> > >> >> >> >> >> feature
>> > >> >> >> >> >> > is
>> > >> >> >> >> >> > >> > clearly a big change and something we should
>> discuss.
>> > >> >> >> >> >> > >> >
>> > >> >> >> >> >> > >> > If we limit ourselves to just the public contracts
>> > the
>> > >> KIP
>> > >> >> >> >> >> introduces
>> > >> >> >> >> >> > the
>> > >> >> >> >> >> > >> > discussion would just be on the new configs and
>> > >> monitoring
>> > >> >> >> >> without
>> > >> >> >> >> >> > >> really a
>> > >> >> >> >> >> > >> > discussion of the design and how it works which is
>> > >> >> obviously
>> > >> >> >> >> closely
>> > >> >> >> >> >> > >> > related.
>> > >> >> >> >> >> > >> >
>> > >> >> >> >> >> > >> > I don't think this should be more work because in
>> > >> practice
>> > >> >> we
>> > >> >> >> are
>> > >> >> >> >> >> > making
>> > >> >> >> >> >> > >> > wiki pages for any big thing anyway. So this would
>> > just
>> > >> be
>> > >> >> a
>> > >> >> >> >> >> > consistent
>> > >> >> >> >> >> > >> way
>> > >> >> >> >> >> > >> > of numbering and structuring these pages. This
>> would
>> > >> also
>> > >> >> >> give a
>> > >> >> >> >> >> clear
>> > >> >> >> >> >> > >> call
>> > >> >> >> >> >> > >> > to action: "hey kafka people, come read my wiki and
>> > >> think
>> > >> >> this
>> > >> >> >> >> >> > through".
>> > >> >> >> >> >> > >> >
>> > >> >> >> >> >> > >> > I actually thinking the voting aspect is less
>> > >> important. I
>> > >> >> >> think
>> > >> >> >> >> it
>> > >> >> >> >> >> is
>> > >> >> >> >> >> > >> > generally clear when there is agreement on
>> something
>> > and
>> > >> >> not.
>> > >> >> >> So
>> > >> >> >> >> >> from
>> > >> >> >> >> >> > my
>> > >> >> >> >> >> > >> > point of view we could actually just eliminate that
>> > >> part if
>> > >> >> >> that
>> > >> >> >> >> is
>> > >> >> >> >> >> > too
>> > >> >> >> >> >> > >> > formal, it just seemed like a good way to formally
>> > adopt
>> > >> >> >> >> something.
>> > >> >> >> >> >> > >> >
>> > >> >> >> >> >> > >> > To address some of your comments from the wiki:
>> > >> >> >> >> >> > >> >
>> > >> >> >> >> >> > >> > 1. This doesn't inhibit someone coming along and
>> > putting
>> > >> >> up a
>> > >> >> >> >> patch.
>> > >> >> >> >> >> > It
>> > >> >> >> >> >> > >> is
>> > >> >> >> >> >> > >> > just that when they do if it is a big thing
>> > introducing
>> > >> new
>> > >> >> >> >> >> > functionality
>> > >> >> >> >> >> > >> > we would ask for a little discussion on the basic
>> > >> >> >> >> feature/contracts
>> > >> >> >> >> >> > prior
>> > >> >> >> >> >> > >> > to code review.
>> > >> >> >> >> >> > >> >
>> > >> >> >> >> >> > >> > 2. We definitely definitely don't want people
>> > >> generating a
>> > >> >> >> lot of
>> > >> >> >> >> >> > these
>> > >> >> >> >> >> > >> > things every time they have an idea that they
>> aren't
>> > >> going
>> > >> >> to
>> > >> >> >> >> >> > implement.
>> > >> >> >> >> >> > >> So
>> > >> >> >> >> >> > >> > this is only applicable to things you absolutely
>> will
>> > >> >> check in
>> > >> >> >> >> code
>> > >> >> >> >> >> > for.
>> > >> >> >> >> >> > >> We
>> > >> >> >> >> >> > >> > also don't want to be making proposals before
>> things
>> > are
>> > >> >> >> thought
>> > >> >> >> >> >> > through,
>> > >> >> >> >> >> > >> > which often requires writing the code. So I think
>> the
>> > >> right
>> > >> >> >> time
>> > >> >> >> >> >> for a
>> > >> >> >> >> >> > >> KIP
>> > >> >> >> >> >> > >> > is when you are far enough along that you know the
>> > >> issues
>> > >> >> and
>> > >> >> >> >> >> > tradeoffs
>> > >> >> >> >> >> > >> but
>> > >> >> >> >> >> > >> > not so far along that you are going to be totally
>> > >> opposed
>> > >> >> to
>> > >> >> >> any
>> > >> >> >> >> >> > change.
>> > >> >> >> >> >> > >> > Sometimes that is prior to writing any code and
>> > >> sometimes
>> > >> >> not
>> > >> >> >> >> until
>> > >> >> >> >> >> > you
>> > >> >> >> >> >> > >> are
>> > >> >> >> >> >> > >> > practically done.
>> > >> >> >> >> >> > >> >
>> > >> >> >> >> >> > >> > The key problem I see this fixing is that there is
>> > >> enough
>> > >> >> new
>> > >> >> >> >> >> > development
>> > >> >> >> >> >> > >> > happening that it is pretty hard for everyone to
>> > review
>> > >> >> every
>> > >> >> >> >> line
>> > >> >> >> >> >> of
>> > >> >> >> >> >> > >> every
>> > >> >> >> >> >> > >> > iteration of every patch. But all of us should be
>> > fully
>> > >> >> aware
>> > >> >> >> of
>> > >> >> >> >> new
>> > >> >> >> >> >> > >> > features, the ramifications, the new public
>> > interfaces,
>> > >> >> etc.
>> > >> >> >> If
>> > >> >> >> >> we
>> > >> >> >> >> >> > aren't
>> > >> >> >> >> >> > >> > aware of that we can't really build a holistic
>> system
>> > >> that
>> > >> >> is
>> > >> >> >> >> >> > beautiful
>> > >> >> >> >> >> > >> and
>> > >> >> >> >> >> > >> > consistent across. So the idea is that if you fully
>> > >> review
>> > >> >> the
>> > >> >> >> >> KIPs
>> > >> >> >> >> >> > you
>> > >> >> >> >> >> > >> can
>> > >> >> >> >> >> > >> > be sure that even if you don't know every new line
>> of
>> > >> code,
>> > >> >> >> you
>> > >> >> >> >> know
>> > >> >> >> >> >> > the
>> > >> >> >> >> >> > >> > major changes coming in.
>> > >> >> >> >> >> > >> >
>> > >> >> >> >> >> > >> > -Jay
>> > >> >> >> >> >> > >> >
>> > >> >> >> >> >> > >> >
>> > >> >> >> >> >> > >> >
>> > >> >> >> >> >> > >> > On Thu, Jan 15, 2015 at 12:18 PM, Joe Stein <
>> > >> >> >> >> joe.st...@stealth.ly>
>> > >> >> >> >> >> > >> wrote:
>> > >> >> >> >> >> > >> >
>> > >> >> >> >> >> > >> > > Thanks Jay for kicking this off! I think the
>> > >> confluence
>> > >> >> page
>> > >> >> >> >> you
>> > >> >> >> >> >> > wrote
>> > >> >> >> >> >> > >> up
>> > >> >> >> >> >> > >> > > is a great start.
>> > >> >> >> >> >> > >> > >
>> > >> >> >> >> >> > >> > >
>> > >> >> >> >> >> > >> > > The KIP makes sense to me (at a minimum) if there
>> > is
>> > >> >> going
>> > >> >> >> to
>> > >> >> >> >> be
>> > >> >> >> >> >> any
>> > >> >> >> >> >> > >> > > "breaking change". This way Kafka can continue to
>> > grow
>> > >> >> and
>> > >> >> >> >> blossom
>> > >> >> >> >> >> > and
>> > >> >> >> >> >> > >> we
>> > >> >> >> >> >> > >> > > have a process in place if we are going to
>> release
>> > a
>> > >> >> thorn
>> > >> >> >> ...
>> > >> >> >> >> and
>> > >> >> >> >> >> > >> when we
>> > >> >> >> >> >> > >> > > do it is *CLEAR* about what and why that is. We
>> can
>> > >> >> easily
>> > >> >> >> >> >> document
>> > >> >> >> >> >> > >> which
>> > >> >> >> >> >> > >> > > KIPs where involved with this release (which I
>> > think
>> > >> >> should
>> > >> >> >> get
>> > >> >> >> >> >> > >> committed
>> > >> >> >> >> >> > >> > > afterwards somewhere so no chance of edit after
>> > >> release).
>> > >> >> >> This
>> > >> >> >> >> >> > >> approach I
>> > >> >> >> >> >> > >> > > had been thinking about also allows changes to
>> > occur
>> > >> as
>> > >> >> >> they do
>> > >> >> >> >> >> now
>> > >> >> >> >> >> > as
>> > >> >> >> >> >> > >> long
>> > >> >> >> >> >> > >> > > as they are backwards compatible. Hopefully we
>> > never
>> > >> >> need a
>> > >> >> >> >> KIP
>> > >> >> >> >> >> but
>> > >> >> >> >> >> > >> when
>> > >> >> >> >> >> > >> > > we do the PMC can vote on it and folks can read
>> the
>> > >> >> release
>> > >> >> >> >> notes
>> > >> >> >> >> >> > with
>> > >> >> >> >> >> > >> > > *CLEAR* understanding what is going to break
>> their
>> > >> >> existing
>> > >> >> >> >> >> > setup... at
>> > >> >> >> >> >> > >> > > least that is how I have been thinking about it.
>> > >> >> >> >> >> > >> > >
>> > >> >> >> >> >> > >> > >
>> > >> >> >> >> >> > >> > > Let me know what you think about this base
>> minimum
>> > >> >> >> approach...
>> > >> >> >> >> I
>> > >> >> >> >> >> > hadn't
>> > >> >> >> >> >> > >> > > really thought of the KIP for *ANY* "major
>> change"
>> > >> and I
>> > >> >> >> have
>> > >> >> >> >> to
>> > >> >> >> >> >> > think
>> > >> >> >> >> >> > >> more
>> > >> >> >> >> >> > >> > > about that. I have some other comments for minor
>> > >> items in
>> > >> >> >> the
>> > >> >> >> >> >> > >> confluence
>> > >> >> >> >> >> > >> > > page I will make once I think more about how I
>> feel
>> > >> >> having a
>> > >> >> >> >> KIP
>> > >> >> >> >> >> for
>> > >> >> >> >> >> > >> more
>> > >> >> >> >> >> > >> > > than what I was thinking about already.
>> > >> >> >> >> >> > >> > >
>> > >> >> >> >> >> > >> > >
>> > >> >> >> >> >> > >> > > I do think we should have "major changes" go
>> > through
>> > >> >> >> >> confluence,
>> > >> >> >> >> >> > >> mailing
>> > >> >> >> >> >> > >> > > list discuss and JIRA but kind of feel we have
>> been
>> > >> doing
>> > >> >> >> that
>> > >> >> >> >> >> > already
>> > >> >> >> >> >> > >> ...
>> > >> >> >> >> >> > >> > > if there are cases where that isn't the case we
>> > should
>> > >> >> >> >> highlight
>> > >> >> >> >> >> and
>> > >> >> >> >> >> > >> learn
>> > >> >> >> >> >> > >> > > from them and formalize that more if need be.
>> > >> >> >> >> >> > >> > >
>> > >> >> >> >> >> > >> > >
>> > >> >> >> >> >> > >> > > /*******************************************
>> > >> >> >> >> >> > >> > > Joe Stein
>> > >> >> >> >> >> > >> > > Founder, Principal Consultant
>> > >> >> >> >> >> > >> > > Big Data Open Source Security LLC
>> > >> >> >> >> >> > >> > > http://www.stealth.ly
>> > >> >> >> >> >> > >> > > Twitter: @allthingshadoop <
>> > >> >> >> >> >> http://www.twitter.com/allthingshadoop>
>> > >> >> >> >> >> > >> > > ********************************************/
>> > >> >> >> >> >> > >> > >
>> > >> >> >> >> >> > >> > > On Thu, Jan 15, 2015 at 1:42 PM, Jay Kreps <
>> > >> >> >> >> jay.kr...@gmail.com>
>> > >> >> >> >> >> > >> wrote:
>> > >> >> >> >> >> > >> > >
>> > >> >> >> >> >> > >> > > > The idea of KIPs came up in a previous
>> discussion
>> > >> but
>> > >> >> >> there
>> > >> >> >> >> was
>> > >> >> >> >> >> no
>> > >> >> >> >> >> > >> real
>> > >> >> >> >> >> > >> > > > crisp definition of what they were. Here is an
>> > >> attempt
>> > >> >> at
>> > >> >> >> >> >> > defining a
>> > >> >> >> >> >> > >> > > > process:
>> > >> >> >> >> >> > >> > > >
>> > >> >> >> >> >> > >> > > >
>> > >> >> >> >> >> > >> > >
>> > >> >> >> >> >> > >>
>> > >> >> >> >> >> >
>> > >> >> >> >> >>
>> > >> >> >> >>
>> > >> >> >>
>> > >> >>
>> > >>
>> >
>> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals
>> > >> >> >> >> >> > >> > > >
>> > >> >> >> >> >> > >> > > > The trick here is to have something
>> light-weight
>> > >> enough
>> > >> >> >> that
>> > >> >> >> >> it
>> > >> >> >> >> >> > >> isn't a
>> > >> >> >> >> >> > >> > > > hassle for small changes, but enough so that
>> > changes
>> > >> >> get
>> > >> >> >> the
>> > >> >> >> >> >> > >> eyeballs of
>> > >> >> >> >> >> > >> > > > the committers and heavy users.
>> > >> >> >> >> >> > >> > > >
>> > >> >> >> >> >> > >> > > > Thoughts?
>> > >> >> >> >> >> > >> > > >
>> > >> >> >> >> >> > >> > > > -Jay
>> > >> >> >> >> >> > >> > > >
>> > >> >> >> >> >> > >> > >
>> > >> >> >> >> >> > >>
>> > >> >> >> >> >> > >>
>> > >> >> >> >> >> > >
>> > >> >> >> >> >> > >
>> > >> >> >> >> >> > > --
>> > >> >> >> >> >> > > Thanks,
>> > >> >> >> >> >> > > Ewen
>> > >> >> >> >> >> >
>> > >> >> >> >> >>
>> > >> >> >> >> >>
>> > >> >> >> >> >>
>> > >> >> >> >> >> --
>> > >> >> >> >> >> -- Guozhang
>> > >> >> >> >> >>
>> > >> >> >> >>
>> > >> >> >>
>> > >> >>
>> > >>
>> >
>>
