Never ever changing the version of VersionReq/resp will be ideal, but it is hard to guarantee that we will never need a change. We should definitely try to avoid it and honestly I do not see a reason, as of now, why it will change. I would say, we should keep it versioned and document any risk of changing it. Does that sound OK?
On Fri, Apr 1, 2016 at 1:12 PM, Gwen Shapira <g...@confluent.io> wrote: > I am happy with: > 1. Adding only validation support in the client for now, deferring backward > compatibility for later. With the odd side-effect of having the C and > Python clients actually better than Java for a bit. > 2. Never ever changing the version of VersionRequest and VersionResponse. > > I'd also like to see: > 1. At least one client implement backward compatibility (so we can use it > in testing) > 2. Backward compatibility tests using that client (including an upgrade > test) > 3. Checklist for reviewers on when protocol bumps are required > > I am strongly against releasing an API that no one is using as intended... > so either we do it in our client or use a 3rd party client. > > Gwen > > On Fri, Apr 1, 2016 at 1:05 PM, Jason Gustafson <ja...@confluent.io> > wrote: > > > I'm sympathetic to Jay's argument, but I feel the version API can be > > justified on its own without requiring a big change in the way we manage > > compatibility. For the official client compatibility story (i.e. what the > > Java client implements), we can use this API to verify compatibility with > > the broker, which is a significant gap at the moment (we can do this now > > before the release if we all agree without much delay). It also happens > to > > help clients trying to maintain backwards compatibility, which is great. > > > > Maybe the question is whether this API will restrict later efforts to > > change the way we manage compatibility? I don't see much risk in terms of > > maintenance of the API, but there is some risk that it could become > > superfluous at some point. For example, if we changed the response header > > to include an error code for an unknown request version, then maybe the > > Java client would no longer need the initial call to the version API > > (though it might arguably still be a good idea to do so to detect > > incompatibility sooner). Maybe there is some risk also that it makes the > > compatibility story more confusing? I actually think it fits pretty well > > with the versioned API approach generally, though it's admittedly a > little > > odd to have the version API versioned itself. Practically speaking, I > think > > this means that we shouldn't increment the version. Otherwise the client > > will need another round trip (after always starting with v0), which might > > be doable even if it seems a little messy. > > > > Thoughts? > > > > -Jason > > > > On Fri, Apr 1, 2016 at 11:36 AM, Gwen Shapira <g...@confluent.io> wrote: > > > > > My understanding is that we are not adding the client portion to the > KIP > > is > > > because we believe it will require a lot of discussion (read: will be > > > really hard to get right). > > > > > > Maybe it is a hint that this protocol is too difficult for clients to > > > implement? > > > I can't see why is it easy in C and Python and super difficult in Java. > > > > > > Even if it is too hard to implement just in Java for some reason (god > > knows > > > the language has issues), isn't it a good reason to come up with > > something > > > we can implement in a reasonable way? > > > > > > Gwen > > > > > > On Fri, Apr 1, 2016 at 10:58 AM, Ashish Singh <asi...@cloudera.com> > > wrote: > > > > > > > That is a fair concern and I think eventually we might want to have > > java > > > > clients backwards compatible. However, blocking KIP-35 on that might > > not > > > be > > > > the best idea. The reason I say so is due to following reasons. > > > > > > > > 1. Backwards compatibility in java clients is a larger discussion, as > > we > > > > have already seen. Having a separate KIP focussing exactly on that > will > > > > help in reducing moving pieces in one KIP. > > > > 2. It probably will fall out of 0.10 due to tight timeline, I am > > assuming > > > > we do not want to delay 0.10 a lot. > > > > 3. Even if we make java clients backward compatible in 0.10, I do not > > > think > > > > it will be able to work with older releases as older broker versions > > > still > > > > won't provide info on supported api versions. If we add api versions > > > > req/resp in 0.10, and add backwards compatibility in java clients in > > > later > > > > versions, they will probably work with 0.10 and we will be able to > test > > > > that. > > > > > > > > On Fri, Apr 1, 2016 at 10:28 AM, Ewen Cheslack-Postava < > > > e...@confluent.io> > > > > wrote: > > > > > > > > > On Fri, Apr 1, 2016 at 10:22 AM, Gwen Shapira <g...@confluent.io> > > > wrote: > > > > > > > > > > > I have two concerns with the proposal as it is: > > > > > > > > > > > > 1. Having an API that publishes protocol is useless for clients > if > > > > > > developers don't bump the API when they should. > > > > > > I would like to see good documentation on when protocols are > bumped > > > > and a > > > > > > proposal on how this will be automatically validated to the > extent > > > > > possible > > > > > > (and to what extent are we still at risk of accidental breakage). > > > > > > Even if we don't implement all of this at first pass, I want to > > know > > > > > which > > > > > > direction we are going in order to solve the client compatibility > > > > issue. > > > > > > > > > > > > 2. In addition to third-party clients, there are stream > processing > > > > > > frameworks who use the Java client we publish as their client and > > > would > > > > > > also like to enjoy the same compatibility benefits C and Python > > > clients > > > > > > will enjoy. It will be very silly if Apache Kafka clients are the > > > worst > > > > > > clients out there from compatibility POV. The KIP can be > > implemented > > > in > > > > > > parts, but I really want to see an effort to build the Java > client > > > > > > compatibility into the KIP and if possible into the release too. > > > > > > > > > > > > > > > > I wouldn't conflate those two things. Changing the compatibility > > > approach > > > > > of the Java clients could easily be another KIP (and probably a > large > > > > one, > > > > > too). There are already high quality, well maintained clients > trying > > to > > > > use > > > > > a different approach and this addresses their needs. Blocking the > > > entire > > > > > ecosystem on the Java client seems problematic. > > > > > > > > > > That said, agreed that it would be bad for the Java client to have > > the > > > > > worst compatibility story. > > > > > > > > > > -Ewen > > > > > > > > > > > > > > > > > > > > > > Gwen > > > > > > > > > > > > On Thu, Mar 31, 2016 at 2:51 PM, Jason Gustafson < > > ja...@confluent.io > > > > > > > > > > wrote: > > > > > > > > > > > > > Bumping this thread. I talked with Ashish and Magnus about this > > KIP > > > > > > offline > > > > > > > and I'm gradually coming over. The new API actually stands by > > > itself > > > > > > > outside of the discussion about whether the client should > support > > > > > > backwards > > > > > > > compatibility or not. For the Java client, we could continue to > > > > support > > > > > > the > > > > > > > current compatibility approach in which the client supports > only > > > > > brokers > > > > > > > with the same version or greater. In that case, we would use > this > > > API > > > > > > only > > > > > > > to assert that the current API versions are all supported, and > > > raise > > > > an > > > > > > > exception if they are not. This gives us the capability going > > > forward > > > > > to > > > > > > > detect when the client is talking to an older broker, which we > > > don't > > > > > have > > > > > > > right now. This check should be straightforward, so we could do > > it > > > > now, > > > > > > > which would resolve some of the uneasiness about having an > unused > > > > > feature > > > > > > > which we depended on other clients to test for us. Does that > make > > > > sense > > > > > > or > > > > > > > not? > > > > > > > > > > > > > > -Jason > > > > > > > > > > > > > > On Thu, Mar 17, 2016 at 4:06 PM, Ashish Singh < > > asi...@cloudera.com > > > > > > > > > > wrote: > > > > > > > > > > > > > > > We have proposed and discussed majorly three approaches so > far, > > > > there > > > > > > > were > > > > > > > > many minor versions with small variations. Comparing them > > really > > > > > > > requires a > > > > > > > > side by side proposal and their pros/cons, and I agree with > > > others > > > > > that > > > > > > > > this has been lacking in the KIP. We just updated the KIP > with > > > > > > following > > > > > > > > details. > > > > > > > > > > > > > > > > 1. Provide proposed changes in all the three proposals we > have > > > > > > discussed > > > > > > > so > > > > > > > > far. Except the current proposal, these proposals are in > > rejected > > > > > > > > alternatives. > > > > > > > > 2. Provide reasoning on why the rejected proposals were > > rejected. > > > > > > > > 3. Add scenarios for all of these proposals from a client > > > developer > > > > > and > > > > > > > > core Kafka developer point of view. > > > > > > > > > > > > > > > > As we are really close to 0.10 deadline, a quick round of > > voting > > > > will > > > > > > > > really help. If you really do not like the idea, please feel > > free > > > > to > > > > > > say > > > > > > > > so. If the vote fails for the current proposal, it can at > lease > > > > > provide > > > > > > > > recommendations that we should consider for next version of > > > > proposal > > > > > > and > > > > > > > > put it up for vote again for next release. However, as stated > > > > earlier > > > > > > by > > > > > > > > multiple people having this ASAP will be awesome. > > > > > > > > > > > > > > > > On Thu, Mar 17, 2016 at 3:29 PM, Dana Powers < > > > > dana.pow...@gmail.com> > > > > > > > > wrote: > > > > > > > > > > > > > > > > > On Thu, Mar 17, 2016 at 1:42 PM, Gwen Shapira < > > > g...@confluent.io > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > "I think I would make this approach work by looking at > the > > > > > released > > > > > > > > > server > > > > > > > > > > version documentation for each version that I am trying > to > > > > > support > > > > > > > and > > > > > > > > > test > > > > > > > > > > against*, manually identify the expected "protocol > vectors" > > > > each > > > > > > > > > supports, > > > > > > > > > > store that as a map of vectors to "broker versions", > check > > > each > > > > > > > vector > > > > > > > > at > > > > > > > > > > runtime until I find a match, and write code > compatibility > > > > checks > > > > > > > from > > > > > > > > > > there." > > > > > > > > > > > > > > > > > > > > How is this better than a global version ID? > > > > > > > > > > > > > > > > > > > > > > > > > > > As a client developer, it seems roughly the same. I think > it > > > > > probably > > > > > > > > > avoids the server development workflow issues, and possibly > > the > > > > > need > > > > > > to > > > > > > > > > agree on semantics of the global version ID? But others > > surely > > > > are > > > > > > more > > > > > > > > > qualified than me to comment on that part. > > > > > > > > > > > > > > > > > > -Dana > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > Regards, > > > > > > > > Ashish > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > Thanks, > > > > > Ewen > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > Regards, > > > > Ashish > > > > > > > > > > -- Regards, Ashish
