@Magnus, Does the latest suggestion sound OK to you. I am planning to update PR based on latest suggestion.
On Mon, Mar 7, 2016 at 10:58 AM, Ashish Singh <asi...@cloudera.com> wrote: > > > On Fri, Mar 4, 2016 at 5:46 PM, Jay Kreps <j...@confluent.io> wrote: > >> Hey Ashish, >> >> Both good points. >> >> I think the issue with the general metadata request is the same as the >> issue with a version-specific metadata request from the other >> proposal--basically it's a chicken and egg problem, to find out anything >> about the cluster you have to be able to communicate something in a format >> the server can understand without knowing a priori what version it's on. I >> guess the question is how can you continue to evolve the metadata request >> (whether it is the existing metadata or a protocol-version specific >> metadata request) given that you need this information to bootstrap you >> have to be more careful in how that request evolves. >> > You are correct. It's just that protocol version request would be very > specific to retrieve the protocol versions. Changes to protocol version > request itself should be very rare, if at all. However, the general > metadata request carries a lot more information and its format is more > probable to evolve. This boils down to higher probability of change vs a > definite network round-trip for each re/connect. It does sound like, it is > better to avoid a definite penalty than to avoid a probable rare issue. > >> >> I think deprecation/removal may be okay. Ultimately clients will always >> use >> the highest possible version of the protocol the server supports so if >> we've already deprecated and removed your highest version then you are >> screwed and you're going to get an error no matter what, right? Basically >> there is nothing dynamic you can do in that case. >> > Sure, this should be expected. Just wanted to make sure deprecation is > still on the table. > >> >> -Jay >> >> On Fri, Mar 4, 2016 at 4:05 PM, Ashish Singh <asi...@cloudera.com> wrote: >> >> > Hello Jay, >> > >> > The overall approach sounds good. I do realize that this discussion has >> > gotten too lengthy and is starting to shoot tangents. Maybe a KIP call >> will >> > help us getting to a decision faster. I do have a few questions though. >> > >> > On Fri, Mar 4, 2016 at 9:52 AM, Jay Kreps <j...@confluent.io> wrote: >> > >> > > Yeah here is my summary of my take: >> > > >> > > 1. Negotiating a per-connection protocol actually does add a lot of >> > > complexity to clients (many more failure states to get right). >> > > >> > > 2. Having the client configure the protocol version manually is doable >> > now >> > > but probably a worse state. I suspect this will lead to more not less >> > > confusion. >> > > >> > > 3. I don't think the current state is actually that bad. Integrators >> > pick a >> > > conservative version and build against that. There is a tradeoff >> between >> > > getting the new features and being compatible with old Kafka versions. >> > But >> > > a large part of this tradeoff is essential since new features aren't >> > going >> > > to magically appear on old servers, so even if you upgrade your client >> > you >> > > likely aren't going to get the new stuff (since we will end up >> > dynamically >> > > turning it off). Having client features that are there but don't work >> > > because you're on an old cluster may actually be a worse experience if >> > not >> > > handled very carefully.. >> > > >> > > 4. The problems Dana brought up are totally orthogonal to the problem >> of >> > > having per-api versions or overall versions. The problem was that we >> > > changed behavior subtly without changing the version. This will be an >> > issue >> > > regardless of whether the version is global or not. >> > > >> > > 5. Using the broker release as the version is strictly worse than >> using a >> > > global protocol version (0, 1, 2, ...) that increments any time any >> api >> > > changes but doesn't increment just because non-protocol code is >> changed. >> > > The problem with using the broker release version is we want to be >> able >> > to >> > > keep Kafka releasable from any commit which means there isn't as >> clear a >> > > sequencing of releases as you would think. >> > > >> > > 6. We need to consider the case of mixed version clusters during the >> time >> > > period when you are upgrading Kafka. >> > > >> > > So overall I think this is not a critical thing to do right now, but >> if >> > we >> > > are going to do it we should do it in a way that actually improves >> > things. >> > > >> > > Here would be one proposal for that: >> > > a. Add a global protocol version that increments with any api version >> > > update. Move the documentation so that the docs are by version. This >> is >> > > basically just a short-hand for a complete set of supported api >> versions. >> > > b. Include a field in the metadata response for each broker that adds >> the >> > > protocol version. >> > > >> > There might be an issue here where the metadata request version sent by >> > client is not supported by broker, an older broker. However, if we are >> > clearly stating that a client is not guaranteed to work with an older >> > broker then this becomes expected. This will potentially limit us in >> terms >> > of supporting downgrades though, if we ever want to. >> > >> > > c. To maintain the protocol version this information will have to get >> > > propagated with the rest of the broker metadata like host, port, id, >> etc. >> > > >> > > The instructions to clients would be: >> > > - By default you build against a single conservative Kafka protocol >> > version >> > > and we carry that support forward, as today >> > > >> > If I am getting this correct, this will mean we will never >> deprecate/remove >> > any protocol version in future. Having some way to deprecate/remove >> older >> > protocol versions will probably be a good idea. It is possible with the >> > global protocol version approach, it could be as simple as marking a >> > protocol deprecated in protocol doc before removing it. Just want to >> make >> > sure deprecation is still on the table. >> > >> > > - If you want to get fancy you can use the protocol version field in >> the >> > > metadata request to more dynamically chose what features are available >> > and >> > > select api versions appropriately. This is purely optional. >> > > >> > > -Jay >> > > >> > > On Thu, Mar 3, 2016 at 9:38 PM, Jason Gustafson <ja...@confluent.io> >> > > wrote: >> > > >> > > > I talked with Jay about this KIP briefly this morning, so let me >> try to >> > > > summarize the discussion (I'm sure he'll jump in if I get anything >> > > wrong). >> > > > Apologies in advance for the length. >> > > > >> > > > I think we both share some skepticism that a request with all the >> > > supported >> > > > versions of all the request APIs is going to be a useful primitive >> to >> > try >> > > > and build client compatibility around. In practice I think people >> would >> > > end >> > > > up checking for particular request versions in order to determine if >> > the >> > > > broker is 0.8 or 0.9 or whatever, and then change behavior >> accordingly. >> > > I'm >> > > > wondering if there's a reasonable way to handle the version >> responses >> > > that >> > > > doesn't amount to that. Maybe you could try to capture feature >> > > > compatibility by checking the versions for a subset of request >> types? >> > For >> > > > example, to ensure that you can use the new consumer API, you check >> > that >> > > > the group coordinator request is present, the offset commit request >> > > version >> > > > is greater than 2, the offset fetch request is greater than 1, and >> the >> > > join >> > > > group request is present. And to ensure compatibility with KIP-32, >> > maybe >> > > > you only need to check the appropriate versions of the fetch and >> > produce >> > > > requests. That sounds kind of complicated to keep track of and you >> > > probably >> > > > end up trying to handle combinations which aren't even possible in >> > > > practice. >> > > > >> > > > The alternative is to use a single API version. It could be the >> Kafka >> > > > release version, but then you need to figure out how to handle users >> > who >> > > > are running off of trunk since multiple API versions will typically >> > > change >> > > > between releases. Perhaps it makes more sense to keep a separate API >> > > > version number which is incremented every time any one of the API >> > > versions >> > > > increases? This also decouples the protocol from the Kafka >> > distribution. >> > > > >> > > > As far as whether there should be a separate request or not, I get >> > > Becket's >> > > > point that you would only need to do the version check once when a >> > > > connection is established, but another round trip still complicates >> the >> > > > picture quite a bit. Before you just need to send a metadata >> request to >> > > > bootstrap yourself to the cluster, but now you need to do version >> > > > negotiation before you can even do that, and then you need to try >> adapt >> > > to >> > > > the versions reported. Jay brought up the point that you probably >> > > wouldn't >> > > > design a protocol from scratch to work this way. Using the metadata >> > > request >> > > > would be better if it's possible, but you need a way to handle the >> fact >> > > > that a broker's version might be stale by the time you connect to >> it. >> > And >> > > > even then you're going to have to deal internally with the >> complexity >> > > > involved in trying to upgrade/downgrade dynamically, which sounds >> to me >> > > > like it would have a ton of edge cases. >> > > > >> > > > Taking a bit of a step back, any solution is probably going to be >> > painful >> > > > since the Kafka protocol was not designed for this use case. >> Currently >> > > what >> > > > that means for clients that /want/ to support compatibility across >> > broker >> > > > versions is that they need to have the user tell them the broker >> > version >> > > > through configuration (e.g. librdkafka has a "protocol.version" >> field >> > for >> > > > this purpose). The only real problem with this in my mind is that we >> > > don't >> > > > have a graceful way to detect request incompatibility, which is why >> > there >> > > > are so many questions on the user list which basically amount to the >> > > client >> > > > hanging because the broker refuses to respond to a request it >> doesn't >> > > > understand. If you solve this problem, then depending on >> configuration >> > > > seems totally reasonable and we can skip trying to implement request >> > > > version negotiation. Magnus's solution in this KIP may seem a little >> > > hacky, >> > > > but it also seems like the only way to do it without changing the >> > header. >> > > > >> > > > The Spark problem mentioned above is interesting and I agree that it >> > > sucks >> > > > for frameworks that need to ship the kafka client library since they >> > have >> > > > to figure out how to bundle multiple versions. Ultimately if we >> want to >> > > > solve this problem, then it sounds like we need to commit to >> > maintaining >> > > > compatibility with older versions of Kafka in the client going >> forward. >> > > > That's a lot bigger decision and it matters less whether the broker >> > > version >> > > > is found through configuration, topic metadata, or a new request >> type. >> > > > >> > > > -Jason >> > > > >> > > > >> > > > On Thu, Mar 3, 2016 at 3:59 PM, Becket Qin <becket....@gmail.com> >> > wrote: >> > > > >> > > > > Hi Ashish, >> > > > > >> > > > > In approach (1), the clients will still be able to talked to >> multiple >> > > > > versions of Kafka brokers as long as the clients version is not >> > higher >> > > > than >> > > > > the broker version, right? >> > > > > >> > > > > From Spark's point of view, it seems the difference is whether >> Spark >> > > can >> > > > > independently update their Kafka clients dependency or not. More >> > > > > specifically, consider the following three scenarios: >> > > > > A. Spark has some new features that do not rely on clients or >> brokers >> > > in >> > > > a >> > > > > new Kafka release. >> > > > > B. Spark has some new features that only rely on the clients in a >> new >> > > > Kafka >> > > > > release, but not rely on the brokers in a new Kafka release. e.g. >> New >> > > > > client provides a listTopic() method. >> > > > > C. Spark has some new features that rely on both the clients and >> > > brokers >> > > > in >> > > > > a new Kafka release. e.g timestamp field. >> > > > > >> > > > > For A, Spark does not need to update the Kafka dependency because >> > there >> > > > is >> > > > > no need and the old clients can talk to both new and old Kafka >> > brokers. >> > > > > For C, Spark has to wait for broker upgrade anyways. >> > > > > So in the above two scenarios, there is not much difference >> between >> > > > > approach (1) and (2). >> > > > > >> > > > > B is a tricky scenario. Because it is possible that we introduce >> both >> > > > > listTopic() and the timestamp field in the same Kafka release, >> and we >> > > > don't >> > > > > know if Spark needs both or only uses listTopic(). >> > > > > This indicates the client should work fine if a method is >> supported >> > and >> > > > > should throw exception when a method is not supported. I think we >> can >> > > do >> > > > > the following: >> > > > > 0. Clients always use its highest request version. The clients >> keeps >> > a >> > > > > static final map recording the minimum required ApiVersion for >> each >> > > > > request. >> > > > > 1. When connect to a broker, the clients always send an >> > > ApiVersionRequest >> > > > > to the broker. >> > > > > 2. The broker replies with the its highest supported ApiVersion. >> > > > > 3. Before sending a request, the clients checks the minimum >> required >> > > > > ApiVersion for that request. If the broker returned ApiVersion is >> > > higher >> > > > > than this minimum required ApiVersion, then we can proceed. >> Otherwise >> > > we >> > > > > throw something like NotSupportedOperationException. >> > > > > >> > > > > With this approach, scenario B will also work unless Spark calls >> some >> > > > > function that is not supported by the Kafka broker, which makes it >> > > become >> > > > > scenario C. >> > > > > >> > > > > Thoughts? >> > > > > >> > > > > Thanks, >> > > > > >> > > > > Jiangjie (Becket) Qin >> > > > > >> > > > > On Thu, Mar 3, 2016 at 11:24 AM, Ashish Singh < >> asi...@cloudera.com> >> > > > wrote: >> > > > > >> > > > > > On Wed, Mar 2, 2016 at 8:29 PM, Becket Qin < >> becket....@gmail.com> >> > > > wrote: >> > > > > > >> > > > > > > Hi Jason, >> > > > > > > >> > > > > > > I was thinking that every time when we connect to a broker, we >> > > first >> > > > > send >> > > > > > > the version check request. (The version check request itself >> > should >> > > > be >> > > > > > very >> > > > > > > simple and never changes across all server releases.) This >> does >> > add >> > > > an >> > > > > > > additional round trip, but given reconnect is rare, it is >> > probably >> > > > > fine. >> > > > > > On >> > > > > > > the client side, the client will always send request using the >> > > lowest >> > > > > > > supported version across all brokers. That means if a Kafka >> > cluster >> > > > is >> > > > > > > downgrading, we will use the downgraded protocol as soon as >> the >> > > > client >> > > > > > > connected to an older broker. >> > > > > > > >> > > > > > This sounds interesting and very similar to current suggestion. >> > > > However, >> > > > > > just to make sure I am getting it right, you are suggesting >> send a >> > > > > separate >> > > > > > request only for release version? >> > > > > > >> > > > > > > >> > > > > > > @Ashish, >> > > > > > > Can you help me understand the pain points from other open >> source >> > > > > > projects >> > > > > > > that you mentioned a little more? There are two different >> levels >> > of >> > > > > > > requirements: >> > > > > > > 1. User wants to know if the client is compatible with the >> broker >> > > or >> > > > > not. >> > > > > > > 2. User wants the client and the broker to negotiate the >> protocol >> > > on >> > > > > > their >> > > > > > > own. >> > > > > > > >> > > > > > Not sure which category it falls in, but below is the excerpt >> from >> > > > Mark, >> > > > > a >> > > > > > spark dev, who has been trying to upgrade spark kafka >> integration >> > to >> > > > use >> > > > > > 0.9 clients. >> > > > > > >> > > > > > Based on what I understand, users of Kafka need to upgrade their >> > > > brokers >> > > > > to >> > > > > > Kafka 0.9.x first, before they upgrade their clients to Kafka >> > 0.9.x. >> > > > > > >> > > > > > > However, that presents a problem to other projects that >> integrate >> > > > with >> > > > > > Kafka (Spark, Flume, Storm, etc.). From here on, I will speak >> for >> > > > Spark + >> > > > > > Kafka, since that's the one I am most familiar with. >> > > > > > In the light of compatibility (or the lack thereof) between >> 0.8.x >> > and >> > > > > > 0.9.x, Spark is faced with a problem of what version(s) of >> Kafka to >> > > be >> > > > > > compatible with, and has 2 options (discussed in this PR >> > > > > > <https://github.com/apache/spark/pull/11143>): >> > > > > > 1. We either upgrade to Kafka 0.9, dropping support for 0.8. >> Storm >> > > and >> > > > > > Flume are already on this path. >> > > > > > 2. We introduce complexity in our code to support both 0.8 and >> 0.9 >> > > for >> > > > > the >> > > > > > entire duration of our next major release (Apache Spark 2.x). >> > > > > > I'd love to hear your thoughts on which option, you recommend. >> > > > > > Long term, I'd really appreciate if Kafka could do something >> that >> > > > doesn't >> > > > > > make Spark having to support two, or even more versions of >> Kafka. >> > > And, >> > > > if >> > > > > > there is something that I, personally, and Spark project can do >> in >> > > your >> > > > > > next release candidate phase to make things easier, please do >> let >> > us >> > > > > know. >> > > > > > >> > > > > > This issue has made other projects worry about how they are >> going >> > to >> > > > keep >> > > > > > up with Kafka releases. Last I heard, take this with a pinch of >> > salt, >> > > > > Spark >> > > > > > folks are discussing about using Maven profiles to build against >> > > > multiple >> > > > > > Kafka versions at compile time, etc. Also, there are clients who >> > are >> > > > > > relying on class-loading tricks with custom implementation of >> OSGi >> > to >> > > > > solve >> > > > > > such issues. Don't quote me on the stuff I just mentioned, as >> this >> > is >> > > > > what >> > > > > > I have heard during casual discussions. The point I am trying to >> > make >> > > > is >> > > > > > that Kafka clients are worried about being able to support >> multiple >> > > > Kafka >> > > > > > broker versions. I am sure we all agree on that. >> > > > > > >> > > > > > I think the second requirement makes more sense from a client >> > > > > perspective. >> > > > > > First req will just tell them that there is a problem, but no >> way >> > to >> > > > work >> > > > > > around it. >> > > > > > >> > > > > > > >> > > > > > > Currently in Kafka the principle we are following is to let >> > clients >> > > > > stick >> > > > > > > to a certain version and server will adapt to the clients >> > > > accordingly. >> > > > > > > If this KIP doesn't want to break this rule, it seems we >> should >> > > > simply >> > > > > > let >> > > > > > > the clients send the ApiVersion it is using to the brokers and >> > the >> > > > > > brokers >> > > > > > > will decide whether to accept or reject the clients. This >> means >> > > user >> > > > > have >> > > > > > > to upgrade broker before they upgrade clients. This satisfies >> (1) >> > > so >> > > > > > that a >> > > > > > > newer client will know it does not compatible with an older >> > server >> > > > > > > immediately. >> > > > > > > If this KIP will change that to let the newer clients adapt to >> > the >> > > > > older >> > > > > > > brokers, compatibility wise it is a good thing to have. With >> > this >> > > > now >> > > > > > > users are able to upgrade clients before they upgrade Kafka >> > > brokers. >> > > > > This >> > > > > > > means user can upgrade clients even before upgrade servers. >> This >> > > > > > satisfies >> > > > > > > (2) as the newer clients can also talk to the older servers. >> > > > > > > >> > > > > > More importantly, this will allow a client to talk to multiple >> > > versions >> > > > > of >> > > > > > Kafka. >> > > > > >> > > > > >> > > > > > > >> > > > > > > If we decide to go with (2). The benefit is that a newer >> client >> > > won't >> > > > > > break >> > > > > > > when talking to an older broker. But functionality wise, it >> might >> > > be >> > > > > the >> > > > > > > same as an older clients. >> > > > > > > In the downgrading case, we probably still have to notify all >> the >> > > > > users. >> > > > > > > For example, if application is sending messages with timestamp >> > and >> > > > the >> > > > > > > broker got downgraded to an older version that does not >> support >> > > > > > timestamp. >> > > > > > > The clients will suddenly start to throw away timestamps. This >> > > might >> > > > > > affect >> > > > > > > the application logic. In this case even if we have clients >> > > > > automatically >> > > > > > > adapted to a lower version broker, the applications might >> still >> > > > break. >> > > > > > > Hence we still need to notify the users about the case when >> the >> > > > clients >> > > > > > is >> > > > > > > newer than the brokers. This is the same for both (1) and (2). >> > > > > > > Supporting (2) will introduce more complication on the client >> > side. >> > > > And >> > > > > > we >> > > > > > > may also have to communicate with users about what function is >> > > > > supported >> > > > > > in >> > > > > > > the new clients and what is not supported after the protocol >> > > > > negotiation >> > > > > > > finishes. >> > > > > > > >> > > > > > Totally agreed, however only if clients want to support multiple >> > > broker >> > > > > > versions. If they want to, then I am sure they are willing to >> add >> > > some >> > > > > > logic on their end. >> > > > > > >> > > > > > > >> > > > > > > Thanks, >> > > > > > > >> > > > > > > Jiangjie (Becket) Qin >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > On Wed, Mar 2, 2016 at 5:58 PM, Dana Powers < >> > dana.pow...@gmail.com >> > > > >> > > > > > wrote: >> > > > > > > >> > > > > > > > In kafka-python we've been doing something like: >> > > > > > > > >> > > > > > > > if version >= (0, 9): >> > > > > > > > Do cool new stuff >> > > > > > > > elif version >= (0, 8, 2): >> > > > > > > > Do some older stuff >> > > > > > > > .... >> > > > > > > > else: >> > > > > > > > raise UnsupportedVersionError >> > > > > > > > >> > > > > > > > This will break if / when the new 0.9 apis are completely >> > removed >> > > > > from >> > > > > > > some >> > > > > > > > future release, but should handle intermediate broker >> upgrades. >> > > > > Because >> > > > > > > we >> > > > > > > > can't add support for future apis a priori, I think the >> best we >> > > > could >> > > > > > do >> > > > > > > > here is throw an error that request protocol version X is >> not >> > > > > > supported. >> > > > > > > > For now that comes through as a broken socket connection, so >> > > there >> > > > is >> > > > > > an >> > > > > > > > error - just not a super helpful one. >> > > > > > > > >> > > > > > > > For that reason I'm also in favor of a generic error >> response >> > > when >> > > > a >> > > > > > > > protocol req is not recognized. >> > > > > > > > >> > > > > > > > -Dana >> > > > > > > > On Mar 2, 2016 5:38 PM, "Jay Kreps" <j...@confluent.io> >> wrote: >> > > > > > > > >> > > > > > > > > But won't it be the case that what clients end up doing >> would >> > > be >> > > > > > > > something >> > > > > > > > > like >> > > > > > > > > if(version != 0.8.1) >> > > > > > > > > throw new UnsupportedVersionException() >> > > > > > > > > which then means the client is broken as soon as we >> release a >> > > new >> > > > > > > server >> > > > > > > > > version even though the protocol didn't change. I'm >> actually >> > > not >> > > > > sure >> > > > > > > how >> > > > > > > > > you could use that information in a forward compatible way >> > > since >> > > > > you >> > > > > > > > can't >> > > > > > > > > know a priori if you will work with the next release until >> > you >> > > > know >> > > > > > if >> > > > > > > > the >> > > > > > > > > protocol changed. >> > > > > > > > > >> > > > > > > > > -Jay >> > > > > > > > > >> > > > > > > > > On Wed, Mar 2, 2016 at 5:28 PM, Jason Gustafson < >> > > > > ja...@confluent.io> >> > > > > > > > > wrote: >> > > > > > > > > >> > > > > > > > > > Hey Jay, >> > > > > > > > > > >> > > > > > > > > > Yeah, I wasn't suggesting that we eliminate request API >> > > > versions. >> > > > > > > > They're >> > > > > > > > > > definitely needed on the broker to support >> compatibility. I >> > > was >> > > > > > just >> > > > > > > > > saying >> > > > > > > > > > that if a client wants to support multiple broker >> versions >> > > > (e.g. >> > > > > > 0.8 >> > > > > > > > and >> > > > > > > > > > 0.9), then it makes more sense to me to make the kafka >> > > release >> > > > > > > version >> > > > > > > > > > available in order to determine which version of the >> > request >> > > > API >> > > > > > > should >> > > > > > > > > be >> > > > > > > > > > used rather than adding a new request type which exposes >> > all >> > > of >> > > > > the >> > > > > > > > > > different supported versions for all of the request >> types. >> > > > > Request >> > > > > > > API >> > > > > > > > > > versions all change in lockstep with Kafka releases >> anyway. >> > > > > > > > > > >> > > > > > > > > > -Jason >> > > > > > > > > > >> > > > > > > > > > On Wed, Mar 2, 2016 at 5:15 PM, Becket Qin < >> > > > becket....@gmail.com >> > > > > > >> > > > > > > > wrote: >> > > > > > > > > > >> > > > > > > > > > > I think using Kafka release version makes sense. More >> > > > > > particularly, >> > > > > > > > we >> > > > > > > > > > can >> > > > > > > > > > > use the ApiVersion and this will cover all the >> interval >> > > > version >> > > > > > as >> > > > > > > > > well. >> > > > > > > > > > In >> > > > > > > > > > > KAFKA-3025, we added the ApiVersion to message format >> > > version >> > > > > > > > mapping, >> > > > > > > > > We >> > > > > > > > > > > can add the ApiKey to version mapping to ApiVersion as >> > > well. >> > > > We >> > > > > > can >> > > > > > > > > move >> > > > > > > > > > > ApiVersion class to o.a.k.c package and use it for >> both >> > > > server >> > > > > > and >> > > > > > > > > > clients. >> > > > > > > > > > > >> > > > > > > > > > > @Jason, if we cache the release info in metadata and >> not >> > > > > > > re-validate >> > > > > > > > > the >> > > > > > > > > > > release on reconnect, would it still work if we do a >> > > rolling >> > > > > > > > downgrade? >> > > > > > > > > > > >> > > > > > > > > > > On Wed, Mar 2, 2016 at 3:16 PM, Jason Gustafson < >> > > > > > > ja...@confluent.io> >> > > > > > > > > > > wrote: >> > > > > > > > > > > >> > > > > > > > > > > > I think Dana's suggestion to include the Kafka >> release >> > > > > version >> > > > > > > > makes >> > > > > > > > > a >> > > > > > > > > > > lot >> > > > > > > > > > > > of sense. I'm actually wondering why you would need >> the >> > > > > > > individual >> > > > > > > > > API >> > > > > > > > > > > > versions if you have that? It sounds like keeping >> track >> > > of >> > > > > all >> > > > > > > the >> > > > > > > > > api >> > > > > > > > > > > > version information would add a lot of complexity to >> > > > clients >> > > > > > > since >> > > > > > > > > > > they'll >> > > > > > > > > > > > have to try to handle different version permutations >> > > which >> > > > > are >> > > > > > > not >> > > > > > > > > > > actually >> > > > > > > > > > > > possible in practice. Wouldn't it be simpler to know >> > that >> > > > > > you're >> > > > > > > > > > talking >> > > > > > > > > > > to >> > > > > > > > > > > > an 0.9 broker than that you're talking to a broker >> > which >> > > > > > supports >> > > > > > > > > > > version 2 >> > > > > > > > > > > > of the group coordinator request, version 1 of fetch >> > > > request, >> > > > > > > etc? >> > > > > > > > > > Also, >> > > > > > > > > > > > the release version could be included in the broker >> > > > > information >> > > > > > > in >> > > > > > > > > the >> > > > > > > > > > > > topic metadata request which would save the need for >> > the >> > > > > > > additional >> > > > > > > > > > round >> > > > > > > > > > > > trip on every reconnect. >> > > > > > > > > > > > >> > > > > > > > > > > > -Jason >> > > > > > > > > > > > >> > > > > > > > > > > > On Tue, Mar 1, 2016 at 7:59 PM, Ashish Singh < >> > > > > > > asi...@cloudera.com> >> > > > > > > > > > > wrote: >> > > > > > > > > > > > >> > > > > > > > > > > > > On Tue, Mar 1, 2016 at 6:30 PM, Gwen Shapira < >> > > > > > > g...@confluent.io> >> > > > > > > > > > > wrote: >> > > > > > > > > > > > > >> > > > > > > > > > > > > > One more thing, the KIP actually had 3 parts: >> > > > > > > > > > > > > > 1. The version protocol >> > > > > > > > > > > > > > 2. New response on messages of wrong API key or >> > wrong >> > > > > > version >> > > > > > > > > > > > > > 3. Protocol documentation >> > > > > > > > > > > > > > >> > > > > > > > > > > > > There is a WIP patch for adding protocol docs, >> > > > > > > > > > > > > https://github.com/apache/kafka/pull/970 . By >> > protocol >> > > > > > > > > > documentation, >> > > > > > > > > > > > you >> > > > > > > > > > > > > mean updating this, right? >> > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > I understand that you are offering to only >> > implement >> > > > part >> > > > > > 1? >> > > > > > > > > > > > > > But the KIP discussion and vote should still >> cover >> > > all >> > > > > > three >> > > > > > > > > parts, >> > > > > > > > > > > > > > they will just be implemented in separate JIRA? >> > > > > > > > > > > > > > >> > > > > > > > > > > > > The patch for KAFKA-3307, >> > > > > > > > https://github.com/apache/kafka/pull/986 >> > > > > > > > > , >> > > > > > > > > > > > covers >> > > > > > > > > > > > > 1 and 2. KAFKA-3309 tracks documentation part. >> Yes, >> > we >> > > > > should >> > > > > > > > > include >> > > > > > > > > > > all >> > > > > > > > > > > > > the three points you mentioned while discussing or >> > > voting >> > > > > for >> > > > > > > > > KIP-35. >> > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > On Tue, Mar 1, 2016 at 5:25 PM, Ashish Singh < >> > > > > > > > > asi...@cloudera.com> >> > > > > > > > > > > > > wrote: >> > > > > > > > > > > > > > > On Tue, Mar 1, 2016 at 5:11 PM, Gwen Shapira < >> > > > > > > > > g...@confluent.io> >> > > > > > > > > > > > > wrote: >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> I don't see a use for the name - clients >> should >> > be >> > > > > able >> > > > > > to >> > > > > > > > > > > translate >> > > > > > > > > > > > > > >> ApiKey to name for any API they support, and >> I'm >> > > not >> > > > > > sure >> > > > > > > > why >> > > > > > > > > > > would >> > > > > > > > > > > > a >> > > > > > > > > > > > > > >> client need to log anything about APIs it >> does >> > not >> > > > > > > support. >> > > > > > > > > Am I >> > > > > > > > > > > > > > >> missing something? >> > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > Yea, it is a fair assumption that client would >> > know >> > > > > about >> > > > > > > > APIs >> > > > > > > > > it >> > > > > > > > > > > > > > supports. >> > > > > > > > > > > > > > > It could have been helpful for client users to >> > see >> > > > new >> > > > > > APIs >> > > > > > > > > > though, >> > > > > > > > > > > > > > however >> > > > > > > > > > > > > > > users can always refer to protocol doc of new >> > > version >> > > > > to >> > > > > > > find >> > > > > > > > > > > > > > corresponding >> > > > > > > > > > > > > > > names of the new APIs. >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > >> On a related note, Magnus is currently on >> > > vacation, >> > > > > but >> > > > > > he >> > > > > > > > > > should >> > > > > > > > > > > be >> > > > > > > > > > > > > > >> back at the end of next week. I'd like to >> hold >> > off >> > > > on >> > > > > > the >> > > > > > > > vote >> > > > > > > > > > > until >> > > > > > > > > > > > > > >> he gets back since his experience in >> > implementing >> > > > > > clients >> > > > > > > > and >> > > > > > > > > > his >> > > > > > > > > > > > > > >> opinions will be very valuable for this >> > > discussion. >> > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > That is great. It will be valuable to have his >> > > > > feedback. >> > > > > > I >> > > > > > > > will >> > > > > > > > > > > hold >> > > > > > > > > > > > > off >> > > > > > > > > > > > > > on >> > > > > > > > > > > > > > > removing "api_name" and >> "api_deprecated_versions" >> > > or >> > > > > > adding >> > > > > > > > > > release >> > > > > > > > > > > > > > version. >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > >> Gwen >> > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > >> On Tue, Mar 1, 2016 at 4:24 PM, Ashish Singh >> < >> > > > > > > > > > asi...@cloudera.com >> > > > > > > > > > > > >> > > > > > > > > > > > > > wrote: >> > > > > > > > > > > > > > >> > Works with me. I will update PR to remove >> > this. >> > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > >> > Also, "api_name" have been pointed out as a >> > > > concern. >> > > > > > > > > However, >> > > > > > > > > > it >> > > > > > > > > > > > can >> > > > > > > > > > > > > > be >> > > > > > > > > > > > > > >> > handy for logging and similar purposes. Any >> > take >> > > > on >> > > > > > > that? >> > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > >> > On Tue, Mar 1, 2016 at 3:46 PM, Gwen >> Shapira < >> > > > > > > > > > g...@confluent.io >> > > > > > > > > > > > >> > > > > > > > > > > > > > wrote: >> > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > >> >> Jay also mentioned: >> > > > > > > > > > > > > > >> >> "Or, alternately, since deprecation has no >> > > > > functional >> > > > > > > > > impact >> > > > > > > > > > > and >> > > > > > > > > > > > is >> > > > > > > > > > > > > > >> >> just a message >> > > > > > > > > > > > > > >> >> to developers, we could just leave it out >> of >> > > the >> > > > > > > protocol >> > > > > > > > > and >> > > > > > > > > > > > just >> > > > > > > > > > > > > > have >> > > > > > > > > > > > > > >> it >> > > > > > > > > > > > > > >> >> in release notes etc." >> > > > > > > > > > > > > > >> >> >> > > > > > > > > > > > > > >> >> I'm in favor of leaving it out of the >> > > protocol. I >> > > > > > can't >> > > > > > > > > > really >> > > > > > > > > > > > see >> > > > > > > > > > > > > a >> > > > > > > > > > > > > > >> >> use-case. >> > > > > > > > > > > > > > >> >> >> > > > > > > > > > > > > > >> >> Gwen >> > > > > > > > > > > > > > >> >> >> > > > > > > > > > > > > > >> >> On Mon, Feb 29, 2016 at 3:55 PM, Ashish >> > Singh < >> > > > > > > > > > > > asi...@cloudera.com >> > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> wrote: >> > > > > > > > > > > > > > >> >> >> > > > > > > > > > > > > > >> >> > I hope it is OK for me to make some >> > progress >> > > > > here. >> > > > > > I >> > > > > > > > have >> > > > > > > > > > > made >> > > > > > > > > > > > > the >> > > > > > > > > > > > > > >> >> > following changes. >> > > > > > > > > > > > > > >> >> > >> > > > > > > > > > > > > > >> >> > 1. Updated KIP-35, to adopt Jay's >> > suggestion >> > > on >> > > > > > > > > maintaining >> > > > > > > > > > > > > > separate >> > > > > > > > > > > > > > >> list >> > > > > > > > > > > > > > >> >> > of deprecated versions, instead of >> using a >> > > > > version >> > > > > > of >> > > > > > > > -1. >> > > > > > > > > > > > > > >> >> > 2. Added information on required >> > permissions, >> > > > > > > Describe >> > > > > > > > > > action >> > > > > > > > > > > > on >> > > > > > > > > > > > > > >> Cluster >> > > > > > > > > > > > > > >> >> > resource, to be able to retrieve >> protocol >> > > > > versions >> > > > > > > > from a >> > > > > > > > > > > auth >> > > > > > > > > > > > > > enabled >> > > > > > > > > > > > > > >> >> > Kafka cluster. >> > > > > > > > > > > > > > >> >> > >> > > > > > > > > > > > > > >> >> > Created >> > > > > > > > https://issues.apache.org/jira/browse/KAFKA-3304 >> > > > > > > > > . >> > > > > > > > > > > > > Primary >> > > > > > > > > > > > > > >> patch >> > > > > > > > > > > > > > >> >> is >> > > > > > > > > > > > > > >> >> > available to review, >> > > > > > > > > > > https://github.com/apache/kafka/pull/986 >> > > > > > > > > > > > > > >> >> > >> > > > > > > > > > > > > > >> >> > On Thu, Feb 25, 2016 at 1:27 PM, Ashish >> > > Singh < >> > > > > > > > > > > > > asi...@cloudera.com >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> >> wrote: >> > > > > > > > > > > > > > >> >> > >> > > > > > > > > > > > > > >> >> > > Kafka clients in Hadoop ecosystem, >> Flume, >> > > > > Spark, >> > > > > > > etc, >> > > > > > > > > > have >> > > > > > > > > > > > > found >> > > > > > > > > > > > > > it >> > > > > > > > > > > > > > >> >> > really >> > > > > > > > > > > > > > >> >> > > difficult to cope up with Kafka >> releases >> > as >> > > > > they >> > > > > > > want >> > > > > > > > > to >> > > > > > > > > > > > > support >> > > > > > > > > > > > > > >> users >> > > > > > > > > > > > > > >> >> on >> > > > > > > > > > > > > > >> >> > > different Kafka versions. Capability >> to >> > > > > retrieve >> > > > > > > > > protocol >> > > > > > > > > > > > > version >> > > > > > > > > > > > > > >> will >> > > > > > > > > > > > > > >> >> > go a >> > > > > > > > > > > > > > >> >> > > long way to ease out those pain >> points. I >> > > > will >> > > > > be >> > > > > > > > happy >> > > > > > > > > > to >> > > > > > > > > > > > help >> > > > > > > > > > > > > > out >> > > > > > > > > > > > > > >> >> with >> > > > > > > > > > > > > > >> >> > > the work on this KIP. @Magnus, thanks >> for >> > > > > driving >> > > > > > > > this, >> > > > > > > > > > is >> > > > > > > > > > > it >> > > > > > > > > > > > > OK >> > > > > > > > > > > > > > if >> > > > > > > > > > > > > > >> I >> > > > > > > > > > > > > > >> >> > carry >> > > > > > > > > > > > > > >> >> > > forward the work from here. It will be >> > > ideal >> > > > to >> > > > > > > have >> > > > > > > > > this >> > > > > > > > > > > in >> > > > > > > > > > > > > > >> 0.10.0.0. >> > > > > > > > > > > > > > >> >> > > >> > > > > > > > > > > > > > >> >> > > On Mon, Oct 12, 2015 at 9:29 PM, Jay >> > Kreps >> > > < >> > > > > > > > > > > j...@confluent.io >> > > > > > > > > > > > > >> > > > > > > > > > > > > > >> wrote: >> > > > > > > > > > > > > > >> >> > > >> > > > > > > > > > > > > > >> >> > >> I wonder if we need to solve the >> error >> > > > > problem? >> > > > > > I >> > > > > > > > > think >> > > > > > > > > > > this >> > > > > > > > > > > > > KIP >> > > > > > > > > > > > > > >> >> gives a >> > > > > > > > > > > > > > >> >> > >> descent work around. >> > > > > > > > > > > > > > >> >> > >> >> > > > > > > > > > > > > > >> >> > >> Probably we should have included an >> > error >> > > in >> > > > > the >> > > > > > > > > > response >> > > > > > > > > > > > > > header, >> > > > > > > > > > > > > > >> but >> > > > > > > > > > > > > > >> >> we >> > > > > > > > > > > > > > >> >> > >> debated it at the time decided not to >> > and >> > > > now >> > > > > it >> > > > > > > is >> > > > > > > > > > pretty >> > > > > > > > > > > > > hard >> > > > > > > > > > > > > > to >> > > > > > > > > > > > > > >> add >> > > > > > > > > > > > > > >> >> > >> because the headers aren't versioned >> > > (d'oh). >> > > > > > > > > > > > > > >> >> > >> >> > > > > > > > > > > > > > >> >> > >> It seems like any other solution is >> > going >> > > to >> > > > > be >> > > > > > > kind >> > > > > > > > > of >> > > > > > > > > > a >> > > > > > > > > > > > > hack, >> > > > > > > > > > > > > > >> right? >> > > > > > > > > > > > > > >> >> > >> Sending malformed responses back >> seems >> > > like >> > > > > not >> > > > > > a >> > > > > > > > > clean >> > > > > > > > > > > > > > solution... >> > > > > > > > > > > > > > >> >> > >> >> > > > > > > > > > > > > > >> >> > >> (Not sure if I was pro- having a >> > top-level >> > > > > error >> > > > > > > or >> > > > > > > > > not, >> > > > > > > > > > > but >> > > > > > > > > > > > > in >> > > > > > > > > > > > > > any >> > > > > > > > > > > > > > >> >> case >> > > > > > > > > > > > > > >> >> > >> the rationale for the decision was >> that >> > so >> > > > > many >> > > > > > of >> > > > > > > > the >> > > > > > > > > > > > > requests >> > > > > > > > > > > > > > >> were >> > > > > > > > > > > > > > >> >> > >> per-partition or per-topic or >> whatever >> > and >> > > > > hence >> > > > > > > > fail >> > > > > > > > > or >> > > > > > > > > > > > > > succeed at >> > > > > > > > > > > > > > >> >> that >> > > > > > > > > > > > > > >> >> > >> level and this makes it hard to know >> > what >> > > > the >> > > > > > > right >> > > > > > > > > > > > top-level >> > > > > > > > > > > > > > error >> > > > > > > > > > > > > > >> >> code >> > > > > > > > > > > > > > >> >> > >> is >> > > > > > > > > > > > > > >> >> > >> and hard for the client to figure out >> > what >> > > > to >> > > > > do >> > > > > > > > with >> > > > > > > > > > the >> > > > > > > > > > > > top >> > > > > > > > > > > > > > level >> > > > > > > > > > > > > > >> >> > error >> > > > > > > > > > > > > > >> >> > >> if some of the partitions succeed but >> > > there >> > > > > is a >> > > > > > > > > > top-level >> > > > > > > > > > > > > > error). >> > > > > > > > > > > > > > >> >> > >> >> > > > > > > > > > > > > > >> >> > >> I think actually this new API >> actually >> > > > gives a >> > > > > > way >> > > > > > > > to >> > > > > > > > > > > handle >> > > > > > > > > > > > > > this >> > > > > > > > > > > > > > >> >> > >> gracefully on the client side by just >> > > having >> > > > > > > clients >> > > > > > > > > > that >> > > > > > > > > > > > want >> > > > > > > > > > > > > > to >> > > > > > > > > > > > > > >> be >> > > > > > > > > > > > > > >> >> > >> graceful check for support for their >> > > > version. >> > > > > > > > Clients >> > > > > > > > > > that >> > > > > > > > > > > > do >> > > > > > > > > > > > > > that >> > > > > > > > > > > > > > >> >> will >> > > > > > > > > > > > > > >> >> > >> have a graceful message. >> > > > > > > > > > > > > > >> >> > >> >> > > > > > > > > > > > > > >> >> > >> At some point if we're ever reworking >> > the >> > > > > > headers >> > > > > > > we >> > > > > > > > > > > should >> > > > > > > > > > > > > > really >> > > > > > > > > > > > > > >> >> > >> consider >> > > > > > > > > > > > > > >> >> > >> (a) versioning them and (b) adding a >> > > > top-level >> > > > > > > error >> > > > > > > > > > code >> > > > > > > > > > > in >> > > > > > > > > > > > > the >> > > > > > > > > > > > > > >> >> > response. >> > > > > > > > > > > > > > >> >> > >> But given this would be a big >> breaking >> > > > change >> > > > > > and >> > > > > > > > this >> > > > > > > > > > is >> > > > > > > > > > > > > really >> > > > > > > > > > > > > > >> just >> > > > > > > > > > > > > > >> >> to >> > > > > > > > > > > > > > >> >> > >> give a nicer error message seems >> like it >> > > > > > probably >> > > > > > > > > isn't >> > > > > > > > > > > > worth >> > > > > > > > > > > > > > it to >> > > > > > > > > > > > > > >> >> try >> > > > > > > > > > > > > > >> >> > to >> > > > > > > > > > > > > > >> >> > >> do something now. >> > > > > > > > > > > > > > >> >> > >> >> > > > > > > > > > > > > > >> >> > >> -Jay >> > > > > > > > > > > > > > >> >> > >> >> > > > > > > > > > > > > > >> >> > >> >> > > > > > > > > > > > > > >> >> > >> >> > > > > > > > > > > > > > >> >> > >> On Mon, Oct 12, 2015 at 8:11 PM, >> > Jiangjie >> > > > Qin >> > > > > > > > > > > > > > >> >> <j...@linkedin.com.invalid >> > > > > > > > > > > > > > >> >> > > >> > > > > > > > > > > > > > >> >> > >> wrote: >> > > > > > > > > > > > > > >> >> > >> >> > > > > > > > > > > > > > >> >> > >> > I am thinking instead of returning >> an >> > > > empty >> > > > > > > > > response, >> > > > > > > > > > it >> > > > > > > > > > > > > > would be >> > > > > > > > > > > > > > >> >> > >> better to >> > > > > > > > > > > > > > >> >> > >> > return an explicit >> > > > > UnsupportedVersionException >> > > > > > > > code. >> > > > > > > > > > > > > > >> >> > >> > >> > > > > > > > > > > > > > >> >> > >> > Today KafkaApis handles the error >> in >> > the >> > > > > > > following >> > > > > > > > > > way: >> > > > > > > > > > > > > > >> >> > >> > 1. For requests/responses using old >> > > Scala >> > > > > > > classes, >> > > > > > > > > > > > KafkaApis >> > > > > > > > > > > > > > uses >> > > > > > > > > > > > > > >> >> > >> > RequestOrResponse.handleError() to >> > > return >> > > > an >> > > > > > > error >> > > > > > > > > > > > response. >> > > > > > > > > > > > > > >> >> > >> > 2. For requests/response using Java >> > > > classes >> > > > > > > (only >> > > > > > > > > > > > > > >> JoinGroupRequest >> > > > > > > > > > > > > > >> >> and >> > > > > > > > > > > > > > >> >> > >> > Heartbeat now), KafkaApis calls >> > > > > > > > > > > > > > >> AbstractRequest.getErrorResponse() >> > > > > > > > > > > > > > >> >> to >> > > > > > > > > > > > > > >> >> > >> > return an error response. >> > > > > > > > > > > > > > >> >> > >> > >> > > > > > > > > > > > > > >> >> > >> > In KAFKA-2512, I am returning an >> > > > > > > > > > > > UnsupportedVersionException >> > > > > > > > > > > > > > for >> > > > > > > > > > > > > > >> >> case >> > > > > > > > > > > > > > >> >> > >> [1] >> > > > > > > > > > > > > > >> >> > >> > when see an unsupported version. >> This >> > > will >> > > > > put >> > > > > > > the >> > > > > > > > > > error >> > > > > > > > > > > > > code >> > > > > > > > > > > > > > per >> > > > > > > > > > > > > > >> >> > topic >> > > > > > > > > > > > > > >> >> > >> or >> > > > > > > > > > > > > > >> >> > >> > partition for most of the requests, >> > but >> > > > > might >> > > > > > > not >> > > > > > > > > work >> > > > > > > > > > > all >> > > > > > > > > > > > > the >> > > > > > > > > > > > > > >> time. >> > > > > > > > > > > > > > >> >> > >> e.g. >> > > > > > > > > > > > > > >> >> > >> > TopicMetadataRequest with an empty >> > topic >> > > > > set. >> > > > > > > > > > > > > > >> >> > >> > >> > > > > > > > > > > > > > >> >> > >> > Case [2] does not quite work for >> > > > unsupported >> > > > > > > > > version, >> > > > > > > > > > > > > because >> > > > > > > > > > > > > > we >> > > > > > > > > > > > > > >> >> will >> > > > > > > > > > > > > > >> >> > >> > thrown an uncaught exception when >> > > version >> > > > is >> > > > > > not >> > > > > > > > > > > > recognized >> > > > > > > > > > > > > > (BTW >> > > > > > > > > > > > > > >> >> this >> > > > > > > > > > > > > > >> >> > >> is a >> > > > > > > > > > > > > > >> >> > >> > bug). Part of the reason is that >> for >> > > some >> > > > > > > response >> > > > > > > > > > > types, >> > > > > > > > > > > > > > error >> > > > > > > > > > > > > > >> code >> > > > > > > > > > > > > > >> >> > is >> > > > > > > > > > > > > > >> >> > >> not >> > > > > > > > > > > > > > >> >> > >> > part of the response level field. >> > > > > > > > > > > > > > >> >> > >> > >> > > > > > > > > > > > > > >> >> > >> > Maybe it worth checking how each >> > > response >> > > > is >> > > > > > > > dealing >> > > > > > > > > > > with >> > > > > > > > > > > > > > error >> > > > > > > > > > > > > > >> code >> > > > > > > > > > > > > > >> >> > >> today. >> > > > > > > > > > > > > > >> >> > >> > A scan of the response formats >> gives >> > the >> > > > > > > following >> > > > > > > > > > > result: >> > > > > > > > > > > > > > >> >> > >> > 1. TopicMetadataResponse - per >> topic >> > > error >> > > > > > code, >> > > > > > > > > does >> > > > > > > > > > > not >> > > > > > > > > > > > > work >> > > > > > > > > > > > > > >> when >> > > > > > > > > > > > > > >> >> > the >> > > > > > > > > > > > > > >> >> > >> > topic set is empty in the request. >> > > > > > > > > > > > > > >> >> > >> > 2. ProduceResonse - per partition >> > error >> > > > > code. >> > > > > > > > > > > > > > >> >> > >> > 3. OffsetCommitResponse - per >> > partition. >> > > > > > > > > > > > > > >> >> > >> > 4. OffsetFetchResponse - per >> > partition. >> > > > > > > > > > > > > > >> >> > >> > 5. OffsetResponse - per partition. >> > > > > > > > > > > > > > >> >> > >> > 6. FetchResponse - per partition >> > > > > > > > > > > > > > >> >> > >> > 7. ConsumerMetadataResponse - >> response >> > > > level >> > > > > > > > > > > > > > >> >> > >> > 8. ControlledShutdownResponse - >> > response >> > > > > level >> > > > > > > > > > > > > > >> >> > >> > 9. JoinGroupResponse - response >> level >> > > > > > > > > > > > > > >> >> > >> > 10. HearbeatResponse - response >> level >> > > > > > > > > > > > > > >> >> > >> > 11. LeaderAndIsrResponse - response >> > > level >> > > > > > > > > > > > > > >> >> > >> > 12. StopReplicaResponse - response >> > level >> > > > > > > > > > > > > > >> >> > >> > 13. UpdateMetadataResponse - >> response >> > > > level >> > > > > > > > > > > > > > >> >> > >> > >> > > > > > > > > > > > > > >> >> > >> > So from the list above it looks for >> > each >> > > > > > > response >> > > > > > > > we >> > > > > > > > > > are >> > > > > > > > > > > > > > actually >> > > > > > > > > > > > > > >> >> able >> > > > > > > > > > > > > > >> >> > >> to >> > > > > > > > > > > > > > >> >> > >> > return an error code, as long as we >> > make >> > > > > sure >> > > > > > > the >> > > > > > > > > > topic >> > > > > > > > > > > or >> > > > > > > > > > > > > > >> partition >> > > > > > > > > > > > > > >> >> > >> won't >> > > > > > > > > > > > > > >> >> > >> > be empty when the error code is at >> > topic >> > > > or >> > > > > > > > > partition >> > > > > > > > > > > > level. >> > > > > > > > > > > > > > >> Luckily >> > > > > > > > > > > > > > >> >> > in >> > > > > > > > > > > > > > >> >> > >> the >> > > > > > > > > > > > > > >> >> > >> > above list we only need to worry >> about >> > > > > > > > > > > > > TopicMetadataResponse. >> > > > > > > > > > > > > > >> >> > >> > >> > > > > > > > > > > > > > >> >> > >> > Maybe error handling is out of the >> > scope >> > > > of >> > > > > > this >> > > > > > > > > KIP, >> > > > > > > > > > > but >> > > > > > > > > > > > I >> > > > > > > > > > > > > > >> prefer >> > > > > > > > > > > > > > >> >> we >> > > > > > > > > > > > > > >> >> > >> think >> > > > > > > > > > > > > > >> >> > >> > through how to deal with error code >> > for >> > > > the >> > > > > > > > > requests, >> > > > > > > > > > > > > because >> > > > > > > > > > > > > > >> there >> > > > > > > > > > > > > > >> >> > are >> > > > > > > > > > > > > > >> >> > >> > more request types to be added in >> > > > KAFKA-2464 >> > > > > > and >> > > > > > > > > > future >> > > > > > > > > > > > > > patches. >> > > > > > > > > > > > > > >> >> > >> > >> > > > > > > > > > > > > > >> >> > >> > Thanks, >> > > > > > > > > > > > > > >> >> > >> > >> > > > > > > > > > > > > > >> >> > >> > Jiangjie (Becket) Qin >> > > > > > > > > > > > > > >> >> > >> > >> > > > > > > > > > > > > > >> >> > >> > On Mon, Oct 12, 2015 at 6:04 PM, >> Jay >> > > > Kreps < >> > > > > > > > > > > > > j...@confluent.io> >> > > > > > > > > > > > > > >> >> wrote: >> > > > > > > > > > > > > > >> >> > >> > >> > > > > > > > > > > > > > >> >> > >> > > Two quick pieces of feedback: >> > > > > > > > > > > > > > >> >> > >> > > 1. The use of a version of -1 as >> > > magical >> > > > > > entry >> > > > > > > > > > > dividing >> > > > > > > > > > > > > > between >> > > > > > > > > > > > > > >> >> > >> > deprecated >> > > > > > > > > > > > > > >> >> > >> > > versions is a bit hacky. What >> about >> > > > > instead >> > > > > > > > having >> > > > > > > > > > an >> > > > > > > > > > > > > array >> > > > > > > > > > > > > > of >> > > > > > > > > > > > > > >> >> > >> supported >> > > > > > > > > > > > > > >> >> > >> > > versions and a separate array of >> > > > > deprecated >> > > > > > > > > > versions. >> > > > > > > > > > > > The >> > > > > > > > > > > > > > >> >> deprecated >> > > > > > > > > > > > > > >> >> > >> > > versions would always be a >> subset of >> > > the >> > > > > > > > supported >> > > > > > > > > > > > > versions. >> > > > > > > > > > > > > > >> Or, >> > > > > > > > > > > > > > >> >> > >> > > alternately, since deprecation >> has >> > no >> > > > > > > functional >> > > > > > > > > > > impact >> > > > > > > > > > > > > and >> > > > > > > > > > > > > > is >> > > > > > > > > > > > > > >> >> just >> > > > > > > > > > > > > > >> >> > a >> > > > > > > > > > > > > > >> >> > >> > > message to developers, we could >> just >> > > > leave >> > > > > > it >> > > > > > > > out >> > > > > > > > > of >> > > > > > > > > > > the >> > > > > > > > > > > > > > >> protocol >> > > > > > > > > > > > > > >> >> > and >> > > > > > > > > > > > > > >> >> > >> > just >> > > > > > > > > > > > > > >> >> > >> > > have it in release notes etc. >> > > > > > > > > > > > > > >> >> > >> > > 2. I think including the api name >> > may >> > > > > cause >> > > > > > > some >> > > > > > > > > > > > problems. >> > > > > > > > > > > > > > >> >> Currently >> > > > > > > > > > > > > > >> >> > >> the >> > > > > > > > > > > > > > >> >> > >> > > api key is the primary key that >> we >> > > keep >> > > > > > > > consistent >> > > > > > > > > > but >> > > > > > > > > > > > we >> > > > > > > > > > > > > > have >> > > > > > > > > > > > > > >> >> > >> actually >> > > > > > > > > > > > > > >> >> > >> > > evolved the english description >> of >> > the >> > > > > apis >> > > > > > as >> > > > > > > > > they >> > > > > > > > > > > have >> > > > > > > > > > > > > > >> changed. >> > > > > > > > > > > > > > >> >> > The >> > > > > > > > > > > > > > >> >> > >> > only >> > > > > > > > > > > > > > >> >> > >> > > use I can think of for the name >> > would >> > > be >> > > > > if >> > > > > > > > people >> > > > > > > > > > > used >> > > > > > > > > > > > > the >> > > > > > > > > > > > > > >> >> logical >> > > > > > > > > > > > > > >> >> > >> name >> > > > > > > > > > > > > > >> >> > >> > > and tried to resolve the api key, >> > but >> > > > that >> > > > > > > would >> > > > > > > > > be >> > > > > > > > > > > > wrong. >> > > > > > > > > > > > > > Not >> > > > > > > > > > > > > > >> >> sure >> > > > > > > > > > > > > > >> >> > >> if we >> > > > > > > > > > > > > > >> >> > >> > > actually need the english name, >> if >> > > there >> > > > > is >> > > > > > a >> > > > > > > > use >> > > > > > > > > > > case I >> > > > > > > > > > > > > > guess >> > > > > > > > > > > > > > >> >> we'll >> > > > > > > > > > > > > > >> >> > >> just >> > > > > > > > > > > > > > >> >> > >> > > have to be very clear that the >> name >> > is >> > > > > just >> > > > > > > > > > > > documentation >> > > > > > > > > > > > > > and >> > > > > > > > > > > > > > >> can >> > > > > > > > > > > > > > >> >> > >> change >> > > > > > > > > > > > > > >> >> > >> > > any time. >> > > > > > > > > > > > > > >> >> > >> > > >> > > > > > > > > > > > > > >> >> > >> > > -Jay >> > > > > > > > > > > > > > >> >> > >> > > >> > > > > > > > > > > > > > >> >> > >> > > On Fri, Sep 25, 2015 at 2:53 PM, >> > > Magnus >> > > > > > > > Edenhill < >> > > > > > > > > > > > > > >> >> > mag...@edenhill.se> >> > > > > > > > > > > > > > >> >> > >> > > wrote: >> > > > > > > > > > > > > > >> >> > >> > > >> > > > > > > > > > > > > > >> >> > >> > > > Good evening, >> > > > > > > > > > > > > > >> >> > >> > > > >> > > > > > > > > > > > > > >> >> > >> > > > KIP-35 was created to address >> > > current >> > > > > and >> > > > > > > > future >> > > > > > > > > > > > > > >> broker-client >> > > > > > > > > > > > > > >> >> > >> > > > compatibility. >> > > > > > > > > > > > > > >> >> > >> > > > >> > > > > > > > > > > > > > >> >> > >> > > > >> > > > > > > > > > > > > > >> >> > >> > > > >> > > > > > > > > > > > > > >> >> > >> > > >> > > > > > > > > > > > > > >> >> > >> > >> > > > > > > > > > > > > > >> >> > >> >> > > > > > > > > > > > > > >> >> > >> > > > > > > > > > > > > > >> >> >> > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-35+-+Retrieving+protocol+version >> > > > > > > > > > > > > > >> >> > >> > > > >> > > > > > > > > > > > > > >> >> > >> > > > Summary: >> > > > > > > > > > > > > > >> >> > >> > > > * allow clients to retrieve >> the >> > > > > broker's >> > > > > > > > > protocol >> > > > > > > > > > > > > version >> > > > > > > > > > > > > > >> >> > >> > > > * make broker handle unknown >> > > protocol >> > > > > > > > requests >> > > > > > > > > > > > > gracefully >> > > > > > > > > > > > > > >> >> > >> > > > >> > > > > > > > > > > > > > >> >> > >> > > > Feedback and comments welcome! >> > > > > > > > > > > > > > >> >> > >> > > > >> > > > > > > > > > > > > > >> >> > >> > > > Regards, >> > > > > > > > > > > > > > >> >> > >> > > > Magnus >> > > > > > > > > > > > > > >> >> > >> > > > >> > > > > > > > > > > > > > >> >> > >> > > >> > > > > > > > > > > > > > >> >> > >> > >> > > > > > > > > > > > > > >> >> > >> >> > > > > > > > > > > > > > >> >> > > >> > > > > > > > > > > > > > >> >> > > >> > > > > > > > > > > > > > >> >> > > >> > > > > > > > > > > > > > >> >> > > -- >> > > > > > > > > > > > > > >> >> > > >> > > > > > > > > > > > > > >> >> > > Regards, >> > > > > > > > > > > > > > >> >> > > Ashish >> > > > > > > > > > > > > > >> >> > > >> > > > > > > > > > > > > > >> >> > >> > > > > > > > > > > > > > >> >> > >> > > > > > > > > > > > > > >> >> > >> > > > > > > > > > > > > > >> >> > -- >> > > > > > > > > > > > > > >> >> > >> > > > > > > > > > > > > > >> >> > Regards, >> > > > > > > > > > > > > > >> >> > Ashish >> > > > > > > > > > > > > > >> >> > >> > > > > > > > > > > > > > >> >> >> > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > >> > -- >> > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > >> > Regards, >> > > > > > > > > > > > > > >> > Ashish >> > > > > > > > > > > > > > >> >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > -- >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > Regards, >> > > > > > > > > > > > > > > Ashish >> > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > -- >> > > > > > > > > > > > > >> > > > > > > > > > > > > Regards, >> > > > > > > > > > > > > Ashish >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > -- >> > > > > > >> > > > > > Regards, >> > > > > > Ashish >> > > > > > >> > > > > >> > > > >> > > >> > >> > >> > >> > -- >> > >> > Regards, >> > Ashish >> > >> > > > > -- > > Regards, > Ashish > -- Regards, Ashish
