These are all very good questions. Thanks for being lazy and trying the documentation, this feedback is helpful. - I added documentation on the ApiKey values for each request type to the wiki. - We are baselining all versions to 0, since we redid all the apis (looking at the code, I see we left a few at 1 in our own producer, so I need to change those). Since there is effectively only one version of the protocol right now we aren't checking this field.
With respect to versioning the response, you are assuming the problem is with you but I actually think the problem is with that field. I think the fact is we hadn't rigorously thought through and documented how this would work. So, now being the very last moment, would be a good time to do that. Here is what I recall of the thinking: First we thought it would be better to version on a per-api basis rather than version the whole protocol since many people implement just use a subset of the apis. So the intention was to have a separate version number per API. Our thought, iirc, was to version changes in both request and response separately. So if we added a field to the request it would be move to version 1 but the response would remain version 0 since it hadn't changed. Thinking about this from the client implementors point of view, though, this is not a good thing. In practice we are going to need to specify the correspondence between the request version and the response version. And if we do this why not just make them the same number? And if they are the same number why not omit the response version entirely? Thinking about it, I don't think that field is really adding value and I think we should just delete it. The only possible value would be for a client checking that it is in fact getting the format version that it expects, but I don't feel that that is something that needs to be checked on each request. That's slightly embarrassing, but I say we just delete it. -Jay On Thu, Dec 6, 2012 at 3:12 PM, Milind Parikh <milindpar...@gmail.com>wrote: > I am in the process of updating my erlang producer/consumer client to 0.8. > > Being lazy, I don't really want to go to Scala code. So I am just going > through > > https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol > .< > https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol > >Most > things are clear and clearer with discussion in this thread. > > However a few queries: > > RequestObject > (a) What are the specific values for the different requests in the > ApiKey? > (b) What are the specific legal values for ApiVersions? I assume that > this might change for future versions of kafka. > > ResponseObject > (a) Would there be a difference between ResponseVersion and the > ApiVersions of the RequestObject? Would be possible, in the future, to > request ApiVersion of X and get a ResponseVersion of Y? How would the > cliient know how to parse ResponseVersion of Y. This makes me think that I > don't understand the motivation behind ApiVersions. Can someone shed some > light? > > Thanks > Milind > > > > > > > > On Thu, Dec 6, 2012 at 9:21 AM, Jay Kreps <jay.kr...@gmail.com> wrote: > > > Also we are trying to document all this stuff well with the 0.8 release. > > What could I do to improve this document: > > > > > https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol > > > > Its kind of hard to write good documentation, so any feedback would be > > appreciated. > > > > -Jay > > > > > > On Thu, Dec 6, 2012 at 9:20 AM, Jay Kreps <jay.kr...@gmail.com> wrote: > > > > > Hi Ben, > > > > > > If I understand your question, you are asking if it is possible to > > > decouple or pipeline the sending of the request and the reading of the > > > response. E.g. do something like > > > send(req1) > > > send(req2) > > > recv(resp1) > > > recv(resp2) > > > I think your question is, if I do this how do I know what response goes > > to > > > what request. > > > > > > The answer is that the server guarantees in-order processing of > requests > > > on a given connection. So correlating requests to responses can be done > > by > > > keeping a queue of as-yet unanswered requests. In 0.8 we also added an > > > official correlation id (an integer field you can set that the server > > > passes back unchanged in the response. This is meant to help do tracing > > > across client and server and may help to debug your implementation. We > do > > > not currently send back the API key since there is only one correct > value > > > that could have--whatever the api of the corresponding request was. > > > > > > One caveat is that the server doesn't really fully pipeline request > > > processing at this time so if you do pipeline you may not get quite as > > much > > > parallelism as you expect (you can never get full parallelism because > of > > > the ordering guarantee). But in the current code until a response is > sent > > > for req1 the server won't start reading req2. This is not hard to fix > and > > > we will get to it in the next major release I think. > > > > > > -Jay > > > > > > > > > > > > On Thu, Dec 6, 2012 at 7:50 AM, ben fleis <b...@monkey.org> wrote: > > > > > >> I just realized something important. The BoundedByteBufferSend > comment > > >> above is not in sync with my original question. I was asking about > > >> Responses -- meaning, can I multiplex off a single socket, and when > > >> reading > > >> an arbitrary response, switch on request_id to the right handler? > > >> > > >> In the blocking console sync client (which is the only trivial way to > > >> follow the code that I've found -- please enlighten if there's an > async > > >> thing I can easily trace), this is a non issue, since you request, > wait, > > >> parse expected response. > > >> > > >> Or do I need instead to just open separate sockets? (Not a big deal, > > but > > >> nice to understand clearly.) > > >> > > >> ben > > >> > > > > > > > > >
