Re: 0.8 producer -- many questions

I considered this as well; in my opinion, not all code will be so layered as the scala code. A topic metadata response would use the same mechanism, including the [reference]s. So in reality it's not *that* far away from BNF, but I am asking about more emphasis on layout, and revealing type/size

Re: 0.8 producer -- many questions

Jay, You're probably right about the version thing -- I tend toward the paranoid, but if support tooling is clear and easy to use, it turns out better for everybody. Especially if they are nicely versioned, and live nearby actual protocol docs :) b

Re: 0.8 producer -- many questions

I agree that BNFs are not the most readable thing in the world, but the fact is that our protocol has variable length fields and lots of repeated structures which are pretty confusing in a positional form. In practice I think the code one writes looks a lot like the BNF (e.g. readTopicMetadataReque

Re: 0.8 producer -- many questions

WRT the version I don't think it will really help to catch bugs. It is definitely possible for us to have protocol bugs, but I don't think that that version would catch the problem since the person breaking the protocol would probably not change the version number. I also think it is confusing for

Re: 0.8 producer -- many questions

Ben, It doesn't look like that you are subscribing to our mailing list. As a result, your post doesn't automatically show up in the mailing list and has to be manually approved. Please subscribe to our mailing list by emailing to the following addresses. dev-subscr...@kafka.apache.org users-sub

Re: 0.8 producer -- many questions

First off, let me amend my previous votes: +1 for both ApiKey and ApiVersion in all responses, even @ 4 bytes. The number of bytes is insignificant, useful in the debugging and multiplex scenarios, even if the underlying service doesn't yet support it. (The protocol still can.) CorrelationId can

Re: 0.8 producer -- many questions

+1 on keeping the field for sanity checking (it's only 1 byte) +1 on each API/protocol having a singular version, ie Request+Response are a single moving version, even if one side changes w/o the other changing b

Re: 0.8 producer -- many questions

These are all very good questions. Thanks for being lazy and trying the documentation, this feedback is helpful. - I added documentation on the ApiKey values for each request type to the wiki. - We are baselining all versions to 0, since we redid all the apis (looking at the code, I see we left a f

Re: 0.8 producer -- many questions

I am in the process of updating my erlang producer/consumer client to 0.8. Being lazy, I don't really want to go to Scala code. So I am just going through https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol.

Re: 0.8 producer -- many questions

My bad - you're right. I either missed that bit or was looking at the 0.7 code. Hopefully the remainder of this thread answers your original question. On Thu, Dec 6, 2012 at 7:07 AM, ben fleis wrote: > Since the v0.8 protocol wire changes went in, it's time for me to fix my > protocol things.

Re: 0.8 producer -- many questions

Cool, so did that make sense? tldr: requests are processed in the order sent and responses are returned in the order of requests. -Jay On Thu, Dec 6, 2012 at 9:40 AM, ben fleis wrote: > Ok. This is actually what I did in my first version of the code (the queue > of outstanding requests). I m

Re: 0.8 producer -- many questions

Ok. This is actually what I did in my first version of the code (the queue of outstanding requests). I merely thought that it might offer parallelism between variant request types. As you mentioned, increased parallelism is a good thing for performance, but my question was more about what is pro

Re: 0.8 producer -- many questions

Also we are trying to document all this stuff well with the 0.8 release. What could I do to improve this document: https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol Its kind of hard to write good documentation, so any feedback would be appreciated. -Jay On Thu, Dec

Re: 0.8 producer -- many questions

Hi Ben, If I understand your question, you are asking if it is possible to decouple or pipeline the sending of the request and the reading of the response. E.g. do something like send(req1) send(req2) recv(resp1) recv(resp2) I think your question is, if I do this how do I know what respons

Re: 0.8 producer -- many questions

I just realized something important. The BoundedByteBufferSend comment above is not in sync with my original question. I was asking about Responses -- meaning, can I multiplex off a single socket, and when reading an arbitrary response, switch on request_id to the right handler? In the blocking

Re: 0.8 producer -- many questions

Since the v0.8 protocol wire changes went in, it's time for me to fix my protocol things. One immediate question: On Tue, Nov 27, 2012 at 1:32 AM, Joel Koshy wrote: > The requestId for each request-type is written out to the wire (see > BoundedByteBufferSend) > The code I find writes a reques