Thanks for drafting this. Made some minor edits and added some comments. Thanks,
Jun On Thu, Nov 29, 2012 at 3:17 PM, Jay Kreps <jay.kr...@gmail.com> wrote: > I started trying to document the 0.8 protocol from the code and write a > guide to client implementation. This is meant to be a more user-friendly > and up-to-date version of the proposal wiki we had on the protocol changes. > > Here is what I wrote up so far: > > https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol > > I would love feedback on this document. It would be great if we could > document everything you need to know to write a client so people don't need > to reverse engineer our code. > > In doing this I found a number of things, some which I feel should be fixed > in 0.8 some of which maybe can wait. > > 1. Correlation id is not used across all the requests. I don't think it can > work as intended because of this. > 2. On reflection I am not sure that we need a correlation id field. I think > that since we need to guarantee that processing is sequential on any > particular socket we can correlate with a simple queue. (e.g. as the client > sends messages it adds them to a queue and as it receives responses it just > correlates to whatever is at the head of the queue). > 3. The metadata response seems to have a number of problems. Among them is > that it weirdly repeats all the broker information many times. The response > includes the ISR, leader (maybe), and the replicas. Each of these repeat > all the broker information. This is super weird. I think what we should be > doing here is including all broker information for all brokers and then > just having the appropriate ids for the isr, leader, and replicas. > 4. For topic discovery I think we need to support the case where no topics > are specified in the metadata request and for this return information about > all topics. I don't think we do this now. > 5. I don't understand what the creator id is. > 6. The offset request and response is not fully thought through and should > be generalized. > > -Jay >
