[jira] [Updated] (KAFKA-757) System Test Hard Failure cases : "Fatal error during KafkaServerStable startup" when hard-failed broker is re-started

2013-02-13 Thread Swapnil Ghike (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Swapnil Ghike updated KAFKA-757: Attachment: kafka-757-v1.patch There are two parts to this patch : A. Move the sanity check to dete

[jira] [Updated] (KAFKA-757) System Test Hard Failure cases : "Fatal error during KafkaServerStable startup" when hard-failed broker is re-started

2013-02-13 Thread Swapnil Ghike (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Swapnil Ghike updated KAFKA-757: Attachment: kafka-757-v2.patch Sorry, in indexSlotFor() we should check if lastOffset is the same as

updating API guide on wiki

2013-02-13 Thread David Arthur
FYI, I made a few updates to https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol * Added offset commit/fetch APIs * Updated error codes * Updated RequestKeys I noticed there are two APIs that are not yet documented: LeaderAndIsr and StopReplica. Also I'm not sure ho

Re: Clients and partition leaders

2013-02-13 Thread David Arthur
Thanks, Jun, this answers my questions. I wasn't necessarily thinking of an HTTP interface like Solr, but rather the way it routes requests to leaders. However, since brokers are not aware of all the partition leaders, then the Solr approach will not work. I actually worked a bit on a REST in

Re: updating API guide on wiki

2013-02-13 Thread Jun Rao
David, Thanks for updating the wiki. StaleControllerEpochCode is used in LeaderAndIsr. It indicates that a broker receives a request from an old controller, potentially due to controller failover. We probably should document LeaderAndIsr and StopReplica. However, they are used internally in Kafka

[jira] [Commented] (KAFKA-757) System Test Hard Failure cases : "Fatal error during KafkaServerStable startup" when hard-failed broker is re-started

2013-02-13 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13577681#comment-13577681 ] Jun Rao commented on KAFKA-757: --- Thanks for the patch. Good catch. I think problem B can expl

Re: Clients and partition leaders

2013-02-13 Thread Jay Kreps
This is a good idea. There are actually two ways to implement this: 1. A RESTFUL interface, as Jun mentions. This might make more sense since if you don't mind the overhead of sending all the data twice then you probably won't mind the overhead of HTTP. 2. Re-route misdirected requests in the broke

Re: updating API guide on wiki

2013-02-13 Thread Jay Kreps
Since that code is in trunk not in 0.8 we are going to need to introduce some kind of versioning entry for each API. I would recommend we leave the internal broker-broker APIs undocumented for now. -Jay On Wed, Feb 13, 2013 at 6:58 AM, David Arthur wrote: > FYI, I made a few updates to > https:

[jira] [Commented] (KAFKA-736) Add an option to the 0.8 producer to mimic 0.7 producer behavior

2013-02-13 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13577800#comment-13577800 ] Jun Rao commented on KAFKA-736: --- Because of the problem identified in KAFKA-706, only the v3

[jira] [Updated] (KAFKA-757) System Test Hard Failure cases : "Fatal error during KafkaServerStable startup" when hard-failed broker is re-started

2013-02-13 Thread Swapnil Ghike (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Swapnil Ghike updated KAFKA-757: Attachment: kafka-757-v3.patch Yes, your point is valid. Jay also suggested to implement truncate()

versionId in responses (and general API versioning questions)

2013-02-13 Thread David Arthur
Just noticed that most of the API response structs do not include the ApiVersion. This will make it hard for clients to determine how to handle responses once the APIs start getting updated (which is pretty much inevitable). I'd propose we update the standard response envelope to include it R

Re: versionId in responses (and general API versioning questions)

2013-02-13 Thread Jay Kreps
Hey David, We ended up not versioning the response, instead the version must correspond to the request version. This makes sense from the client point of view. If you send a request using version X of the protocol you know you will get a response in format X. Separately versioning the response wo

[jira] [Resolved] (KAFKA-757) System Test Hard Failure cases : "Fatal error during KafkaServerStable startup" when hard-failed broker is re-started

2013-02-13 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao resolved KAFKA-757. --- Resolution: Fixed Fix Version/s: 0.8 Thanks for patch v3. Committed to 0.8. > System T

Re: versionId in responses (and general API versioning questions)

2013-02-13 Thread David Arthur
On 2/13/13 4:08 PM, Jay Kreps wrote: Hey David, We ended up not versioning the response, instead the version must correspond to the request version. This makes sense from the client point of view. If you send a request using version X of the protocol you know you will get a response in format X

[jira] [Commented] (KAFKA-524) Kafka Appender causes Log4j Deadlock

2013-02-13 Thread David Arthur (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13578104#comment-13578104 ] David Arthur commented on KAFKA-524: I have seen this as well. Particularly this is an