Re: kafka pull request: zkclient and scalatest library updates

2013-03-25 Thread Neha Narkhede
Thanks for the pull request, do you mind attaching the patches to the respective JIRAs ? That is how we review and accept patches in Kafka. -Neha On Mon, Mar 25, 2013 at 4:37 PM, polymorphic wrote: > GitHub user polymorphic opened a pull request: > > https://github.com/apache/kafka/pull/3 >

Re: Any update on the "distributed commit" problem?

2013-03-25 Thread Neha Narkhede
In Kafka 0.8, SimpleConsumer returns a list of (message,offset) pairs. Hence, for whatever batching you have in mind, you need to keep track of the start and end offset of the batch that you've fetched from Kafka. This is what your consumer needs to keep track of for every partition that it consume

kafka pull request: zkclient and scalatest library updates

2013-03-25 Thread polymorphic
GitHub user polymorphic opened a pull request: https://github.com/apache/kafka/pull/3 zkclient and scalatest library updates Following https://issues.apache.org/jira/browse/KAFKA-826 I forked the code and included fixes for 2 bugs I reported, https://issues.apache.org/jira/browse/K

Re: Any update on the "distributed commit" problem?

2013-03-25 Thread Scott Carey
In order to succeed in a client with two phase commit one needs the "coordinates" of the current batch of records from the client perspective of the batch (which may be many SimpleConsumer smaller batches). I believe this is the range of offsets plus any partition information that describe the bat

[jira] [Commented] (KAFKA-826) Make Kafka 0.8 depend on metrics 2.2.0 instead of 3.x

2013-03-25 Thread Dragos Manolescu (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13613243#comment-13613243 ] Dragos Manolescu commented on KAFKA-826: Good to see this pop on your radar screen.

[jira] [Commented] (KAFKA-826) Make Kafka 0.8 depend on metrics 2.2.0 instead of 3.x

2013-03-25 Thread Scott Carey (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13613192#comment-13613192 ] Scott Carey commented on KAFKA-826: --- Thank you! We will be able to test and validate thi

[jira] [Created] (KAFKA-826) Make Kafka 0.8 depend on metrics 2.2.0 instead of 3.x

2013-03-25 Thread Neha Narkhede (JIRA)
Neha Narkhede created KAFKA-826: --- Summary: Make Kafka 0.8 depend on metrics 2.2.0 instead of 3.x Key: KAFKA-826 URL: https://issues.apache.org/jira/browse/KAFKA-826 Project: Kafka Issue Type: B

[jira] [Commented] (KAFKA-133) Publish kafka jar to a public maven repository

2013-03-25 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13613151#comment-13613151 ] Neha Narkhede commented on KAFKA-133: - I filed KAFKA-826 to track the metrics 2.2.0 cha

[jira] [Updated] (KAFKA-791) Fix validation bugs in System Test

2013-03-25 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-791: -- Resolution: Fixed Fix Version/s: 0.8 Status: Resolved (was: Patch Available) Thanks for patch

[jira] [Commented] (KAFKA-133) Publish kafka jar to a public maven repository

2013-03-25 Thread Darren Sargent (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13613096#comment-13613096 ] Darren Sargent commented on KAFKA-133: -- This ticket is marked as fixed, but the latest

Re: Any update on the "distributed commit" problem?

2013-03-25 Thread Neha Narkhede
Today, the only safe way of controlling consumer state management is by using the SimpleConsumer. The application is responsible for checkpointing offsets. So, in your example, when you commit a database transaction, you can store your consumer's offset as part of the txn. So either your txn succee

[jira] Subscription: outstanding kafka patches

2013-03-25 Thread jira
Issue Subscription Filter: outstanding kafka patches (61 issues) The list of outstanding kafka patches Subscriber: kafka-mailing-list Key Summary KAFKA-823 merge 0.8 (51421fcc0111031bb77f779a6f6c00520d526a34) to trunk https://issues.apache.org/jira/browse/KAFKA-823 KA

Any update on the "distributed commit" problem?

2013-03-25 Thread Darren Sargent
This is where you are reading messages from a broker, doing something with the messages, then commit them to some permanent storage such as HBase. There is a race condition in commiting the offsets to Zookeeper; if the DB write succeeds, but the ZK commit fails for any reason, you'll get a dupli

[jira] [Updated] (KAFKA-791) Fix validation bugs in System Test

2013-03-25 Thread John Fung (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-791: Attachment: kafka-791-v4.patch Uploaded kafka-791-v4.patch with the following changes: 1. Added system_test

[jira] [Closed] (KAFKA-825) KafkaController.isActive() needs to be synchronized

2013-03-25 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede closed KAFKA-825. --- > KafkaController.isActive() needs to be synchronized > --

[jira] [Updated] (KAFKA-825) KafkaController.isActive() needs to be synchronized

2013-03-25 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-825: -- Resolution: Fixed Fix Version/s: 0.8 Status: Resolved (was: Patch Available) Thanks for the r

[jira] [Commented] (KAFKA-825) KafkaController.isActive() needs to be synchronized

2013-03-25 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13612880#comment-13612880 ] Neha Narkhede commented on KAFKA-825: - Thanks for the patch, +1 > Kafk

[jira] [Updated] (KAFKA-825) KafkaController.isActive() needs to be synchronized

2013-03-25 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-825: -- Status: Patch Available (was: Open) > KafkaController.isActive() needs to be synchronized > ---

[jira] [Updated] (KAFKA-825) KafkaController.isActive() needs to be synchronized

2013-03-25 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-825: -- Attachment: kafka-825.patch Attach a patch. > KafkaController.isActive() needs to be synchroniz

[jira] [Created] (KAFKA-825) KafkaController.isActive() needs to be synchronized

2013-03-25 Thread Jun Rao (JIRA)
Jun Rao created KAFKA-825: - Summary: KafkaController.isActive() needs to be synchronized Key: KAFKA-825 URL: https://issues.apache.org/jira/browse/KAFKA-825 Project: Kafka Issue Type: Bug C

[jira] [Closed] (KAFKA-813) Minor cleanup in Controller

2013-03-25 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede closed KAFKA-813. --- > Minor cleanup in Controller > --- > > Key: KAFKA-813 >

[jira] [Updated] (KAFKA-813) Minor cleanup in Controller

2013-03-25 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-813: Component/s: controller > Minor cleanup in Controller > --- > >

[jira] [Resolved] (KAFKA-813) Minor cleanup in Controller

2013-03-25 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao resolved KAFKA-813. --- Resolution: Fixed Thanks for patch v4. +1 and committed to 0.8. > Minor cleanup in Controller