KafkaStream, ConsumerConnector, and 0.8

2013-01-22 Thread David Arthur
By centralizing offset management (with the new Commit/FetchOffsets APIs), consuming is now much simpler and no longer relies on clients connecting directly to ZooKeeper. As such, I'm wondering what the fate of ZookeeperConsumerConnector is. Will it continue to be supported, or will it be repl

Re: KafkaStream, ConsumerConnector, and 0.8

2013-01-22 Thread Neha Narkhede
> As such, I'm wondering what the fate of ZookeeperConsumerConnector is. > Will it continue to be supported, or will it be replaced by a non-ZK > implementation (one that uses SimpleConsumer maybe?). We've been collecting feedback on a client re-design proposal for Kafka 0.9. Feel free to add com

[jira] [Updated] (KAFKA-708) ISR becomes empty while marking a partition offline

2013-01-22 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-708: Attachment: kafka-708-v2.patch 1. Good point. I moved setting the leader to -1 to the removeReplicaF

[jira] [Commented] (KAFKA-705) Controlled shutdown doesn't seem to work on more than one broker in a cluster

2013-01-22 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13559793#comment-13559793 ] Neha Narkhede commented on KAFKA-705: - +1 > Controlled shutdown doesn'

[jira] [Commented] (KAFKA-330) Add delete topic support

2013-01-22 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13559813#comment-13559813 ] Neha Narkhede commented on KAFKA-330: - Hey Prashanth, how's this JIRA coming along ?

[jira] [Created] (KAFKA-718) kafka-run-class.sh should use reasonable gc settings

2013-01-22 Thread Jay Kreps (JIRA)
Jay Kreps created KAFKA-718: --- Summary: kafka-run-class.sh should use reasonable gc settings Key: KAFKA-718 URL: https://issues.apache.org/jira/browse/KAFKA-718 Project: Kafka Issue Type: Bug Af

[jira] [Created] (KAFKA-719) Kafka broker shuts down due to irrecoverable IO error

2013-01-22 Thread Neha Narkhede (JIRA)
Neha Narkhede created KAFKA-719: --- Summary: Kafka broker shuts down due to irrecoverable IO error Key: KAFKA-719 URL: https://issues.apache.org/jira/browse/KAFKA-719 Project: Kafka Issue Type: B

[jira] [Created] (KAFKA-720) Migration tool halts

2013-01-22 Thread Neha Narkhede (JIRA)
Neha Narkhede created KAFKA-720: --- Summary: Migration tool halts Key: KAFKA-720 URL: https://issues.apache.org/jira/browse/KAFKA-720 Project: Kafka Issue Type: Bug Components: core

[jira] [Assigned] (KAFKA-718) kafka-run-class.sh should use reasonable gc settings

2013-01-22 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede reassigned KAFKA-718: --- Assignee: Neha Narkhede > kafka-run-class.sh should use reasonable gc settings > -

[jira] [Updated] (KAFKA-720) Migration tool halts

2013-01-22 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-720: Attachment: migration-tool-halts > Migration tool halts > > >

[jira] [Commented] (KAFKA-719) Kafka broker shuts down due to irrecoverable IO error

2013-01-22 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13559875#comment-13559875 ] Jay Kreps commented on KAFKA-719: - Isn't this the same as KAFKA-695? > Kaf

[jira] [Created] (KAFKA-721) FileNotFoundException on index file

2013-01-22 Thread Jun Rao (JIRA)
Jun Rao created KAFKA-721: - Summary: FileNotFoundException on index file Key: KAFKA-721 URL: https://issues.apache.org/jira/browse/KAFKA-721 Project: Kafka Issue Type: Bug Components: core,

[jira] [Created] (KAFKA-722) Path typos in kafka-run-class.sh

2013-01-22 Thread John Fung (JIRA)
John Fung created KAFKA-722: --- Summary: Path typos in kafka-run-class.sh Key: KAFKA-722 URL: https://issues.apache.org/jira/browse/KAFKA-722 Project: Kafka Issue Type: Bug Reporter: John

[jira] [Commented] (KAFKA-721) FileNotFoundException on index file

2013-01-22 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13559897#comment-13559897 ] Jun Rao commented on KAFKA-721: --- The following is what happened: 1. We are starting a followe

[jira] [Commented] (KAFKA-721) FileNotFoundException on index file

2013-01-22 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13559901#comment-13559901 ] Neha Narkhede commented on KAFKA-721: - >> Not quite sure how the index file can be gone

[jira] [Updated] (KAFKA-722) Path typos in kafka-run-class.sh

2013-01-22 Thread John Fung (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-722: Attachment: kafka-722.patch Upload kafka-722.patch > Path typos in kafka-run-class.sh > ---

[jira] [Updated] (KAFKA-722) Path typos in kafka-run-class.sh

2013-01-22 Thread John Fung (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-722: Status: Patch Available (was: Open) Fixing the path from scala_2.8.0 => scala-2.8.0 > Path

[jira] [Created] (KAFKA-723) Scala's default case class toString() is very inefficient

2013-01-22 Thread Neha Narkhede (JIRA)
Neha Narkhede created KAFKA-723: --- Summary: Scala's default case class toString() is very inefficient Key: KAFKA-723 URL: https://issues.apache.org/jira/browse/KAFKA-723 Project: Kafka Issue Typ

[jira] [Commented] (KAFKA-695) Broker shuts down due to attempt to read a closed index file

2013-01-22 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13559935#comment-13559935 ] Jay Kreps commented on KAFKA-695: - Some research: 1. Taking also KAFKA-719 we have examples

[jira] [Commented] (KAFKA-705) Controlled shutdown doesn't seem to work on more than one broker in a cluster

2013-01-22 Thread Joel Koshy (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13559968#comment-13559968 ] Joel Koshy commented on KAFKA-705: -- Thanks for reviewing. I checked-in the incremental pat

[jira] [Updated] (KAFKA-716) SimpleConsumerPerformance does not consume all available messages

2013-01-22 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-716: Priority: Blocker (was: Major) > SimpleConsumerPerformance does not consume all available messa

[jira] [Closed] (KAFKA-722) Path typos in kafka-run-class.sh

2013-01-22 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede closed KAFKA-722. --- > Path typos in kafka-run-class.sh > > > Key: KAFKA-7

[jira] [Commented] (KAFKA-722) Path typos in kafka-run-class.sh

2013-01-22 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13559972#comment-13559972 ] Neha Narkhede commented on KAFKA-722: - +1 > Path typos in kafka-run-c

[jira] [Updated] (KAFKA-722) Path typos in kafka-run-class.sh

2013-01-22 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-722: Resolution: Fixed Assignee: John Fung Status: Resolved (was: Patch Available) > P

[jira] [Created] (KAFKA-724) Allow automatic socket.send.buffer from operating system

2013-01-22 Thread Pablo Barrera (JIRA)
Pablo Barrera created KAFKA-724: --- Summary: Allow automatic socket.send.buffer from operating system Key: KAFKA-724 URL: https://issues.apache.org/jira/browse/KAFKA-724 Project: Kafka Issue Type

[jira] [Commented] (KAFKA-724) Allow automatic socket.send.buffer from operating system

2013-01-22 Thread Pablo Barrera (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13559984#comment-13559984 ] Pablo Barrera commented on KAFKA-724: - Apparently this is already fixed in 0.8.

[jira] [Commented] (KAFKA-133) Publish kafka jar to a public maven repository

2013-01-22 Thread Rajasekar Elango (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13560026#comment-13560026 ] Rajasekar Elango commented on KAFKA-133: [~charmalloc] Are we expecting 0.7.3 rele

[jira] [Comment Edited] (KAFKA-133) Publish kafka jar to a public maven repository

2013-01-22 Thread Rajasekar Elango (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13560026#comment-13560026 ] Rajasekar Elango edited comment on KAFKA-133 at 1/22/13 9:13 PM:

[jira] [Updated] (KAFKA-703) A fetch request in Fetch Purgatory can double count the bytes from the same delayed produce request

2013-01-22 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-703: Labels: p3 (was: ) > A fetch request in Fetch Purgatory can double count the bytes from the sam

[jira] [Closed] (KAFKA-699) Disallow clients to set replicaId in FetchRequest

2013-01-22 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede closed KAFKA-699. --- > Disallow clients to set replicaId in FetchRequest >

[jira] [Updated] (KAFKA-720) Migration tool halts

2013-01-22 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-720: Labels: p1 (was: ) > Migration tool halts > > > Key: KAFKA

[jira] [Updated] (KAFKA-695) Broker shuts down due to attempt to read a closed index file

2013-01-22 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-695: Labels: p1 (was: p2) > Broker shuts down due to attempt to read a closed index file > -

[jira] [Updated] (KAFKA-703) A fetch request in Fetch Purgatory can double count the bytes from the same delayed produce request

2013-01-22 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-703: Affects Version/s: (was: 0.8) 0.8.1 > A fetch request in Fetch Purgat

[jira] [Updated] (KAFKA-703) A fetch request in Fetch Purgatory can double count the bytes from the same delayed produce request

2013-01-22 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-703?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-703: Labels: (was: p3) > A fetch request in Fetch Purgatory can double count the bytes from the sam

[jira] [Updated] (KAFKA-570) Kafka should not need snappy jar at runtime

2013-01-22 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-570: Labels: bugs (was: bugs p4) > Kafka should not need snappy jar at runtime > ---

[jira] [Updated] (KAFKA-570) Kafka should not need snappy jar at runtime

2013-01-22 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-570: Priority: Major (was: Blocker) > Kafka should not need snappy jar at runtime >

[jira] [Updated] (KAFKA-700) log client ip when we log each request on the broker

2013-01-22 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-700: Labels: p2 (was: p1) > log client ip when we log each request on the broker > -

[jira] [Updated] (KAFKA-696) Fix toString() API for all requests to make logging easier to read

2013-01-22 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-696: Priority: Critical (was: Major) > Fix toString() API for all requests to make logging easier to

[jira] [Assigned] (KAFKA-720) Migration tool halts

2013-01-22 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede reassigned KAFKA-720: --- Assignee: Sriram Subramanian > Migration tool halts > > >

[jira] [Assigned] (KAFKA-695) Broker shuts down due to attempt to read a closed index file

2013-01-22 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede reassigned KAFKA-695: --- Assignee: Jun Rao > Broker shuts down due to attempt to read a closed index file > ---

[jira] [Updated] (KAFKA-721) FileNotFoundException on index file

2013-01-22 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-721: Labels: p1 (was: ) > FileNotFoundException on index file > ---

[jira] [Updated] (KAFKA-719) Kafka broker shuts down due to irrecoverable IO error

2013-01-22 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-719: Labels: bugs p1 (was: bugs) > Kafka broker shuts down due to irrecoverable IO error > -

[jira] [Commented] (KAFKA-714) ConsoleConsumer throws SocketTimeoutException when fetching topic metadata

2013-01-22 Thread John Fung (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13560156#comment-13560156 ] John Fung commented on KAFKA-714: - Thanks Jun for the patch. I have verified it with the la

[jira] [Created] (KAFKA-725) Broker Exception: Attempt to read with a maximum offset less than start offset

2013-01-22 Thread Chris Riccomini (JIRA)
Chris Riccomini created KAFKA-725: - Summary: Broker Exception: Attempt to read with a maximum offset less than start offset Key: KAFKA-725 URL: https://issues.apache.org/jira/browse/KAFKA-725 Project:

[jira] [Created] (KAFKA-726) Add ReplicaFetcherThread name to mbean names

2013-01-22 Thread Swapnil Ghike (JIRA)
Swapnil Ghike created KAFKA-726: --- Summary: Add ReplicaFetcherThread name to mbean names Key: KAFKA-726 URL: https://issues.apache.org/jira/browse/KAFKA-726 Project: Kafka Issue Type: Bug Af

[jira] [Commented] (KAFKA-725) Broker Exception: Attempt to read with a maximum offset less than start offset

2013-01-22 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13560246#comment-13560246 ] Neha Narkhede commented on KAFKA-725: - This looks exactly like KAFKA-698. Chris, did yo

[jira] [Updated] (KAFKA-726) Add ReplicaFetcherThread name to mbean names

2013-01-22 Thread Swapnil Ghike (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Swapnil Ghike updated KAFKA-726: Attachment: KAFKA-726.patch As described in the description above. > Add ReplicaFet

Re: Abou Kafka 0.8 producer throughput test

2013-01-22 Thread S Ahmed
Neha, I see, so that is a fairly substantial change, ofcourse it has its advantage of guaranteeing a higher degree of durability but as a significant cost (round trip that the consumer has to wait for). I know someone mentioned creating a asych. consumer with a future. Do you have a 'gut' feelin

Re: Abou Kafka 0.8 producer throughput test

2013-01-22 Thread Jay Kreps
This is a good question. As mentioned we have some experience running this with no ack and there are a lot of downsides. We considered making the ack optional, but this would complicate the producer api since we could give back the offset only in the case where there is an ack. Thinking about it

[jira] [Created] (KAFKA-727) broker can still expose uncommitted data to a consumer

2013-01-22 Thread Jun Rao (JIRA)
Jun Rao created KAFKA-727: - Summary: broker can still expose uncommitted data to a consumer Key: KAFKA-727 URL: https://issues.apache.org/jira/browse/KAFKA-727 Project: Kafka Issue Type: Bug

[jira] [Commented] (KAFKA-727) broker can still expose uncommitted data to a consumer

2013-01-22 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13560402#comment-13560402 ] Jun Rao commented on KAFKA-727: --- One way to fix this is for FileMessageSet.searchFor() to ret

[jira] [Commented] (KAFKA-727) broker can still expose uncommitted data to a consumer

2013-01-22 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13560410#comment-13560410 ] Jay Kreps commented on KAFKA-727: - Fantastic catch. I think another fix is to just save th

[jira] [Assigned] (KAFKA-727) broker can still expose uncommitted data to a consumer

2013-01-22 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao reassigned KAFKA-727: - Assignee: Jay Kreps > broker can still expose uncommitted data to a consumer > ---

[jira] [Commented] (KAFKA-727) broker can still expose uncommitted data to a consumer

2013-01-22 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13560412#comment-13560412 ] Jun Rao commented on KAFKA-727: --- Jay, sure, you can take this on. The way we saw this is that

[jira] [Commented] (KAFKA-726) Add ReplicaFetcherThread name to mbean names

2013-01-22 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13560421#comment-13560421 ] Jun Rao commented on KAFKA-726: --- Thanks for the patch. In ReplicaFetcherManager, it's actuall

[jira] [Commented] (KAFKA-726) Add ReplicaFetcherThread name to mbean names

2013-01-22 Thread Swapnil Ghike (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13560423#comment-13560423 ] Swapnil Ghike commented on KAFKA-726: - I actually deleted brokerConfig.brokerId. Replic

[jira] [Commented] (KAFKA-726) Add ReplicaFetcherThread name to mbean names

2013-01-22 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13560428#comment-13560428 ] Jun Rao commented on KAFKA-726: --- Sorry, I missed the brokerId change. Yes, you are right. Th