[jira] [Commented] (KAFKA-597) Refactor KafkaScheduler

2012-12-07 Thread Joel Koshy (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13527084#comment-13527084 ] Joel Koshy commented on KAFKA-597: -- On daemon vs non-daemon and shutdown vs shutdownNow. I

[jira] [Updated] (KAFKA-646) Provide aggregate stats at the high level Producer and ZookeeperConsumerConnector level

2012-12-07 Thread Swapnil Ghike (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Swapnil Ghike updated KAFKA-646: Attachment: kafka-646-patch-num1-v2.patch Attached patch v2. The changes from patch v1: 1. Deleted

[jira] [Comment Edited] (KAFKA-646) Provide aggregate stats at the high level Producer and ZookeeperConsumerConnector level

2012-12-07 Thread Swapnil Ghike (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13527078#comment-13527078 ] Swapnil Ghike edited comment on KAFKA-646 at 12/8/12 6:17 AM: --

[jira] [Commented] (KAFKA-664) Kafka server threads die due to OOME during long running test

2012-12-07 Thread Joel Koshy (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13527039#comment-13527039 ] Joel Koshy commented on KAFKA-664: -- +1 Some minor comments: - We can probably remove the

[jira] [Updated] (KAFKA-597) Refactor KafkaScheduler

2012-12-07 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps updated KAFKA-597: Attachment: KAFKA-597-v5.patch Thanks, new patch v5 addresses your comments: - Improved javadoc - This is ac

[jira] [Commented] (KAFKA-646) Provide aggregate stats at the high level Producer and ZookeeperConsumerConnector level

2012-12-07 Thread Swapnil Ghike (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13527019#comment-13527019 ] Swapnil Ghike commented on KAFKA-646: - Actually I just realized that the Aggregated*Sta

[jira] [Updated] (KAFKA-664) Kafka server threads die due to OOME during long running test

2012-12-07 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-664: Attachment: watchersForKey.png kafka-664-draft.patch The problem was ever increasing

[jira] [Comment Edited] (KAFKA-664) Kafka server threads die due to OOME during long running test

2012-12-07 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13526973#comment-13526973 ] Neha Narkhede edited comment on KAFKA-664 at 12/8/12 1:56 AM: --

[jira] [Commented] (KAFKA-597) Refactor KafkaScheduler

2012-12-07 Thread Joel Koshy (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13526966#comment-13526966 ] Joel Koshy commented on KAFKA-597: -- I haven't fully reviewed, but a couple of initial comm

[jira] [Updated] (KAFKA-646) Provide aggregate stats at the high level Producer and ZookeeperConsumerConnector level

2012-12-07 Thread Swapnil Ghike (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Swapnil Ghike updated KAFKA-646: Attachment: kafka-646-patch-num1-v1.patch This patch has a bunch of refactoring changes and a couple

[jira] [Updated] (KAFKA-644) System Test should run properly with mixed File System Pathname

2012-12-07 Thread John Fung (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-644: Attachment: kafka-644-v2.patch Uploaded kafka-644-v2.patch which supports the property "auto_create_topic"

[jira] [Commented] (KAFKA-664) Kafka server threads die due to OOME during long running test

2012-12-07 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13526887#comment-13526887 ] Jay Kreps commented on KAFKA-664: - Another issue is that we are saving the full producer re

[jira] [Commented] (KAFKA-664) Kafka server threads die due to OOME during long running test

2012-12-07 Thread Joel Koshy (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13526883#comment-13526883 ] Joel Koshy commented on KAFKA-664: -- Okay I'm slightly confused. Even on expiration the req

[jira] [Updated] (KAFKA-636) Make log segment delete asynchronous

2012-12-07 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps updated KAFKA-636: Attachment: KAFKA-636-v1.patch This patch implements asynchronous delete in the log. To do this Log.scala n

[jira] [Commented] (KAFKA-664) Kafka server threads die due to OOME during long running test

2012-12-07 Thread Joel Koshy (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13526876#comment-13526876 ] Joel Koshy commented on KAFKA-664: -- To clarify, the map itself shouldn't grow indefinitely

[jira] [Updated] (KAFKA-597) Refactor KafkaScheduler

2012-12-07 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps updated KAFKA-597: Attachment: KAFKA-597-v4.patch Patch v4. - Rebased - Makes use of thread factory - Fixed broken scaladoc

[jira] [Closed] (KAFKA-644) System Test should run properly with mixed File System Pathname

2012-12-07 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede closed KAFKA-644. --- > System Test should run properly with mixed File System Pathname > --

[jira] [Updated] (KAFKA-644) System Test should run properly with mixed File System Pathname

2012-12-07 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-644: Resolution: Fixed Status: Resolved (was: Patch Available) > System Test should run prop

[jira] [Commented] (KAFKA-644) System Test should run properly with mixed File System Pathname

2012-12-07 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13526851#comment-13526851 ] Neha Narkhede commented on KAFKA-644: - +1. Thanks for the patch ! > Sy

[jira] [Updated] (KAFKA-666) Fetch requests from the replicas take several seconds to complete on the leader

2012-12-07 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-666: Description: I've seen that fetch requests from the replicas take several seconds to complete. The

[jira] [Created] (KAFKA-666) Fetch requests from the replicas take several seconds to complete on the leader

2012-12-07 Thread Neha Narkhede (JIRA)
Neha Narkhede created KAFKA-666: --- Summary: Fetch requests from the replicas take several seconds to complete on the leader Key: KAFKA-666 URL: https://issues.apache.org/jira/browse/KAFKA-666 Project: Ka

[jira] [Created] (KAFKA-665) Outgoing responses delayed on a busy Kafka broker

2012-12-07 Thread Neha Narkhede (JIRA)
Neha Narkhede created KAFKA-665: --- Summary: Outgoing responses delayed on a busy Kafka broker Key: KAFKA-665 URL: https://issues.apache.org/jira/browse/KAFKA-665 Project: Kafka Issue Type: Bug

[jira] [Commented] (KAFKA-664) Kafka server threads die due to OOME during long running test

2012-12-07 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13526808#comment-13526808 ] Neha Narkhede commented on KAFKA-664: - The root cause seems to be that watchersForKey m

[jira] [Assigned] (KAFKA-664) Kafka server threads die due to OOME during long running test

2012-12-07 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps reassigned KAFKA-664: --- Assignee: Jay Kreps > Kafka server threads die due to OOME during long running test >

[jira] [Commented] (KAFKA-664) Kafka server threads die due to OOME during long running test

2012-12-07 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13526805#comment-13526805 ] Jay Kreps commented on KAFKA-664: - Looks like the problem is in request purgatory--watchers

[jira] [Commented] (KAFKA-664) Kafka server threads die due to OOME during long running test

2012-12-07 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13526720#comment-13526720 ] Neha Narkhede commented on KAFKA-664: - I'm re-running the tests with that option now

[jira] [Commented] (KAFKA-664) Kafka server threads die due to OOME during long running test

2012-12-07 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13526719#comment-13526719 ] Neha Narkhede commented on KAFKA-664: - Heap dump is here - http://people.apache.org/~n

[jira] [Commented] (KAFKA-664) Kafka server threads die due to OOME during long running test

2012-12-07 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13526713#comment-13526713 ] Jay Kreps commented on KAFKA-664: - One pain of oom is that the thing leaking the memory is

[jira] [Commented] (KAFKA-657) Add an API to commit offsets

2012-12-07 Thread David Arthur (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13526687#comment-13526687 ] David Arthur commented on KAFKA-657: I have started a wiki page for this design discuss

[jira] [Commented] (KAFKA-374) Move to java CRC32 implementation

2012-12-07 Thread David Arthur (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13526624#comment-13526624 ] David Arthur commented on KAFKA-374: Not sure how you guys feel about having Java in th

[jira] [Comment Edited] (KAFKA-374) Move to java CRC32 implementation

2012-12-07 Thread David Arthur (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13526624#comment-13526624 ] David Arthur edited comment on KAFKA-374 at 12/7/12 6:38 PM: - N

[jira] [Issue Comment Deleted] (KAFKA-374) Move to java CRC32 implementation

2012-12-07 Thread David Arthur (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Arthur updated KAFKA-374: --- Comment: was deleted (was: Pure-java crc32 taken from Hadoop) > Move to java CRC32 implementation

[jira] [Updated] (KAFKA-374) Move to java CRC32 implementation

2012-12-07 Thread David Arthur (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Arthur updated KAFKA-374: --- Attachment: KAFKA-374.patch Pure-java crc32 taken from Hadoop > Move to java CRC32 im

[jira] [Updated] (KAFKA-651) Create testcases on auto create topics

2012-12-07 Thread John Fung (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-651: Status: Patch Available (was: Open) > Create testcases on auto create topics >

[jira] [Updated] (KAFKA-651) Create testcases on auto create topics

2012-12-07 Thread John Fung (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-651: Attachment: kafka-651-v1.patch Uploaded kafka-651-v1.patch with 1 testcase to cover each functional group: t

[jira] [Commented] (KAFKA-664) Kafka server threads die due to OOME during long running test

2012-12-07 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13526590#comment-13526590 ] Neha Narkhede commented on KAFKA-664: - Another observation - The server is probably GCi

[jira] [Updated] (KAFKA-664) Kafka server threads die due to OOME during long running test

2012-12-07 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-664: Attachment: thread-dump.log Attaching a thread dump that shows - 1. 4 processor threads and the acc

[jira] [Created] (KAFKA-664) Kafka server threads die due to OOME during long running test

2012-12-07 Thread Neha Narkhede (JIRA)
Neha Narkhede created KAFKA-664: --- Summary: Kafka server threads die due to OOME during long running test Key: KAFKA-664 URL: https://issues.apache.org/jira/browse/KAFKA-664 Project: Kafka Issu

Re: 0.8 Protocol Status

2012-12-07 Thread Neha Narkhede
Ben, I would try to run DumpLogSegments to check if the server's data is not corrupted due to a bug in the producer. Thanks, Neha On Fri, Dec 7, 2012 at 7:17 AM, ben fleis wrote: > I was testing my own code, and using the console consumer against my > seemingly-working-producer code. Since the

[jira] [Updated] (KAFKA-662) Create testcases for unclean shut down

2012-12-07 Thread John Fung (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-662: Issue Type: Task (was: Bug) > Create testcases for unclean shut down >

[jira] [Created] (KAFKA-663) Add "deploy" feature to System Test

2012-12-07 Thread John Fung (JIRA)
John Fung created KAFKA-663: --- Summary: Add "deploy" feature to System Test Key: KAFKA-663 URL: https://issues.apache.org/jira/browse/KAFKA-663 Project: Kafka Issue Type: Task Reporter:

[jira] [Created] (KAFKA-662) Create testcases for unclean shut down

2012-12-07 Thread John Fung (JIRA)
John Fung created KAFKA-662: --- Summary: Create testcases for unclean shut down Key: KAFKA-662 URL: https://issues.apache.org/jira/browse/KAFKA-662 Project: Kafka Issue Type: Bug Reporter

Re: 0.8 Protocol Status

2012-12-07 Thread ben fleis
I was testing my own code, and using the console consumer against my seemingly-working-producer code. Since the last update, the console consumer crashes. I am going to try to track it down in the debugger and will come back with a patch if found. Command line: KAFKA_OPTS="-Xmx512M -server -Dlog

Re: 0.8/HEAD Console consumer breakage?

2012-12-07 Thread ben fleis
Dah. Misfire. Please ignore if this makes it to an inbox ;) -b On Fri, Dec 7, 2012 at 4:13 PM, ben fleis wrote: > So I was testing my own code, and using the console consumer against my > seemingly-working-producer code. Since the last update, the console > consumer crashes. I am going to

0.8/HEAD Console consumer breakage?

2012-12-07 Thread ben fleis
So I was testing my own code, and using the console consumer against my seemingly-working-producer code. Since the last update, the console consumer crashes. I am going to try to track it down in the debugger and will come back with a patch if found.

[jira] [Comment Edited] (KAFKA-581) provides windows batch script for starting Kafka/Zookeeper

2012-12-07 Thread antoine vianey (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13526440#comment-13526440 ] antoine vianey edited comment on KAFKA-581 at 12/7/12 3:11 PM: --

[jira] [Comment Edited] (KAFKA-581) provides windows batch script for starting Kafka/Zookeeper

2012-12-07 Thread antoine vianey (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13526440#comment-13526440 ] antoine vianey edited comment on KAFKA-581 at 12/7/12 3:11 PM: --

[jira] [Updated] (KAFKA-581) provides windows batch script for starting Kafka/Zookeeper

2012-12-07 Thread antoine vianey (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] antoine vianey updated KAFKA-581: - Attachment: zookeeper-server-stop.bat kafka-server-stop.bat Can you add this scrip