[jira] [Commented] (KAFKA-1008) Unmap before resizing

2013-08-28 Thread Sriram Subramanian (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13753311#comment-13753311 ] Sriram Subramanian commented on KAFKA-1008: --- Looks good. 1. OffsetIndex.scala

config documentation

2013-08-28 Thread Jay Kreps
I took a pass through the configuration docs ( http://localhost/documentation.html#configuration) and tried to make them more understandable. We really could do better at this. If we are going to take the time to make a configuration we should bother to explain exactly what it does and tell people

[jira] [Commented] (KAFKA-1032) Messages sent to the old leader will be lost on broker GC resulted failure

2013-08-28 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13753277#comment-13753277 ] Neha Narkhede commented on KAFKA-1032: -- It seems more natural to send a stop replica

Re: Creating a Consumer Thread Pool

2013-08-28 Thread Jun Rao
Do you "Shutting down Thread: " in the output? Are all threads shut down? Thanks, Jun On Wed, Aug 28, 2013 at 1:52 PM, David Williams wrote: > Hi Jun, > > Thanks for following up. I removed the statement but still see no > messages from the producer. Also when that statement is in with the >

[jira] [Commented] (KAFKA-1032) Messages sent to the old leader will be lost on broker GC resulted failure

2013-08-28 Thread Swapnil Ghike (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13753257#comment-13753257 ] Swapnil Ghike commented on KAFKA-1032: -- The problem is that the leader that GC-ed did

[jira] [Created] (KAFKA-1034) Port AddPartition and ReassignReplicas changes to trunk

2013-08-28 Thread Sriram Subramanian (JIRA)
Sriram Subramanian created KAFKA-1034: - Summary: Port AddPartition and ReassignReplicas changes to trunk Key: KAFKA-1034 URL: https://issues.apache.org/jira/browse/KAFKA-1034 Project: Kafka

[jira] [Updated] (KAFKA-347) change number of partitions of a topic online

2013-08-28 Thread Sriram Subramanian (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sriram Subramanian updated KAFKA-347: - Resolution: Fixed Status: Resolved (was: Patch Available) > change number of

[jira] [Closed] (KAFKA-347) change number of partitions of a topic online

2013-08-28 Thread Sriram Subramanian (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sriram Subramanian closed KAFKA-347. > change number of partitions of a topic online > --

[jira] [Resolved] (KAFKA-988) Make ReassignReplica tool more usable

2013-08-28 Thread Sriram Subramanian (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sriram Subramanian resolved KAFKA-988. -- Resolution: Fixed > Make ReassignReplica tool more usable >

[jira] [Closed] (KAFKA-988) Make ReassignReplica tool more usable

2013-08-28 Thread Sriram Subramanian (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sriram Subramanian closed KAFKA-988. > Make ReassignReplica tool more usable > - > >

[jira] [Closed] (KAFKA-886) Update info on Controlled shutdown and Preferred replica election tool

2013-08-28 Thread Sriram Subramanian (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sriram Subramanian closed KAFKA-886. > Update info on Controlled shutdown and Preferred replica election tool > -

[jira] [Resolved] (KAFKA-886) Update info on Controlled shutdown and Preferred replica election tool

2013-08-28 Thread Sriram Subramanian (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sriram Subramanian resolved KAFKA-886. -- Resolution: Fixed > Update info on Controlled shutdown and Preferred replica electio

[jira] [Created] (KAFKA-1033) Metadata requests do not always return the complete list of available brokers

2013-08-28 Thread Evan Huus (JIRA)
Evan Huus created KAFKA-1033: Summary: Metadata requests do not always return the complete list of available brokers Key: KAFKA-1033 URL: https://issues.apache.org/jira/browse/KAFKA-1033 Project: Kafka

[jira] [Commented] (KAFKA-1032) Messages sent to the old leader will be lost on broker GC resulted failure

2013-08-28 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13753180#comment-13753180 ] Guozhang Wang commented on KAFKA-1032: -- Proposed approach: 1. Add addStopReplicaRequ

[jira] [Created] (KAFKA-1032) Messages sent to the old leader will be lost on broker GC resulted failure

2013-08-28 Thread Guozhang Wang (JIRA)
Guozhang Wang created KAFKA-1032: Summary: Messages sent to the old leader will be lost on broker GC resulted failure Key: KAFKA-1032 URL: https://issues.apache.org/jira/browse/KAFKA-1032 Project: Kaf

[jira] [Commented] (KAFKA-988) Make ReassignReplica tool more usable

2013-08-28 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13753165#comment-13753165 ] Guozhang Wang commented on KAFKA-988: - Is this already resolved? > Mak

[jira] [Commented] (KAFKA-1030) Addition of partitions requires bouncing all the consumers of that topic

2013-08-28 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13753146#comment-13753146 ] Guozhang Wang commented on KAFKA-1030: -- I think the first option might work just okay

[jira] [Updated] (KAFKA-1008) Unmap before resizing

2013-08-28 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps updated KAFKA-1008: - Attachment: KAFKA-0.8-1008-v7.patch Attached a version rebased to 0.8. > Unmap before re

RE: replicas have different earliest offset

2013-08-28 Thread Luke Forehand
Jay, great information thank you. I am in a testing phase so I have been continually resetting the commit offsets of my consumers before re-running consumer performance tests. I realize now my retention policy was set as 7 days, and I had added 3 new brokers at day 5 and reassigned partitions

[jira] [Commented] (KAFKA-1007) Document new tools before 0.8 release

2013-08-28 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13753054#comment-13753054 ] Guozhang Wang commented on KAFKA-1007: -- Doc for ReassignPartitionsCommand is also add

[jira] [Updated] (KAFKA-1007) Document new tools before 0.8 release

2013-08-28 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-1007: - Status: Patch Available (was: Open) https://cwiki.apache.org/confluence/display/KAFKA/Administra

Re: replicas have different earliest offset

2013-08-28 Thread Jay Kreps
On a single server our retention window is always approximate and a lower bound on what is retained since we only discard full partitions at a time. That is if you say you want to retain 100GB and have a 1GB partition size we will discard the last partition when doing so would not bring the retaine

replicas have different earliest offset

2013-08-28 Thread Luke Forehand
I'm running into strange behavior when testing failure scenarios. I have 4 brokers and 8 partitions for a topic called "feed". I wrote a piece of code that prints out the partitionId, leaderId, and earliest offset for each partition. Here is the printed information about partition leader earl

Re: Creating a Consumer Thread Pool

2013-08-28 Thread David Williams
Hi Jun, Thanks for following up. I removed the statement but still see no messages from the producer. Also when that statement is in with the single threaded consumer example, it prints "non-empty iterator" for its toString method versus "empty iterator" in the non working multi stream example.

[jira] [Commented] (KAFKA-1030) Addition of partitions requires bouncing all the consumers of that topic

2013-08-28 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13752793#comment-13752793 ] Neha Narkhede commented on KAFKA-1030: -- Longer term, the right fix will be to move re

[jira] [Commented] (KAFKA-1030) Addition of partitions requires bouncing all the consumers of that topic

2013-08-28 Thread Swapnil Ghike (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13752679#comment-13752679 ] Swapnil Ghike commented on KAFKA-1030: -- Hmm, this will mean that the consumer client

[jira] [Commented] (KAFKA-1030) Addition of partitions requires bouncing all the consumers of that topic

2013-08-28 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13752619#comment-13752619 ] Neha Narkhede commented on KAFKA-1030: -- One alternative is to change the topic metada

[jira] [Updated] (KAFKA-955) After a leader change, messages sent with ack=0 are lost

2013-08-28 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-955: Affects Version/s: 0.8 > After a leader change, messages sent with ack=0 are lost >

[jira] [Updated] (KAFKA-955) After a leader change, messages sent with ack=0 are lost

2013-08-28 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-955: Resolution: Fixed Fix Version/s: 0.8 Status: Resolved (was: Patch Available) Comm

[jira] [Commented] (KAFKA-955) After a leader change, messages sent with ack=0 are lost

2013-08-28 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13752590#comment-13752590 ] Neha Narkhede commented on KAFKA-955: - This is great. +1. One improvement on logging -

[jira] [Commented] (KAFKA-1026) Dynamically Adjust Batch Size Upon Receiving MessageSizeTooLargeException

2013-08-28 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13752577#comment-13752577 ] Jay Kreps commented on KAFKA-1026: -- This is a good idea, but I recommend we put this off

[jira] [Commented] (KAFKA-1024) possible memory leak in 0.8 beta1 producer with G1GC

2013-08-28 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1024?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13752578#comment-13752578 ] Jay Kreps commented on KAFKA-1024: -- Interesting > possible memory le

[jira] [Commented] (KAFKA-955) After a leader change, messages sent with ack=0 are lost

2013-08-28 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13752583#comment-13752583 ] Jay Kreps commented on KAFKA-955: - +1 Gorgeous. > After a leader change, m

[jira] [Resolved] (KAFKA-1031) Little modification to the stop script to be able to kill the proper process

2013-08-28 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao resolved KAFKA-1031. Resolution: Fixed Fix Version/s: 0.8 Assignee: Vladislav Pernin Thanks for the patch. Commi

[jira] [Assigned] (KAFKA-1029) Zookeeper leader election stuck in ephemeral node retry loop

2013-08-28 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao reassigned KAFKA-1029: -- Assignee: Sam Meder (was: Neha Narkhede) > Zookeeper leader election stuck in ephemeral node retry

[jira] [Assigned] (KAFKA-995) Enforce that the value for replica.fetch.max.bytes is always >= the value for message.max.bytes

2013-08-28 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao reassigned KAFKA-995: - Assignee: Sam Meder > Enforce that the value for replica.fetch.max.bytes is always >= the value for >

[jira] [Assigned] (KAFKA-997) Provide a strict verification mode when reading configuration properties

2013-08-28 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao reassigned KAFKA-997: - Assignee: Sam Meder > Provide a strict verification mode when reading configuration properties > -

Re: Creating a Consumer Thread Pool

2013-08-28 Thread Jun Rao
Could you remove the following statement and see if it works? System.out.println("Created iterator " + it.toString() + " thread number " + threadNumber); Thanks, Jun On Tue, Aug 27, 2013 at 3:43 PM, David Williams wrote: > > Hi all, > > I checked out the java source and looked at the java exa

[jira] [Updated] (KAFKA-1030) Addition of partitions requires bouncing all the consumers of that topic

2013-08-28 Thread Swapnil Ghike (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Swapnil Ghike updated KAFKA-1030: - Description: Consumer may not notice new partitions because the propagation of the metadata to s

[jira] [Updated] (KAFKA-1030) Addition of partitions requires bouncing all the consumers of that topic

2013-08-28 Thread Swapnil Ghike (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Swapnil Ghike updated KAFKA-1030: - Description: Consumer may not notice new partitions because the propagation of the metadata to s

[jira] [Updated] (KAFKA-1031) Little modification to the stop script to be able to kill the proper process

2013-08-28 Thread Vladislav Pernin (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1031?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladislav Pernin updated KAFKA-1031: Attachment: 0001-Escape-the-.-in-the-kafka.Kafka-chain.patch > Little modification to t

[jira] [Created] (KAFKA-1031) Little modification to the stop script to be able to kill the proper process

2013-08-28 Thread Vladislav Pernin (JIRA)
Vladislav Pernin created KAFKA-1031: --- Summary: Little modification to the stop script to be able to kill the proper process Key: KAFKA-1031 URL: https://issues.apache.org/jira/browse/KAFKA-1031 Proj