[jira] [Updated] (KAFKA-1036) Unable to rename replication offset checkpoint in windows

2013-08-29 Thread Timothy Chen (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Timothy Chen updated KAFKA-1036: Attachment: (was: kafka-1036.patch) > Unable to rename replication offset checkpoint in win

[jira] [Updated] (KAFKA-1036) Unable to rename replication offset checkpoint in windows

2013-08-29 Thread Timothy Chen (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Timothy Chen updated KAFKA-1036: Status: Open (was: Patch Available) > Unable to rename replication offset checkpoint in window

RE: Review board: Not able to tag kafka group

2013-08-29 Thread Tejas Patil
Correction: Link for [1] is https://reviews.apache.org/groups/kafka/ ~tejas From: Tejas Patil [tejas.patil...@gmail.com] Sent: Thursday, August 29, 2013 6:36 PM To: dev@kafka.apache.org Subject: Review board: Not able to tag kafka group Hi, I am tried out

Review board: Not able to tag kafka group

2013-08-29 Thread Tejas Patil
Hi, I am tried out the instructions over [0] to use review board and it worked like a charm. Over RB, when I tried to tag "kafka" in the "groups" field, it said that the group does not exist. Group tagging sends out notification to all the members of a group (in this case the "kafka" developers gr

[jira] [Commented] (KAFKA-1008) Unmap before resizing

2013-08-29 Thread David Lao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13754293#comment-13754293 ] David Lao commented on KAFKA-1008: -- Thanks Jay. The 0.8 patch seems to be working on Wind

[jira] [Commented] (KAFKA-1012) Implement an Offset Manager and hook offset requests to it

2013-08-29 Thread Tejas Patil (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13754280#comment-13754280 ] Tejas Patil commented on KAFKA-1012: The initial patch has been uploaded to review boa

[jira] [Commented] (KAFKA-1035) Add message-send-max-retries and retry-backoff-ms options to console producer

2013-08-29 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13754254#comment-13754254 ] Guozhang Wang commented on KAFKA-1035: -- Thanks for the patch. One minor comment: Cou

Re: Segment recovery and replication

2013-08-29 Thread Jay Kreps
Can't he get this automatically though with the Sriram's controlled shutdown stuff? -Jay On Thu, Aug 29, 2013 at 2:12 PM, Neha Narkhede wrote: > >> How do you automate waiting for the broker to come up? Just keep > monitoring the process and keep trying to connect to the port? > > Every leader

[jira] [Updated] (KAFKA-1032) Messages sent to the old leader will be lost on broker GC resulted failure

2013-08-29 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-1032: - Status: Patch Available (was: Open) Note that we also need to delay removal of dead brokers afte

[jira] [Updated] (KAFKA-1032) Messages sent to the old leader will be lost on broker GC resulted failure

2013-08-29 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-1032: - Attachment: KAFKA-1032.v1.patch > Messages sent to the old leader will be lost on broker GC r

[jira] [Updated] (KAFKA-1036) Unable to rename replication offset checkpoint in windows

2013-08-29 Thread Timothy Chen (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Timothy Chen updated KAFKA-1036: Affects Version/s: 0.8 Status: Patch Available (was: Open) > Unable to rename r

Re: Segment recovery and replication

2013-08-29 Thread Neha Narkhede
>> How do you automate waiting for the broker to come up? Just keep monitoring the process and keep trying to connect to the port? Every leader in a Kafka cluster exposes the UnderReplicatedPartitionCount metric. The safest way to issue controlled shutdown is to wait until that metric reports 0 on

Patch process

2013-08-29 Thread Timothy Chen
Hi Kafka devs, I've found a problem running Kafka on windows, created an Jira issue ( https://issues.apache.org/jira/browse/KAFKA-1036) and attached the patch on the issue as well. Some projects has a patch available status but I don't see it in the Kafka jira, I wonder what else I need to do? D

[jira] [Updated] (KAFKA-1036) Unable to rename replication offset checkpoint in windows

2013-08-29 Thread Timothy Chen (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Timothy Chen updated KAFKA-1036: Attachment: kafka-1036.patch > Unable to rename replication offset checkpoint in windows >

[jira] [Created] (KAFKA-1036) Unable to rename replication offset checkpoint in windows

2013-08-29 Thread Timothy Chen (JIRA)
Timothy Chen created KAFKA-1036: --- Summary: Unable to rename replication offset checkpoint in windows Key: KAFKA-1036 URL: https://issues.apache.org/jira/browse/KAFKA-1036 Project: Kafka Issue T

[jira] [Updated] (KAFKA-1036) Unable to rename replication offset checkpoint in windows

2013-08-29 Thread Timothy Chen (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Timothy Chen updated KAFKA-1036: Labels: windows (was: ) > Unable to rename replication offset checkpoint in windows >

[jira] [Commented] (KAFKA-1012) Implement an Offset Manager and hook offset requests to it

2013-08-29 Thread Tejas Patil (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13754024#comment-13754024 ] Tejas Patil commented on KAFKA-1012: > zkClient, logManager are implementation specifi

[jira] [Commented] (KAFKA-1014) Junits and system tests for inbuilt offset management

2013-08-29 Thread Tejas Patil (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13754005#comment-13754005 ] Tejas Patil commented on KAFKA-1014: Here is a plan for system tests. For this feature

Re: Segment recovery and replication

2013-08-29 Thread Sam Meder
Ok, I spent some more time staring at our logs and figured out that it was our fault. We were not waiting around for the Kafka broker to fully initialize before moving on to the next broker and loading the data logs can take quite some time (~7 minutes in one case), so we ended up with no repl

Looking for a Kafka 0.8 consultant

2013-08-29 Thread Joanna Kurn
Hello Kafka community, We are in the process of enabling Kafka 0.8 in production. We need to use it, rather than 0.7 because we need the ACK functionality and replication. We are running into some show-stopper issues (i.e. Kafka servers freezing) and need someone to come in for a day or so to help

[jira] Subscription: outstanding kafka patches

2013-08-29 Thread jira
Issue Subscription Filter: outstanding kafka patches (68 issues) The list of outstanding kafka patches Subscriber: kafka-mailing-list Key Summary KAFKA-1035 Add message-send-max-retries and retry-backoff-ms options to console producer https://issues.apache.org/jira/br

[jira] [Updated] (KAFKA-1007) Document new tools before 0.8 release

2013-08-29 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1007: - Resolution: Fixed Status: Resolved (was: Patch Available) Thanks for updating the docume

Re: Segment recovery and replication

2013-08-29 Thread Sam Meder
On Aug 29, 2013, at 5:50 PM, Sriram Subramanian wrote: > Do you know why you timed out on a regular shutdown? No, though I think it may just have been that the timeout we put in was too short. > If the replica had > fallen off of the ISR and shutdown was forced on the leader this could > hap

Re: Segment recovery and replication

2013-08-29 Thread Neha Narkhede
>> Right, although I was under the impression that committed meant replicated not necessarily synced to disk. That's not the case? That's correct. >> But what happens when a node goes down, has log truncated to less that in sync and becomes the leader again? You're saying there should be some cod

Re: Segment recovery and replication

2013-08-29 Thread Sam Meder
On Aug 29, 2013, at 5:44 PM, Jay Kreps wrote: > This should not happen. We have a notion of a "committed" message, which is > a message present on all "in sync" nodes. Right, although I was under the impression that committed meant replicated not necessarily synced to disk. That's not the case

[jira] [Updated] (KAFKA-1035) Add message-send-max-retries and retry-backoff-ms options to console producer

2013-08-29 Thread Rajasekar Elango (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajasekar Elango updated KAFKA-1035: Status: Patch Available (was: Open) See attachment for patch https://issues.apache.org/ji

[jira] [Updated] (KAFKA-1035) Add message-send-max-retries and retry-backoff-ms options to console producer

2013-08-29 Thread Rajasekar Elango (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajasekar Elango updated KAFKA-1035: Attachment: console_producer_add_options.patch Patch for fixing this. > Ad

[jira] [Created] (KAFKA-1035) Add message-send-max-retries and retry-backoff-ms options to console producer

2013-08-29 Thread Rajasekar Elango (JIRA)
Rajasekar Elango created KAFKA-1035: --- Summary: Add message-send-max-retries and retry-backoff-ms options to console producer Key: KAFKA-1035 URL: https://issues.apache.org/jira/browse/KAFKA-1035 Pro

Re: Segment recovery and replication

2013-08-29 Thread Sriram Subramanian
Do you know why you timed out on a regular shutdown? If the replica had fallen off of the ISR and shutdown was forced on the leader this could happen. With ack = -1, we guarantee that all the replicas in the in sync set have received the message before exposing the message to the consumer. On 8/29

Re: Segment recovery and replication

2013-08-29 Thread Jay Kreps
This should not happen. We have a notion of a "committed" message, which is a message present on all "in sync" nodes. We never hand out a message to any consumer until it is committed, and we guarantee that only "in sync" nodes are electable as leaders. Setting acks=-1 means wait until the message

Segment recovery and replication

2013-08-29 Thread Sam Meder
We've recently come across a scenario where we see consumers resetting their offsets to earliest and which as far as I can tell may also lead to data loss (we're running with ack = -1 to avoid loss). This seems to happen when we time out on doing a regular shutdown and instead kill -9 the kafka

[jira] [Commented] (KAFKA-1012) Implement an Offset Manager and hook offset requests to it

2013-08-29 Thread Sriram Subramanian (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-1012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13753726#comment-13753726 ] Sriram Subramanian commented on KAFKA-1012: --- Looks promising. Took a look at Off

hello

2013-08-29 Thread 马占雷
hi, My network does not connect to Internet. I want to install "kafka-0.7.2-incubating-src.tgz (asc, md5)", so I can not exec shell " ./sbt update ./sbt package". My quesion is in this case how I can install the software? mzl