[jira] [Updated] (KAFKA-693) Consumer rebalance fails if no leader available for a partition and stops all fetchers

2013-01-17 Thread Maxime Brugidou (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Brugidou updated KAFKA-693: -- Attachment: KAFKA-693-v3.patch Added v3 with your remarks > Consumer rebalance

Re: dynamic topic config

2013-01-17 Thread Jun Rao
Posted a couple of comments on the wiki. Thanks, Jun On Wed, Jan 16, 2013 at 8:18 PM, Jay Kreps wrote: > I added a first pass at a proposal for moving topic configuration into > zookeeper so that you would specify the config when you created the topic > and config could be altered without boun

Re: zookeeper partition data

2013-01-17 Thread Jun Rao
Using separate ZK paths for replicas assignment and configs seems better since it allows each of them to be updated independently. Yes, we should make a path of values stored in ZK for future extension. Could you file a jira? Thanks, Jun On Wed, Jan 16, 2013 at 7:47 PM, Jay Kreps wrote: > Why

Re: Abou Kafka 0.8 producer throughput test

2013-01-17 Thread Jun Rao
We also did some perf test on 0.8 using the following command. All configs on the broker are the defaults. bin/kafka-run-class.sh kafka.perf.ProducerPerformance --broker-list localhost:9092 --initial-message-id 0 --messages 200 --topics topic_001 --request-num-acks -1 --batch-size 100 --threads

Re: dynamic topic config

2013-01-17 Thread Neha Narkhede
This will be a useful feature to have. Posted few thoughts on an alternative implementation. Thanks, Neha On Thu, Jan 17, 2013 at 7:56 AM, Jun Rao wrote: > Posted a couple of comments on the wiki. > > Thanks, > > Jun > > On Wed, Jan 16, 2013 at 8:18 PM, Jay Kreps wrote: > > > I added a first

Re: Abou Kafka 0.8 producer throughput test

2013-01-17 Thread Neha Narkhede
Looks like Jun's email didn't format the output properly. I've published some preliminary producer throughput performance numbers on our performance wiki - https://cwiki.apache.org/confluence/display/KAFKA/Performance+testing#Performancetesting-Producerthroughput These tests measure producer throu

[jira] [Commented] (KAFKA-691) Fault tolerance broken with replication factor 1

2013-01-17 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13556380#comment-13556380 ] Jun Rao commented on KAFKA-691: --- Actually, the current code works since partitionId is always

[jira] [Updated] (KAFKA-693) Consumer rebalance fails if no leader available for a partition and stops all fetchers

2013-01-17 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-693: -- Resolution: Fixed Fix Version/s: 0.8 Status: Resolved (was: Patch Available) Thanks for the p

[jira] Subscription: outstanding kafka patches

2013-01-17 Thread jira
Issue Subscription Filter: outstanding kafka patches (61 issues) The list of outstanding kafka patches Subscriber: kafka-mailing-list Key Summary KAFKA-709 Default queue.enqueue.timeout.ms to -1 https://issues.apache.org/jira/browse/KAFKA-709 KAFKA-708 ISR becomes e

[jira] [Created] (KAFKA-710) Some arguments are always set to default in ProducerPerformance

2013-01-17 Thread John Fung (JIRA)
John Fung created KAFKA-710: --- Summary: Some arguments are always set to default in ProducerPerformance Key: KAFKA-710 URL: https://issues.apache.org/jira/browse/KAFKA-710 Project: Kafka Issue Type

[jira] [Commented] (KAFKA-710) Some arguments are always set to default in ProducerPerformance

2013-01-17 Thread John Fung (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13556565#comment-13556565 ] John Fung commented on KAFKA-710: - The following arguments are always set to default values

[jira] [Updated] (KAFKA-710) Some arguments are always set to default in ProducerPerformance

2013-01-17 Thread John Fung (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-710: Attachment: kafka-710-v1.patch Uploaded kafka-710-v1.patch to remove the additional variables

[jira] [Updated] (KAFKA-710) Some arguments are always set to default in ProducerPerformance

2013-01-17 Thread John Fung (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-710: Status: Patch Available (was: Open) > Some arguments are always set to default in ProducerPerformance >

parittioner runtime error with kafka 0.8

2013-01-17 Thread Jane Wang
I defined a customized partitioner: class ExpertSearchCollectionPartitioner implements Partitioner { @Override public int partition(Long key, int numPartitions) { System.out.print("partition: " + key + ", " + numPartitions + ", "+ (key.hashCode() % numPartitions) + "\n"); retur

Re: Abou Kafka 0.8 producer throughput test

2013-01-17 Thread S Ahmed
producer.num.acks=-1 means what sorry? is it that all replica's are written too? On Thu, Jan 17, 2013 at 12:09 PM, Neha Narkhede wrote: > Looks like Jun's email didn't format the output properly. I've published > some preliminary producer throughput performance numbers on our performance > wiki

Re: parittioner runtime error with kafka 0.8

2013-01-17 Thread Jay Kreps
Hi Jane, This error is java.lang.NoSuchMethodException: com.twitter.expertsearch.indexing.jit.ExpertSearchCollectionPartitio ner.(kafka.utils.VerifiableProperties) It means that you need to have a constructor which takes a single argument, VerifiableProperties. This is a way to pass any configura

Re: parittioner runtime error with kafka 0.8

2013-01-17 Thread Jane Wang
I also tried with the constructor ExpertSearchCollectionPartitioner(VerifiableProperties props) { } Still the same error. I also tried the constructor without any parameter and with parameter Long. None of them worked. On Thu, Jan 17, 2013 at 7:08 PM, Jay Kreps wrote: > Hi Jane, > > This e

[jira] [Commented] (KAFKA-260) Add audit trail to kafka

2013-01-17 Thread Felix GV (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13556958#comment-13556958 ] Felix GV commented on KAFKA-260: It would be possible to have optional timestamps by using

Re: Abou Kafka 0.8 producer throughput test

2013-01-17 Thread Jun Rao
I means wait for the data reaches all replicas (that are in sync). Thanks, Jun On Thu, Jan 17, 2013 at 6:42 PM, S Ahmed wrote: > producer.num.acks=-1 means what sorry? is it that all replica's are written > too? > > > On Thu, Jan 17, 2013 at 12:09 PM, Neha Narkhede >wrote: > > > Looks like Ju

[jira] [Commented] (KAFKA-709) Default queue.enqueue.timeout.ms to -1

2013-01-17 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13556973#comment-13556973 ] Jay Kreps commented on KAFKA-709: - +1 > Default queue.enqueue.timeout.ms t

[jira] [Commented] (KAFKA-707) Improve error message in the producer when sending data to a partition without an active leader

2013-01-17 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13556974#comment-13556974 ] Jay Kreps commented on KAFKA-707: - What is the resulting error the user will see in this ca

[jira] [Commented] (KAFKA-682) java.lang.OutOfMemoryError: Java heap space

2013-01-17 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13556975#comment-13556975 ] Jay Kreps commented on KAFKA-682: - Yes, Joel, that makes sense. Ricky, do you know the "ack

[jira] [Closed] (KAFKA-709) Default queue.enqueue.timeout.ms to -1

2013-01-17 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede closed KAFKA-709. --- > Default queue.enqueue.timeout.ms to -1 > -- > >

[jira] [Updated] (KAFKA-709) Default queue.enqueue.timeout.ms to -1

2013-01-17 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-709: Status: Patch Available (was: In Progress) > Default queue.enqueue.timeout.ms to -1 > -

[jira] [Updated] (KAFKA-709) Default queue.enqueue.timeout.ms to -1

2013-01-17 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-709: Resolution: Fixed Status: Resolved (was: Patch Available) Thanks for the review

[jira] [Work started] (KAFKA-709) Default queue.enqueue.timeout.ms to -1

2013-01-17 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KAFKA-709 started by Neha Narkhede. > Default queue.enqueue.timeout.ms to -1 > -- > > Key: KAFKA-709 >

[jira] [Commented] (KAFKA-707) Improve error message in the producer when sending data to a partition without an active leader

2013-01-17 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13556979#comment-13556979 ] Neha Narkhede commented on KAFKA-707: - [2013-01-17 21:41:19,151] WARN Failed to send da