[jira] [Commented] (KAFKA-690) TopicMetadataRequest throws exception when no topics are specified

2013-01-09 Thread Maxime Brugidou (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13548436#comment-13548436 ] Maxime Brugidou commented on KAFKA-690: --- this would resolve KAFKA-653

[jira] [Commented] (KAFKA-634) ConsoleProducer compresses messages and ignores the --compress flag

2013-01-09 Thread Maxime Brugidou (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13548441#comment-13548441 ] Maxime Brugidou commented on KAFKA-634: --- KAFKA-506 fixed this (commit f64fd3dcbaace1d

[jira] [Resolved] (KAFKA-634) ConsoleProducer compresses messages and ignores the --compress flag

2013-01-09 Thread Maxime Brugidou (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maxime Brugidou resolved KAFKA-634. --- Resolution: Fixed Fix Version/s: 0.8 > ConsoleProducer compresses messages and igno

[jira] [Commented] (KAFKA-690) TopicMetadataRequest throws exception when no topics are specified

2013-01-09 Thread David Arthur (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13548522#comment-13548522 ] David Arthur commented on KAFKA-690: Ah, I didn't see that JIRA. I was following https

Re: kafka 0.8 producer throughput

2013-01-09 Thread S Ahmed
What's the ack for? If it fails, it will try another broker? Can this be disabled or it's a major design change? On Wed, Jan 9, 2013 at 12:40 AM, Jun Rao wrote: > The 50MB/s number is for 0.7. We haven't carefully measured the performance > in 0.8 yet. We do expect the throughput that a singl

[jira] [Commented] (KAFKA-687) Rebalance algorithm should consider partitions from all topics

2013-01-09 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13548604#comment-13548604 ] Jay Kreps commented on KAFKA-687: - This is a very good point, and not one I had considered.

[jira] [Commented] (KAFKA-689) Can't append to a topic/partition that does not already exist

2013-01-09 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13548613#comment-13548613 ] Jay Kreps commented on KAFKA-689: - This is pretty hacky though, no? fetching metadata shoul

[jira] [Created] (KAFKA-691) Fault tolerance broken with replication factor 1

2013-01-09 Thread Jay Kreps (JIRA)
Jay Kreps created KAFKA-691: --- Summary: Fault tolerance broken with replication factor 1 Key: KAFKA-691 URL: https://issues.apache.org/jira/browse/KAFKA-691 Project: Kafka Issue Type: Bug Affect

Re: Kafka 0.8 without replication - handle broker failure/availability

2013-01-09 Thread Jay Kreps
This is a good point. We have discussed this a little bit before. The key constraint is that with replication factor 1 you can choose one of the following: (1) high availability, (2) correct semantic partitioning. That is to say, if a particular partition is unavailable you have no choice but to gi

[jira] [Commented] (KAFKA-689) Can't append to a topic/partition that does not already exist

2013-01-09 Thread David Arthur (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13548635#comment-13548635 ] David Arthur commented on KAFKA-689: Could the metadata API be modified with an "auto-c

[jira] [Commented] (KAFKA-690) TopicMetadataRequest throws exception when no topics are specified

2013-01-09 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13548638#comment-13548638 ] Jay Kreps commented on KAFKA-690: - +1 Thanks David! Folks, objections to putting this on 0

Re: kafka 0.8 producer throughput

2013-01-09 Thread Jun Rao
In 0.8, ack is always required. The ack returns an errorcode that indicates the reason if a produce request fails (e.g., the request is sent to a broker that's not a leader). It also returns the offset of the produced messages. However, the producer can choose when to receive the acks (e.g., when

[jira] [Commented] (KAFKA-689) Can't append to a topic/partition that does not already exist

2013-01-09 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13548646#comment-13548646 ] Jay Kreps commented on KAFKA-689: - Well I guess what I am saying is that getting metadata i

[jira] [Updated] (KAFKA-687) Rebalance algorithm should consider partitions from all topics

2013-01-09 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-687: Affects Version/s: (was: 0.8.1) 0.9 > Rebalance algorithm should cons

Re: kafka 0.8 producer throughput

2013-01-09 Thread Jay Kreps
We haven't done a ton of performance work on 0.8 yet. Regardless, requiring the ack will certainly reduce per-producer throughput, but it is too early to say by how much. Obviously this won't impact broker throughput (so if you have many producers you may not notice). The plan to fix this is just

[jira] [Commented] (KAFKA-689) Can't append to a topic/partition that does not already exist

2013-01-09 Thread David Arthur (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13548648#comment-13548648 ] David Arthur commented on KAFKA-689: I don't disagree really, but what I'm saying is: w

[jira] [Commented] (KAFKA-648) Use uniform convention for naming properties keys

2013-01-09 Thread John Fung (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13548650#comment-13548650 ] John Fung commented on KAFKA-648: - Hi Sriram, The change in system_test/migration_tool_te

Re: Kafka 0.8 without replication - handle broker failure/availability

2013-01-09 Thread Maxime Brugidou
Thanks for your response. I think the work-around is not really acceptable for me since it will consume 3x the resources (because replication of 3 is the minimum acceptable) and it will still make the cluster less available anyway (unless i have only 3 brokers). The thing is that 0.7 was making th

[jira] [Commented] (KAFKA-691) Fault tolerance broken with replication factor 1

2013-01-09 Thread Maxime Brugidou (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13548667#comment-13548667 ] Maxime Brugidou commented on KAFKA-691: --- I think the work-around is not really accept

[jira] [Commented] (KAFKA-690) TopicMetadataRequest throws exception when no topics are specified

2013-01-09 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13548673#comment-13548673 ] Neha Narkhede commented on KAFKA-690: - The protocol was meant to do this, so checked it

[jira] [Resolved] (KAFKA-690) TopicMetadataRequest throws exception when no topics are specified

2013-01-09 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede resolved KAFKA-690. - Resolution: Fixed Assignee: David Arthur > TopicMetadataRequest throws exception when no

[jira] [Commented] (KAFKA-662) Create testcases for unclean shut down

2013-01-09 Thread John Fung (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13548683#comment-13548683 ] John Fung commented on KAFKA-662: - 1. testcase_9071 - This testcase is to test the basic be

[jira] [Commented] (KAFKA-662) Create testcases for unclean shut down

2013-01-09 Thread John Fung (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13548689#comment-13548689 ] John Fung commented on KAFKA-662: - 2. testcase_9072 - This testcase is to test the basic be

[jira] [Commented] (KAFKA-689) Can't append to a topic/partition that does not already exist

2013-01-09 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13548746#comment-13548746 ] Jay Kreps commented on KAFKA-689: - Well I think there are actually two problems. The first

[jira] [Commented] (KAFKA-689) Can't append to a topic/partition that does not already exist

2013-01-09 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13548748#comment-13548748 ] Jun Rao commented on KAFKA-689: --- Currently, auto-creation is a broker-side flag. Basically, t

[jira] [Commented] (KAFKA-643) Refactor api definition layer

2013-01-09 Thread David Arthur (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13548797#comment-13548797 ] David Arthur commented on KAFKA-643: +1 for splitting generic/specific parts of the API

[jira] [Commented] (KAFKA-691) Fault tolerance broken with replication factor 1

2013-01-09 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13548847#comment-13548847 ] Jun Rao commented on KAFKA-691: --- One thing we can do is to change the partitioner api so that

[jira] [Updated] (KAFKA-688) System Test - Update all testcase_xxxx_properties.json for properties keys uniform naming convention

2013-01-09 Thread John Fung (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-688: Attachment: kafka-688-v1.patch > System Test - Update all testcase__properties.json for properties k

[jira] [Updated] (KAFKA-683) Fix correlation ids in all requests sent to kafka

2013-01-09 Thread Neha Narkhede (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-683: Attachment: kafka-683-v2.patch 1. DefaultEventHandler: Good suggestion, included it 2. FileMessageSe

[jira] [Commented] (KAFKA-688) System Test - Update all testcase_xxxx_properties.json for properties keys uniform naming convention

2013-01-09 Thread John Fung (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13548886#comment-13548886 ] John Fung commented on KAFKA-688: - Uploaded kafka-688-v1.patch which is required after chec

[jira] [Updated] (KAFKA-688) System Test - Update all testcase_xxxx_properties.json for properties keys uniform naming convention

2013-01-09 Thread John Fung (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John Fung updated KAFKA-688: Status: Patch Available (was: Open) > System Test - Update all testcase__properties.json for proper

[jira] [Updated] (KAFKA-648) Use uniform convention for naming properties keys

2013-01-09 Thread Sriram Subramanian (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sriram Subramanian updated KAFKA-648: - Attachment: configchanges-v4.patch Reverted the change from migrationtool/consumer.propert

[jira] [Commented] (KAFKA-648) Use uniform convention for naming properties keys

2013-01-09 Thread John Fung (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13548989#comment-13548989 ] John Fung commented on KAFKA-648: - Tested v4 patch with KAFKA-688-v1.patch with the latest

[jira] [Commented] (KAFKA-133) Publish kafka jar to a public maven repository

2013-01-09 Thread Evan Pollan (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13549067#comment-13549067 ] Evan Pollan commented on KAFKA-133: --- Just to clarify -- this is marked as resolved, but I

[jira] [Commented] (KAFKA-648) Use uniform convention for naming properties keys

2013-01-09 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13549075#comment-13549075 ] Jun Rao commented on KAFKA-648: --- Thanks for patch v4. Made another pass. Should we make the f

[jira] [Commented] (KAFKA-691) Fault tolerance broken with replication factor 1

2013-01-09 Thread Maxime Brugidou (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13549105#comment-13549105 ] Maxime Brugidou commented on KAFKA-691: --- I agree with Jun solution, this would solve

[jira] [Commented] (KAFKA-133) Publish kafka jar to a public maven repository

2013-01-09 Thread Joe Stein (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13549232#comment-13549232 ] Joe Stein commented on KAFKA-133: - << Just to clarify -- this is marked as resolved, but I

Re: kafka mavenification

2013-01-09 Thread Joe Stein
ok, in progress doing that now. On Mon, Jan 7, 2013 at 12:24 PM, Jun Rao wrote: > Joe, > > 0.8 will be released from the 0.8 branch and trunk is for post 0.8. So, you > will need to commit the maven changes to 0.8 and then merge them to trunk. > > Thanks, > > Jun > > On Sun, Jan 6, 2013 at 10:53

[jira] [Commented] (KAFKA-133) Publish kafka jar to a public maven repository

2013-01-09 Thread Joe Stein (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13549250#comment-13549250 ] Joe Stein commented on KAFKA-133: - committed to 0.8 branch also > Publish

[jira] [Commented] (KAFKA-691) Fault tolerance broken with replication factor 1

2013-01-09 Thread Jay Kreps (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13549276#comment-13549276 ] Jay Kreps commented on KAFKA-691: - That would be awesome. If you don't mind just give the p

Re: About Kafka 0.8 producer

2013-01-09 Thread 杨涛
Hi, Then is to say: for kafka 0.8, when you add a new broker into the cluster, you need not to change the producer broker setting, is it right? Bruce

Re: About Kafka 0.8 producer

2013-01-09 Thread Jay Kreps
Yes, in neither 0.7 or 0.8 should you need to change either the producer, consumer, or broker config when adding a new broker. -jay On Wed, Jan 9, 2013 at 8:18 PM, 杨涛 wrote: > Hi, > > Then is to say: for kafka 0.8, when you add a new broker into the > cluster, you need not to change t

Kafka open source projects

2013-01-09 Thread Jay Kreps
I have been noticing a lot of cool Kafka integrations floating around. I took some time and went through github and emails and tried to update the some of the pointers we have. S Powered By - If you are using Kafka, add yourself here. https://cwiki.apache.org/confluence/display/KAFKA/Powered

[jira] [Commented] (KAFKA-691) Fault tolerance broken with replication factor 1

2013-01-09 Thread Jun Rao (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13549373#comment-13549373 ] Jun Rao commented on KAFKA-691: --- DefaultEventHander.getPartitionListForTopic() returns Seq[Pa