Re: kafka performance question

How many brokers and partitions do you have? You may try increasing batch.num.messages. Thanks, Jun On Tue, May 13, 2014 at 5:56 PM, Zhujie (zhujie, Smartcare) < first.zhu...@huawei.com> wrote: > Dear all, > > We want to use kafka to collect and dispatch data file, but the > performance is may

[jira] [Updated] (KAFKA-1445) New Producer should send all partitions that have non-empty batches when on of them is ready

[ https://issues.apache.org/jira/browse/KAFKA-1445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-1445: - Attachment: KAFKA-1445_2014-05-14_16:24:25.patch > New Producer should send all partitions that h

[jira] [Commented] (KAFKA-1431) ConsoleConsumer - Option to clean zk consumer path

[ https://issues.apache.org/jira/browse/KAFKA-1431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13993141#comment-13993141 ] Sriharsha Chintalapani commented on KAFKA-1431: --- Created reviewboard agains

[jira] [Created] (KAFKA-1446) Consumer metrics for rebalance

Clark Haskins created KAFKA-1446: Summary: Consumer metrics for rebalance Key: KAFKA-1446 URL: https://issues.apache.org/jira/browse/KAFKA-1446 Project: Kafka Issue Type: Improvement

Re: Review Request 21123: Patch for KAFKA-1437

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/21123/#review42514 --- Ship it! Ship It! - Neha Narkhede On May 6, 2014, 5:56 p.m., Joe

Re: Review Request 21174: Fix KAFKA-1396

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/21174/#review42534 --- core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTe

Re: Review Request 19731: Patch for KAFKA-1328

> On May 7, 2014, 4:50 p.m., Jun Rao wrote: > > clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerRecord.java, > > lines 37-38 > > > > > > partitionId can probably be just partition to be consistent wit

kafka performance question

Dear all, We want to use kafka to collect and dispatch data file, but the performance is maybe lower than we want. In our cluster,there is a provider and a broker. We use a one thread read file from local disk of provider and send it to broker. The average throughput is only 3 MB/S~4MB/S. But

Re: Review Request 21239: Patch for KAFKA-1431

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/21239/ --- (Updated May 14, 2014, 6:33 p.m.) Review request for kafka. Bugs: KAFKA-1431

Re: Max Message Size

max.message.bytes This is largest message size Kafka will allow to be appended to this topic. Note that if you increase this size you must also increase your consumer's fetch size so they can fetch messages this large. When the message length is larger than max,message.bytes maybe throw an excep

Re: Review Request 21398: Fix KAFKA-1445 v2

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/21398/#review42985 --- clients/src/main/java/org/apache/kafka/clients/producer/internals/R

[jira] [Commented] (KAFKA-1182) Topic not created if number of live brokers less than # replicas

[ https://issues.apache.org/jira/browse/KAFKA-1182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13997445#comment-13997445 ] Simon Cooper commented on KAFKA-1182: - The current behaviour is causing problems for u

Re: Review Request 21239: Patch for KAFKA-1431

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/21239/ --- (Updated May 14, 2014, 7:08 p.m.) Review request for kafka. Bugs: KAFKA-1431

[jira] [Commented] (KAFKA-1449) Extend wire protocol to allow CRC32C

[ https://issues.apache.org/jira/browse/KAFKA-1449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13997681#comment-13997681 ] Neha Narkhede commented on KAFKA-1449: -- Interesting. Would you like to write up a pro

[jira] [Updated] (KAFKA-1431) ConsoleConsumer - Option to clean zk consumer path

[ https://issues.apache.org/jira/browse/KAFKA-1431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sriharsha Chintalapani updated KAFKA-1431: -- Attachment: KAFKA-1431_2014-05-14_11:34:00.patch > ConsoleConsumer - Option to

[jira] [Commented] (KAFKA-1450) check invalid leader in a more robust way

[ https://issues.apache.org/jira/browse/KAFKA-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13997267#comment-13997267 ] Jun Rao commented on KAFKA-1450: Created reviewboard https://reviews.apache.org/r/21428/

[jira] [Updated] (KAFKA-1451) Broker stuck due to leader election race

[ https://issues.apache.org/jira/browse/KAFKA-1451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciek Makowski updated KAFKA-1451: --- Description: h3. Symptoms The broker does not become available due to being stuck in an infi

Re: Review Request 21398: Fix KAFKA-1445 v2

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/21398/#review42982 --- clients/src/main/java/org/apache/kafka/clients/producer/internals/S

Re: Review Request 21398: Fix KAFKA-1445 v2

> On May 13, 2014, 6:37 p.m., Timothy Chen wrote: > > clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java, > > line 323 > > > > > > Should we only call forceUpdate once if we get multiple unk

[jira] [Commented] (KAFKA-1393) I wrote this conflicted ephemeral node at /brokers/ids/199 a while back in a different session, hence I will backoff for this node to be deleted by Zookeeper and retry

[ https://issues.apache.org/jira/browse/KAFKA-1393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13996052#comment-13996052 ] Yongkun Wang commented on KAFKA-1393: - Check whether have set different broker.id for

[jira] [Updated] (KAFKA-1445) New Producer should send all partitions that have non-empty batches when on of them is ready

[ https://issues.apache.org/jira/browse/KAFKA-1445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-1445: - Attachment: KAFKA-1445_2014-05-13_11:25:13.patch > New Producer should send all partitions that h

Re: Review Request 21428: Patch for KAFKA-1450

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/21428/#review42967 --- core/src/main/scala/kafka/api/FetchRequest.scala

[jira] [Created] (KAFKA-1450) check invalid leader in a more robust way

Jun Rao created KAFKA-1450: -- Summary: check invalid leader in a more robust way Key: KAFKA-1450 URL: https://issues.apache.org/jira/browse/KAFKA-1450 Project: Kafka Issue Type: Bug Affects Versi

Re: kafka performance question

Hi Zhujie, When you use a single thread to read data from disk and send to Kafka server, which client/API did you use? Guozhang On Tue, May 13, 2014 at 6:32 PM, Zhujie (zhujie, Smartcare) < first.zhu...@huawei.com> wrote: > our version is kafka_2.10-0.8.1 > > 发件人: Zhujie (zhujie, Smartcare) >

[jira] [Updated] (KAFKA-1451) Broker stuck due to leader election race

[ https://issues.apache.org/jira/browse/KAFKA-1451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciek Makowski updated KAFKA-1451: --- Description: h3. Symptoms The broker does not become available due to being stuck in an infi

Re: How can I step through the Kafka code using a debugger

Tim, Thanks for the reply! I have followed the IDE setup instructions and tried using IntelliJ to build Kafka. I used java 1.7.0_55-b13 on Mac OS X Mavericks. The "gradlew jar" commands etc. worked fine. However, I cannot use IntelliJ to build the project. I am using IntelliJ 13.1 to build the p

[jira] [Updated] (KAFKA-1445) New Producer should send all partitions that have non-empty batches when on of them is ready

[ https://issues.apache.org/jira/browse/KAFKA-1445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-1445: - Attachment: KAFKA-1445.patch > New Producer should send all partitions that have non-empty batche

[jira] [Commented] (KAFKA-1447) Controlled shutdown deadlock when trying to send state updates

[ https://issues.apache.org/jira/browse/KAFKA-1447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13997524#comment-13997524 ] Sam Meder commented on KAFKA-1447: -- We'll be rolling out 0.8.1.1 soon, but since we can't

[jira] [Created] (KAFKA-1451) Broker stuck due to leader election race

Maciek Makowski created KAFKA-1451: -- Summary: Broker stuck due to leader election race Key: KAFKA-1451 URL: https://issues.apache.org/jira/browse/KAFKA-1451 Project: Kafka Issue Type: Bug

[jira] [Commented] (KAFKA-1316) Refactor Sender

[ https://issues.apache.org/jira/browse/KAFKA-1316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13996687#comment-13996687 ] Jun Rao commented on KAFKA-1316: Some of the failure handling seems to be common btw the p

[jira] [Updated] (KAFKA-1451) Broker stuck due to leader election race

[ https://issues.apache.org/jira/browse/KAFKA-1451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maciek Makowski updated KAFKA-1451: --- Description: h3. Symptoms The broker does not become available, due to being stuck in an inf

[jira] [Updated] (KAFKA-1450) check invalid leader in a more robust way

[ https://issues.apache.org/jira/browse/KAFKA-1450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-1450: --- Attachment: KAFKA-1450.patch > check invalid leader in a more robust way > --

Review Request 21428: Patch for KAFKA-1450

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/21428/ --- Review request for kafka. Bugs: KAFKA-1450 https://issues.apache.org/jira/b

[jira] Subscription: outstanding kafka patches

Issue Subscription Filter: outstanding kafka patches (90 issues) The list of outstanding kafka patches Subscriber: kafka-mailing-list Key Summary KAFKA-1445 New Producer should send all partitions that have non-empty batches when on of them is ready https://issues.apache.org/

[jira] [Comment Edited] (KAFKA-1393) I wrote this conflicted ephemeral node at /brokers/ids/199 a while back in a different session, hence I will backoff for this node to be deleted by Zookeeper and

[ https://issues.apache.org/jira/browse/KAFKA-1393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13996052#comment-13996052 ] Yongkun Wang edited comment on KAFKA-1393 at 5/13/14 4:51 AM: --

Max Message Size

Hi Kafka Team, Is there any message size limitation from producer side ? If there, is what happens to message, does it get truncated or message is lost ? Thanks, Bhavesh

Re: How can I step through the Kafka code using a debugger

Hi Sheng Wang, If you look at the Kafka website there is already a IDE setup page you can follow. Once your IDE is setup you can simply run Kafka via the Kafka main method and set breakpoints in the broker. Tim On Tue, May 13, 2014 at 11:44 PM, Sheng Wang wrote: > Hi, > > I want to learn more