[jira] [Comment Edited] (KAFKA-1835) Kafka new producer needs options to make blocking behavior explicit

[ https://issues.apache.org/jira/browse/KAFKA-1835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14519023#comment-14519023 ] Stefan Miklosovic edited comment on KAFKA-1835 at 4/29/15 9:30 AM: -

[jira] [Comment Edited] (KAFKA-1835) Kafka new producer needs options to make blocking behavior explicit

[ https://issues.apache.org/jira/browse/KAFKA-1835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14519023#comment-14519023 ] Stefan Miklosovic edited comment on KAFKA-1835 at 4/29/15 9:31 AM: -

[jira] [Comment Edited] (KAFKA-1835) Kafka new producer needs options to make blocking behavior explicit

[ https://issues.apache.org/jira/browse/KAFKA-1835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14519023#comment-14519023 ] Stefan Miklosovic edited comment on KAFKA-1835 at 4/29/15 9:27 AM: -

[jira] [Comment Edited] (KAFKA-1835) Kafka new producer needs options to make blocking behavior explicit

[ https://issues.apache.org/jira/browse/KAFKA-1835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14519023#comment-14519023 ] Stefan Miklosovic edited comment on KAFKA-1835 at 4/29/15 9:28 AM: -

[jira] [Created] (KAFKA-2155) Add option to control ZK root for kafka.tools.ConsumerOffsetChecker

Kjell Tore Fossbakk created KAFKA-2155: -- Summary: Add option to control ZK root for kafka.tools.ConsumerOffsetChecker Key: KAFKA-2155 URL: https://issues.apache.org/jira/browse/KAFKA-2155 Project

[jira] [Commented] (KAFKA-1835) Kafka new producer needs options to make blocking behavior explicit

[ https://issues.apache.org/jira/browse/KAFKA-1835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14519023#comment-14519023 ] Stefan Miklosovic commented on KAFKA-1835: -- [~dhay] Isn't the only prerequisity

[jira] [Commented] (KAFKA-1835) Kafka new producer needs options to make blocking behavior explicit

[ https://issues.apache.org/jira/browse/KAFKA-1835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14519095#comment-14519095 ] Stefan Miklosovic commented on KAFKA-1835: -- It seems to me it does not matter if

[jira] [Comment Edited] (KAFKA-1835) Kafka new producer needs options to make blocking behavior explicit

[ https://issues.apache.org/jira/browse/KAFKA-1835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14519095#comment-14519095 ] Stefan Miklosovic edited comment on KAFKA-1835 at 4/29/15 10:21 AM:

[jira] [Commented] (KAFKA-1835) Kafka new producer needs options to make blocking behavior explicit

[ https://issues.apache.org/jira/browse/KAFKA-1835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14519105#comment-14519105 ] Stefan Miklosovic commented on KAFKA-1835: -- what I am basically able to achieve b

[jira] [Commented] (KAFKA-1387) Kafka getting stuck creating ephemeral node it has already created when two zookeeper sessions are established in a very short period of time

[ https://issues.apache.org/jira/browse/KAFKA-1387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14519189#comment-14519189 ] Marcus Aidley commented on KAFKA-1387: -- I've also encountered this issue running Kafk

[jira] [Created] (KAFKA-2156) Possibility to plug in custom MetricRegistry

Andras Sereny created KAFKA-2156: Summary: Possibility to plug in custom MetricRegistry Key: KAFKA-2156 URL: https://issues.apache.org/jira/browse/KAFKA-2156 Project: Kafka Issue Type: Improv

Re: [DISCUSSION] java.io.Closeable in KAFKA-2121

If you do this, the code is no longer simple, which defeats the benefit of extending Closeable. We can define our own Closeable that doesn't throw exceptions, but it may be confusing. So, it seems the original code is probably better. Thanks, Jun On Tue, Apr 28, 2015 at 3:11 PM, Steven Wu wrote

Re: Perf testing flush() - issues found

Note that the batch size is per partition. The more partitions you have, the longer it will take to fill up all partitions with the same batch size. So, you probably need to increase the linger time such that in dependent of the number of partitions, the configured batch size can be reached. There

Re: Review Request 33125: Add comment to timing fix

> On April 25, 2015, 5:11 a.m., Guozhang Wang wrote: > > clients/src/test/java/org/apache/kafka/clients/MetadataTest.java, line 58 > > > > > > Do we need this sleep? Guozhang, Sorry, had missed this note. The sleep avo

Re: Adding multi-tenancy capabilities to Kafka

Thanks for your response. I agree. I think it would be useful to get consensus on how namespaces and topic-hierarchies relate to one another. To seed the discussion - here's my viewpoint, which I hope others will challenge: I see namespaces as something the broker provides to ensure that two te

[jira] [Comment Edited] (KAFKA-1835) Kafka new producer needs options to make blocking behavior explicit

[ https://issues.apache.org/jira/browse/KAFKA-1835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14518184#comment-14518184 ] David Hay edited comment on KAFKA-1835 at 4/29/15 3:00 PM: --- [~sm

[jira] [Commented] (KAFKA-1835) Kafka new producer needs options to make blocking behavior explicit

[ https://issues.apache.org/jira/browse/KAFKA-1835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14519506#comment-14519506 ] David Hay commented on KAFKA-1835: -- Fixed some syntax errors in my code sample. This exa

Re: [DISCUSS] KIP-12 - Kafka Sasl/Kerberos implementation

Harsha, I like the separation of transport layer and authentication. I have a few questions (maybe it will be clearer when the code is ready for review) 1. I wasn't sure what the "security protocol" in the endpoint now stands for. Is that the transport protocol? Or is that really a choice

Re: ZooKeeper Session Timeout During Startup Caused By Long ParNewGC Pauses

Here is what the ParNewGC pauses look like in the Kafka logs in one of our environments (earlier I provided logs from the test program). In this case I had already increased the ZooKeeper session timeout to 20 seconds so that it did not trigger a communication failure while waiting for GC to fi

Re: [DISCUSSION] java.io.Closeable in KAFKA-2121

Jun, we still get the benefit of extending Closeable. e.g. Utils.closeQuietly() can take FooSerializer as an argument. we can avoid the duplication of boiler-plate code. class FooSerializer implements Serializer { @Override public void close() { // may throw unchecked RuntimeExce

Re: [DISCUSS] KIP-12 - Kafka Sasl/Kerberos implementation

I wasn't sure what the "security protocol" in the endpoint now stands for. Is that the transport protocol? Or is that really a choice between secure/insecure with parameters for secure transport specified separately? Would inter-broker transport properties be configurable?            My thinking

[jira] [Created] (KAFKA-2157) kafka-console-consumer.sh: Mismatch in CLI "usage" docs vs. Scala Option parsing

Thomas Vaughan created KAFKA-2157: - Summary: kafka-console-consumer.sh: Mismatch in CLI "usage" docs vs. Scala Option parsing Key: KAFKA-2157 URL: https://issues.apache.org/jira/browse/KAFKA-2157 Proj

[jira] [Commented] (KAFKA-2157) kafka-console-consumer.sh: Mismatch in CLI "usage" docs vs. Scala Option parsing

[ https://issues.apache.org/jira/browse/KAFKA-2157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14519612#comment-14519612 ] Thomas Vaughan commented on KAFKA-2157: --- I'm happy to submit a patch for this, but w

Re: Perf testing flush() - issues found

Just want to confirm that when you say batch.size and number of records will be equal you don't mean that literally. The batch.size is in bytes so if you wanted a batch size of 16 1k messages for a single partition then you are setting batch.size=16*1024. -Jay On Tue, Apr 28, 2015 at 5:58 PM, Ros

Re: Review Request 33634: Patch for KAFKA-2129

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33634/#review81972 --- Ship it! Ship It! - Jay Kreps On April 28, 2015, 6:12 p.m., Tim

[jira] [Commented] (KAFKA-1660) Ability to call close() with a timeout on the Java Kafka Producer.

[ https://issues.apache.org/jira/browse/KAFKA-1660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14519657#comment-14519657 ] Jay Kreps commented on KAFKA-1660: -- Hey [~becket_qin], I went back and looked at version

Re: Perf testing flush() - issues found

Just to add to Jun's suggestion: 1. since the batch.size config is per-partition, with for example 4K messages * 1K message size between flush() and batch.size set to 4Mb, then with 8 partition, by the time of flush() each partition will get 0.5Mb only, meaning you may not be batching sufficiently

Re: [DISCUSSION] java.io.Closeable in KAFKA-2121

I think I agree with Jay / Steven, that we still get the benefit of extending Closeable and overriding close(). Guozhang On Wed, Apr 29, 2015 at 8:36 AM, Steven Wu wrote: > Jun, > > we still get the benefit of extending Closeable. e.g. Utils.closeQuietly() > can take FooSerializer as an argumen

Re: [DISCUSS] KIP-12 - Kafka Sasl/Kerberos implementation

Just to make sure we are on the same page: After yesterday's call, it sounded like we have two orthogonal decisions: 1) Do we want the channel to implement SSL or PLAINTEXT for communication 2) Do we want to authenticate with Kerberos or not? I didn't think of this distinction, but it makes perfe

Re: [DISCUSS] KIP-12 - Kafka Sasl/Kerberos implementation

Hi Gwen,      Your understanding is right :) , sorry about the confusion. " If I understand your new suggestion correctly, we will control all options  as separate ports. i.e. a port for each of those:  * SSL non-authenticated (or authenticated with SSL keys)  * SSL + SASL (which is what Hadoop doe

Re: [DISCUSS] KIP-12 - Kafka Sasl/Kerberos implementation

Perfect. Can't wait to review the patch :) On Wed, Apr 29, 2015 at 10:14 AM, Sriharsha Chintalapani wrote: > Hi Gwen, > Your understanding is right :) , sorry about the confusion. > " If I understand your new suggestion correctly, we will control all > options > > as separate ports. i.e. a

[jira] [Commented] (KAFKA-2156) Possibility to plug in custom MetricRegistry

[ https://issues.apache.org/jira/browse/KAFKA-2156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14519813#comment-14519813 ] Aditya Auradkar commented on KAFKA-2156: [~junrao] Do we want to support this chan

[jira] [Commented] (KAFKA-1660) Ability to call close() with a timeout on the Java Kafka Producer.

[ https://issues.apache.org/jira/browse/KAFKA-1660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14520141#comment-14520141 ] Jiangjie Qin commented on KAFKA-1660: - Hi [~jkreps], I prefer the counter as well beca

[jira] [Comment Edited] (KAFKA-1835) Kafka new producer needs options to make blocking behavior explicit

[ https://issues.apache.org/jira/browse/KAFKA-1835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14520168#comment-14520168 ] Stefan Miklosovic edited comment on KAFKA-1835 at 4/29/15 8:34 PM: -

[jira] [Commented] (KAFKA-1835) Kafka new producer needs options to make blocking behavior explicit

[ https://issues.apache.org/jira/browse/KAFKA-1835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14520168#comment-14520168 ] Stefan Miklosovic commented on KAFKA-1835: -- [~dhay] thanks for your correction.

[jira] [Commented] (KAFKA-1660) Ability to call close() with a timeout on the Java Kafka Producer.

[ https://issues.apache.org/jira/browse/KAFKA-1660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14520277#comment-14520277 ] Joel Koshy commented on KAFKA-1660: --- [~becket_qin] and I discussed an alternate fix whic

[jira] [Commented] (KAFKA-1660) Ability to call close() with a timeout on the Java Kafka Producer.

[ https://issues.apache.org/jira/browse/KAFKA-1660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14520341#comment-14520341 ] Joel Koshy commented on KAFKA-1660: --- (There also needs to be a {{batch.done()}} in the i

Re: Review Request 33614: Patch for KAFKA-2132

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33614/#review82056 --- Overall, looks good. I had a bunch of nits :) build.gradle

[jira] [Commented] (KAFKA-1660) Ability to call close() with a timeout on the Java Kafka Producer.

[ https://issues.apache.org/jira/browse/KAFKA-1660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14520557#comment-14520557 ] Jiangjie Qin commented on KAFKA-1660: - Updated reviewboard https://reviews.apache.org/

Re: Review Request 31850: Patch for KAFKA-1660

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31850/ --- (Updated April 29, 2015, 11:58 p.m.) Review request for kafka. Bugs: KAFKA-16

[jira] [Updated] (KAFKA-1660) Ability to call close() with a timeout on the Java Kafka Producer.

[ https://issues.apache.org/jira/browse/KAFKA-1660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiangjie Qin updated KAFKA-1660: Attachment: KAFKA-1660_2015-04-29_16:58:43.patch > Ability to call close() with a timeout on the Jav

[jira] [Commented] (KAFKA-1660) Ability to call close() with a timeout on the Java Kafka Producer.

[ https://issues.apache.org/jira/browse/KAFKA-1660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14520619#comment-14520619 ] Jiangjie Qin commented on KAFKA-1660: - Updated reviewboard https://reviews.apache.org/

Re: Review Request 31850: Patch for KAFKA-1660

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31850/ --- (Updated April 30, 2015, 12:37 a.m.) Review request for kafka. Bugs: KAFKA-16

[jira] [Updated] (KAFKA-1660) Ability to call close() with a timeout on the Java Kafka Producer.

[ https://issues.apache.org/jira/browse/KAFKA-1660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiangjie Qin updated KAFKA-1660: Attachment: KAFKA-1660_2015-04-29_17:37:51.patch > Ability to call close() with a timeout on the Jav

Re: Review Request 33645: Patch for KAFKA-2153

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33645/#review82076 --- Thanks for fixing the review tool! It looks like some changes to te

Re: [DISCUSS] KIP-21 Configuration Management

Hi Aditya, Thanks for the write-up. I like the idea of marking specific configurations as dynamic, I think this is just what we need. Few comments: 1. I have deep concerns about managing configuration in ZooKeeper. First, Producers and Consumers shouldn't depend on ZK at all, this seems

[jira] [Commented] (KAFKA-2147) Unbalanced replication can cause extreme purgatory growth

[ https://issues.apache.org/jira/browse/KAFKA-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14520747#comment-14520747 ] Jun Rao commented on KAFKA-2147: If the replica fetch request is fetching many partitions,

Re: Review Request 33645: Patch for KAFKA-2153

> On April 30, 2015, 12:50 a.m., Gwen Shapira wrote: > > Thanks for fixing the review tool! > > > > It looks like some changes to test-patch.py are included. Can you clean > > them out if they are not an intentional part of this patch? They are intentional :). I am reusing the part of logic al

Re: Perf testing flush() - issues found

@Jay, My bad. I mistook the batch.size to be number of messages instead of bytes. Below are revised measurements based on computing the batch.size in bytes . @Jun, With explicit flush()... linger should not impact. Isn't it ? @Wang, Larger batches are not necessarily giving better numbe

Re: Perf testing flush() - issues found

For some reason the HTML formatting is being dropped from my email.. Making it harder to read the measurements table. On 4/29/15 8:32 PM, "Roshan Naik" wrote: > >@Jay, >My bad. I mistook the batch.size to be number of messages instead of >bytes. Below are revised measurements based on computing

[jira] [Commented] (KAFKA-1387) Kafka getting stuck creating ephemeral node it has already created when two zookeeper sessions are established in a very short period of time

[ https://issues.apache.org/jira/browse/KAFKA-1387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14520867#comment-14520867 ] Thomas Omans commented on KAFKA-1387: - It looks like this "infinite retry" behavior is

Re: Perf testing flush() - issues found

Roshan, The client allocates a batch per partition and has a hard cap on memory usage (default 32MB). When it hits that cap it waits for in-flight requests to complete to use their memory. Setting the batch size to 20M is not good--that means each partition has a 20MB array allocated for it. This