Re: [DISCUSS] Add missing API to old high level consumer

I thought we wanted security on 0.8.3 too... the SSL + Authz patches seem close to ready, no? On Fri, May 15, 2015 at 3:56 AM, Joe Stein wrote: > Hey Becket, yeah good point. Officially there is no 0.8.3 > https://cwiki.apache.org/confluence/display/KAFKA/Future+release+plan > release > planned.

[jira] [Commented] (KAFKA-1690) new java producer needs ssl support as a client

[ https://issues.apache.org/jira/browse/KAFKA-1690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14544793#comment-14544793 ] Sriharsha Chintalapani commented on KAFKA-1690: --- [~junrao] In SASL lets cons

Re: Review Request 33620: Patch for KAFKA-1690

> On May 14, 2015, 10:21 a.m., Rajini Sivaram wrote: > > clients/src/main/java/org/apache/kafka/common/network/Authenticator.java, > > line 47 > > > > > > Don't think authenticator interface should handle selection >

Re: [DISCUSS] Add missing API to old high level consumer

Hey Becket, yeah good point. Officially there is no 0.8.3 https://cwiki.apache.org/confluence/display/KAFKA/Future+release+plan release planned. I agree we should have the new consumer beta and a patch for the old one. If we do that in 0.8.3 that makes good sense, yup. We should also include https

Re: Can we throw away "ResponsesBeingSent" Gauge?

+1 for dropping ResponsesBeingSent. Thanks, Jun On Thu, May 14, 2015 at 4:38 PM, Jiangjie Qin wrote: > I think we probably can drop this completely. The new selector should have > provided enough metrics to describe the IO thread status. And it is async > anyway, so ³being sent² does not look

[jira] [Updated] (KAFKA-1928) Move kafka.network over to using the network classes in org.apache.kafka.common.network

[ https://issues.apache.org/jira/browse/KAFKA-1928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-1928: Attachment: KAFKA-1928_2015-05-15_03.patch > Move kafka.network over to using the network classes in

Re: Review Request 33065: Patch for KAFKA-1928

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33065/ --- (Updated May 15, 2015, 12:07 a.m.) Review request for kafka. Changes ---

[jira] [Updated] (KAFKA-1928) Move kafka.network over to using the network classes in org.apache.kafka.common.network

[ https://issues.apache.org/jira/browse/KAFKA-1928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-1928: Attachment: (was: KAFKA-1928_2015-05-15_03%3A05%3A37.patch.patch) > Move kafka.network over to u

[jira] [Updated] (KAFKA-1928) Move kafka.network over to using the network classes in org.apache.kafka.common.network

[ https://issues.apache.org/jira/browse/KAFKA-1928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-1928: Attachment: KAFKA-1928_2015-05-15_03%3A05%3A37.patch.patch > Move kafka.network over to using the ne

Re: [DISCUSS] Add missing API to old high level consumer

Hey Joe, Actually this API was asked for before, and we have several use cases in LinkedIn as well. I thought we have added that in KAFKA-1650 but obviously I forgot to do that. My understanding is that we won¹t really deprecate high level consumer until we move to 0.9.0. So we can have this API

Re: Can we throw away "ResponsesBeingSent" Gauge?

I think we probably can drop this completely. The new selector should have provided enough metrics to describe the IO thread status. And it is async anyway, so ³being sent² does not look really helpful. Jiangjie (Becket) Qin On 5/14/15, 1:06 PM, "Todd Palino" wrote: >I don't believe we're using

[jira] [Commented] (KAFKA-2190) Incorporate close(timeout) to Mirror Maker

[ https://issues.apache.org/jira/browse/KAFKA-2190?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14544610#comment-14544610 ] Jiangjie Qin commented on KAFKA-2190: - Created reviewboard https://reviews.apache.org/

[jira] [Updated] (KAFKA-2190) Incorporate close(timeout) to Mirror Maker

[ https://issues.apache.org/jira/browse/KAFKA-2190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiangjie Qin updated KAFKA-2190: Attachment: KAFKA-2190.patch > Incorporate close(timeout) to Mirror Maker >

[jira] [Updated] (KAFKA-2190) Incorporate close(timeout) to Mirror Maker

[ https://issues.apache.org/jira/browse/KAFKA-2190?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiangjie Qin updated KAFKA-2190: Status: Patch Available (was: Open) > Incorporate close(timeout) to Mirror Maker >

Review Request 34241: Patch for KAFKA-2190

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34241/ --- Review request for kafka. Bugs: KAFKA-2190 https://issues.apache.org/jira/b

Re: Review Request 33065: Patch for KAFKA-1928

> On May 6, 2015, 10:50 p.m., Jun Rao wrote: > > clients/src/main/java/org/apache/kafka/common/network/Selector.java, lines > > 311-315 > > > > > > In this case, we should probably just propagate the exception and >

Jenkins build is back to normal : KafkaPreCommit #101

See

Re: Review Request 33065: Patch for KAFKA-1928

> On May 6, 2015, 10:50 p.m., Jun Rao wrote: > > clients/src/main/java/org/apache/kafka/common/network/Selector.java, lines > > 311-315 > > > > > > In this case, we should probably just propagate the exception and >

Re: Review Request 33065: Patch for KAFKA-1928

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33065/ --- (Updated May 14, 2015, 9:30 p.m.) Review request for kafka. Changes ---

Re: Review Request 33065: Patch for KAFKA-1928

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33065/#review83841 --- core/src/main/scala/kafka/network/SocketServer.scala

Re: Review Request 33065: Patch for KAFKA-1928

> On May 6, 2015, 10:50 p.m., Jun Rao wrote: > > clients/src/main/java/org/apache/kafka/common/network/Selector.java, lines > > 311-315 > > > > > > In this case, we should probably just propagate the exception and >

Re: Writing a client: Connection pooling

The C# client (kafka-net) isn't written by me; I'm just working on it. It has a separate producer and consumer, but the client is designed to connect to each broker exactly once and then reuse (multiplex over) that one connection across all consumers and producers. It's a bit disappointing to see

Build failed in Jenkins: Kafka-trunk #489

See Changes: [wangguoz] KAFKA-1334; Add the heartbeat logic to consumer coordinator; reviewed by Guozhang Wang -- [...truncated 2413 lines...] at junit.framework.Assert.assertEquals(Assert.ja

Re: Review Request 33088: add heartbeat to coordinator

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33088/#review83851 --- Ship it! Ship It! - Guozhang Wang On May 14, 2015, 7:38 a.m., On

Re: Review Request 33065: Patch for KAFKA-1928

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33065/ --- (Updated May 14, 2015, 9:51 p.m.) Review request for kafka. Changes ---

[jira] [Updated] (KAFKA-1334) Add failure detection capability to the coordinator / consumer

[ https://issues.apache.org/jira/browse/KAFKA-1334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-1334: - Status: In Progress (was: Patch Available) > Add failure detection capability to the coordinator

[jira] [Commented] (KAFKA-1334) Add failure detection capability to the coordinator / consumer

[ https://issues.apache.org/jira/browse/KAFKA-1334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14544431#comment-14544431 ] Guozhang Wang commented on KAFKA-1334: -- Committed the first draft patch to trunk. Lea

[jira] [Updated] (KAFKA-1928) Move kafka.network over to using the network classes in org.apache.kafka.common.network

[ https://issues.apache.org/jira/browse/KAFKA-1928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-1928: Attachment: (was: KAFKA-1928_2015-05-15.patch) > Move kafka.network over to using the network cl

[jira] [Updated] (KAFKA-1928) Move kafka.network over to using the network classes in org.apache.kafka.common.network

[ https://issues.apache.org/jira/browse/KAFKA-1928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-1928: Attachment: KAFKA-1928_2015-05-15.patch > Move kafka.network over to using the network classes in >

[jira] [Updated] (KAFKA-1928) Move kafka.network over to using the network classes in org.apache.kafka.common.network

[ https://issues.apache.org/jira/browse/KAFKA-1928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira updated KAFKA-1928: Attachment: KAFKA-1928_2015-05-15.patch > Move kafka.network over to using the network classes in >

[jira] [Updated] (KAFKA-2193) Intermittent network + DNS issues can cause brokers to permanently drop out of a cluster

[ https://issues.apache.org/jira/browse/KAFKA-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom Lee updated KAFKA-2193: --- Description: Our Kafka cluster recently experienced some intermittent network & DNS resolution issues such th

[jira] [Created] (KAFKA-2193) Intermittent network + DNS issues can cause brokers to permanently drop out of a cluster

Tom Lee created KAFKA-2193: -- Summary: Intermittent network + DNS issues can cause brokers to permanently drop out of a cluster Key: KAFKA-2193 URL: https://issues.apache.org/jira/browse/KAFKA-2193 Project: K

Re: Can we throw away "ResponsesBeingSent" Gauge?

I don't believe we're using the ResponsesBeingSent information at all. I know we use NetworkProcessorAvgIdlePercent to keep track of the utilization of the pool. -Todd On Thu, May 14, 2015 at 11:36 AM, Joel Koshy wrote: > I'm also not sure how useful it is, but there is some discussion on it >

Re: [VOTE] KIP-4 Admin Commands / Phase-1

+1 On Thu, May 07, 2015 at 04:25:06PM -0700, Jun Rao wrote: > +1 on the vote. > > Thanks, > > Jun > > On Thu, May 7, 2015 at 8:29 AM, Andrii Biletskyi < > andrii.bilets...@stealth.ly> wrote: > > > Jun, > > > > 1. My bad, updated description, but forgot the schema itself. Fixed. > > > > 2. Okay

[jira] [Created] (KAFKA-2192) Async producer is not async when not connected

Erik created KAFKA-2192: --- Summary: Async producer is not async when not connected Key: KAFKA-2192 URL: https://issues.apache.org/jira/browse/KAFKA-2192 Project: Kafka Issue Type: Bug Componen

Re: Can we throw away "ResponsesBeingSent" Gauge?

I'm also not sure how useful it is, but there is some discussion on it here: https://issues.apache.org/jira/browse/KAFKA-1597 On Thu, May 14, 2015 at 09:26:02PM +0300, Gwen Shapira wrote: > Hi, > > As part of KAFKA-1928, we need to consolidate existing SocketServer metrics > into Selector metric

Can we throw away "ResponsesBeingSent" Gauge?

Hi, As part of KAFKA-1928, we need to consolidate existing SocketServer metrics into Selector metrics. The two metrics are: 1. NetworkProcessorAvgIdlePercent 2. ResponsesBeingSent NetworkProcessorAvgIdlePercent is currently implemented as Meter, which seems to translate nicely into KafkaMetrics

Re: Review Request 33065: Patch for KAFKA-1928

> On May 6, 2015, 10:50 p.m., Jun Rao wrote: > > core/src/main/scala/kafka/network/SocketServer.scala, line 455 > > > > > > This is just registering for reads, not for writes, right? Maybe I misunderstood, but it look

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

For ListTopics, we decided not to add a ListTopics request for now and just rely on passing in an empty list to TMR. We can revisit this in the future if it becomes an issue. Thanks, Jun On Wed, May 13, 2015 at 3:31 PM, Joel Koshy wrote: > Just had a few minor questions before I join the vote

Re: Review Request 34170: Patch for KAFKA-2191

> On May 13, 2015, 5:14 p.m., Jay Kreps wrote: > > clients/src/main/java/org/apache/kafka/common/metrics/stats/Rate.java, line > > 62 > > > > > > Is this actually right? I agree you'll get discontinuities as the > > m

[jira] [Commented] (KAFKA-1690) new java producer needs ssl support as a client

[ https://issues.apache.org/jira/browse/KAFKA-1690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14544028#comment-14544028 ] Jun Rao commented on KAFKA-1690: [~sriharsha], what about SASL? So we also rely on restart

Re: Review Request 33620: Patch for KAFKA-1690

> On May 13, 2015, 3:40 p.m., Jun Rao wrote: > > clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java, > > lines 61-112 > > > > > > Those properties will be used by broker config as well, right? Shou

[jira] [Commented] (KAFKA-2147) Unbalanced replication can cause extreme purgatory growth

[ https://issues.apache.org/jira/browse/KAFKA-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14543986#comment-14543986 ] Jun Rao commented on KAFKA-2147: I added a bit more instrumentation. Could you apply the p

[jira] [Updated] (KAFKA-2147) Unbalanced replication can cause extreme purgatory growth

[ https://issues.apache.org/jira/browse/KAFKA-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-2147: --- Attachment: KAFKA-2147_2015-05-14_09:41:56.patch > Unbalanced replication can cause extreme purgatory growth >

Re: Review Request 34125: Patch for KAFKA-2147

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34125/ --- (Updated May 14, 2015, 4:42 p.m.) Review request for kafka. Bugs: KAFKA-2147

[jira] [Commented] (KAFKA-2147) Unbalanced replication can cause extreme purgatory growth

[ https://issues.apache.org/jira/browse/KAFKA-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14543979#comment-14543979 ] Jun Rao commented on KAFKA-2147: Updated reviewboard https://reviews.apache.org/r/34125/di

Re: Review Request 33620: Patch for KAFKA-1690

> On May 14, 2015, 10:21 a.m., Rajini Sivaram wrote: > > clients/src/main/java/org/apache/kafka/common/network/SSLTransportLayer.java, > > line 190 > > > > > > I think when delegated tasks are run asynchronously, sele

Re: Review Request 33620: Patch for KAFKA-1690

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33620/#review83777 --- clients/src/main/java/org/apache/kafka/common/network/SSLFactory.ja

[jira] [Commented] (KAFKA-2147) Unbalanced replication can cause extreme purgatory growth

[ https://issues.apache.org/jira/browse/KAFKA-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14543761#comment-14543761 ] Evan Huus commented on KAFKA-2147: -- OK, it appears the patch does not fix this issue, we

Re: Review Request 33620: Patch for KAFKA-1690

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33620/#review83767 --- clients/src/test/java/org/apache/kafka/common/network/SSLFactoryTes

Re: Review Request 33620: Patch for KAFKA-1690

> On May 14, 2015, 10:21 a.m., Rajini Sivaram wrote: > > clients/src/test/java/org/apache/kafka/common/network/SSLSelectorTest.java, > > line 36 > > > > > > All these tests were hanging when I ran them from a Windows m

Re: Review Request 33620: Patch for KAFKA-1690

> On May 14, 2015, 10:21 a.m., Rajini Sivaram wrote: > > clients/src/main/java/org/apache/kafka/common/network/SSLTransportLayer.java, > > lines 268-280 > > > > > > Graceful close needs close handshake messages to be

Re: Review Request 33620: Patch for KAFKA-1690

> On May 14, 2015, 10:21 a.m., Rajini Sivaram wrote: > > clients/src/main/java/org/apache/kafka/common/network/Authenticator.java, > > line 47 > > > > > > Don't think authenticator interface should handle selection >

[jira] [Commented] (KAFKA-2147) Unbalanced replication can cause extreme purgatory growth

[ https://issues.apache.org/jira/browse/KAFKA-2147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14543666#comment-14543666 ] Evan Huus commented on KAFKA-2147: -- Increasing the producer batch size has substantially

Re: Review Request 33620: Patch for KAFKA-1690

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33620/#review83741 --- checkstyle/import-control.xml

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations (Thread 2)

Joel, - DecreasePartitionNotAllowed. Yeah, that's kind of subcase of InvalidPartitions... But since client can always request topic metadata and check what exactly is was wrong with Partitions argument, I think, yes, we can remove DecreasePartitionNotAllowed and use InvalidPartitions instead. I'll

Re: Review Request 33088: add heartbeat to coordinator

> On May 7, 2015, 10:07 p.m., Guozhang Wang wrote: > > core/src/main/scala/kafka/coordinator/CoordinatorMetadata.scala, line 53 > > > > > > Add @nonthreadsafe for this function. > > Onur Karaman wrote: > It would

[jira] [Updated] (KAFKA-1334) Add failure detection capability to the coordinator / consumer

[ https://issues.apache.org/jira/browse/KAFKA-1334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Onur Karaman updated KAFKA-1334: Attachment: KAFKA-1334_2015-05-14_00:38:39.patch > Add failure detection capability to the coordinat

[jira] [Commented] (KAFKA-1334) Add failure detection capability to the coordinator / consumer

[ https://issues.apache.org/jira/browse/KAFKA-1334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1454#comment-1454 ] Onur Karaman commented on KAFKA-1334: - Updated reviewboard https://reviews.apache.org/

Re: Review Request 33088: add heartbeat to coordinator

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/33088/ --- (Updated May 14, 2015, 7:38 a.m.) Review request for kafka. Bugs: KAFKA-1334

[jira] [Updated] (KAFKA-2191) Measured rate should not be infinite

[ https://issues.apache.org/jira/browse/KAFKA-2191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Lin updated KAFKA-2191: Attachment: KAFKA-2191_2015-05-14_00:34:30.patch > Measured rate should not be infinite > --

[jira] [Commented] (KAFKA-2191) Measured rate should not be infinite

[ https://issues.apache.org/jira/browse/KAFKA-2191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14543329#comment-14543329 ] Dong Lin commented on KAFKA-2191: - Updated reviewboard https://reviews.apache.org/r/34170/

Re: Review Request 34170: Patch for KAFKA-2191

--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34170/ --- (Updated May 14, 2015, 7:34 a.m.) Review request for kafka. Bugs: KAFKA-2191

Re: Review Request 34170: Patch for KAFKA-2191

> On May 13, 2015, 5:14 p.m., Jay Kreps wrote: > > clients/src/main/java/org/apache/kafka/common/metrics/stats/Rate.java, line > > 62 > > > > > > Is this actually right? I agree you'll get discontinuities as the > > m