[VOTE] KIP-223 - Add per-topic min lead and per-partition lead metrics to KafkaConsumer

Hi Guozhang, Correct me if I am wrong. Of course I could construct the sensor where per-partition metrics reside as a child of the client-level "records-lead-min", but it seems that whether doing that way takes no effects on what this KIP gonna do, so is it a must? __

RE: [DISCUSS]KIP-235 DNS alias and secured connections

True, amended the KIP, thanks. Jonathan Skrzypek Middleware Engineering Messaging Engineering Goldman Sachs International -Original Message- From: Tom Bentley [mailto:t.j.bent...@gmail.com] Sent: 05 December 2017 18:19 To: dev@kafka.apache.org Subject: Re: [DISCUSS]KIP-235 DNS alias an

RE: [DISCUSS]KIP-235 DNS alias and secured connections

Hi Jonathan I think this will be very useful. I reported something similar here : https://issues.apache.org/jira/browse/KAFKA-4781 Please confirm your kip will address it ? Stéphane On 6 Dec. 2017 8:20 pm, "Skrzypek, Jonathan" wrote: > True, amended the KIP, thanks. > > Jonathan Skrzypek > Mi

[jira] [Created] (KAFKA-6314) Add a tool to delete consumer offsets for a given group

Tom Scott created KAFKA-6314: Summary: Add a tool to delete consumer offsets for a given group Key: KAFKA-6314 URL: https://issues.apache.org/jira/browse/KAFKA-6314 Project: Kafka Issue Type: New

Re: [DISCUSS]KIP-235 DNS alias and secured connections

Hi Jonathan, Thank you for the KIP. I think you are proposing that we always do this (i.e. no option to turn it off). If you have a private and public DNS name, at the moment, if SSL certs and keytabs contain the only public DNS name and the bootstrap servers and advertised listeners are configur

Re: [DISCUSS]KIP-235 DNS alias and secured connections

Sorry, the example I used with public/private DNS is wrong. But there is the general issue of multiple DNS names where only one is added to the keytab/certificate, but all names are added to the bootstrap server list. On Wed, Dec 6, 2017 at 12:06 PM, Rajini Sivaram wrote: > Hi Jonathan, > > Than

Re: [DISCUSS] KIP-234: add support for getting topic defaults from AdminClient

Hi Dan, Thank you for the KIP. KIP-226 (https://cwiki.apache.org/ confluence/display/KAFKA/KIP-226+-+Dynamic+Broker+Configuration) proposes to add an option to include all synonyms of a config option when describing configs. This includes any defaults. For example (using Dong's example), if you ha

[jira] [Created] (KAFKA-6315) kafka-console-consumer.sh terminating consumer process only for one partition with java.lang.IllegalArgumentException: Unknown compression type id: 5

VinayKumar created KAFKA-6315: - Summary: kafka-console-consumer.sh terminating consumer process only for one partition with java.lang.IllegalArgumentException: Unknown compression type id: 5 Key: KAFKA-6315 URL: http

[jira] [Created] (KAFKA-6316) Kafka Rebalancing issues when I kill one consumer

Krishan created KAFKA-6316: -- Summary: Kafka Rebalancing issues when I kill one consumer Key: KAFKA-6316 URL: https://issues.apache.org/jira/browse/KAFKA-6316 Project: Kafka Issue Type: Bug

[GitHub] kafka pull request #4257: KAFKA-6174; Add methods in Options classes to keep...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/4257 ---

[jira] [Resolved] (KAFKA-6174) Add methods in Options classes to keep binary compatibility with 0.11

[ https://issues.apache.org/jira/browse/KAFKA-6174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-6174. Resolution: Fixed Fix Version/s: 1.1.0 > Add methods in Options classes to keep binary compat

Re: [VOTE] KIP-223 - Add per-topic min lead and per-partition lead metrics to KafkaConsumer

Hi Hu Xi, As I said before, it is only a clarification question for its internal implementation; it is not related to the public interfaces. Guozhang On Wed, Dec 6, 2017 at 12:34 AM, Hu Xi wrote: > Hi Guozhang, > > > Correct me if I am wrong. Of course I could construct the sensor where > per

[GitHub] kafka-site pull request #49: Fix bad kafka stream link on use cases page

Github user haoch closed the pull request at: https://github.com/apache/kafka-site/pull/49 ---

[GitHub] kafka-site issue #49: Fix bad kafka stream link on use cases page

Github user haoch commented on the issue: https://github.com/apache/kafka-site/pull/49 Thanks @gwenshap @guozhangwang for merging the PR, so close this PR. ---

Re: [DISCUSS] KIP-210: Provide for custom error handling when Kafka Streams fails to produce

There is already a vote thread for this KIP. I can bump it so that it’s towards the top of your inbox. With regard to your concerns: 1) We do not have the "ProductionExceptionHandler" interface defined in the wiki page, thought it is sort of clear that it is a one-function interface with record a

Re: [VOTE] KIP-210: Provide for custom error handling when Kafka Streams fails to produce

Bumping this thread so it’s visible given that the conversation on KIP-210 has converged again. Current tally is 2 binding +1s, and 2 non-binding +1s. On November 8, 2017 at 12:26:32 PM, Damian Guy (damian@gmail.com) wrote: +1 (binding) On Sat, 4 Nov 2017 at 16:50 Matthias J. Sax wrote: >

[GitHub] kafka pull request #4266: MINOR: Add maybeThrow method to ZooKeeperClient As...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/4266 ---

Re: Flaky healthcheck when trying to check Kafka Stream processing app status

Hi Patrice, I haven't forgotten, just sidetracked with other things. I'll get back to you by the end of the week. Thanks, Bill On Wed, Nov 29, 2017 at 10:36 AM, Bill Bejeck wrote: > Patrice, > > Thanks for reporting this. I'll have a look at what you've posted on > Github. > > Thanks, > Bill

[GitHub] kafka pull request #4296: KAFKA-6313: Add SLF4J as direct dependency to Kafk...

GitHub user rhauch opened a pull request: https://github.com/apache/kafka/pull/4296 KAFKA-6313: Add SLF4J as direct dependency to Kafka core Recent changes are now directly using the SLF4J API, so we should have a direct dependency. ### Committer Checklist (excluded from co

Re: [DISCUSS] KIP-210: Provide for custom error handling when Kafka Streams fails to produce

Thanks for the clearly written KIP, no further comments from my end. -Bill On Wed, Dec 6, 2017 at 9:52 AM, Matt Farmer wrote: > There is already a vote thread for this KIP. I can bump it so that it’s > towards the top of your inbox. > > With regard to your concerns: > > 1) We do not have the "P

Re: [VOTE] KIP-210: Provide for custom error handling when Kafka Streams fails to produce

+1 On Wed, Dec 6, 2017 at 9:54 AM, Matt Farmer wrote: > Bumping this thread so it’s visible given that the conversation on KIP-210 > has converged again. > > Current tally is 2 binding +1s, and 2 non-binding +1s. > > On November 8, 2017 at 12:26:32 PM, Damian Guy (damian@gmail.com) > wrote:

Re: [DISCUSS] KIP-233: Simplify StreamsBuilder#addGlobalStore

Thanks Bill. I can't think of reason to keep the old method too so if there is no further discussion by tomorrow, I would like to start the vote thread. On Tue, Dec 5, 2017 at 10:38 PM, Bill Bejeck wrote: > Hi Panuwat, > > Thanks for the KIP, overall looks good to me. > > I want to play the dev

Re: Flaky healthcheck when trying to check Kafka Stream processing app status

Hi Bill, Thanks, I understand. Let me know if you need further information. Regards, Patrice 2017-12-06 16:03 GMT+01:00 Bill Bejeck : > Hi Patrice, > > I haven't forgotten, just sidetracked with other things. I'll get back to > you by the end of the week. > > Thanks, > Bill > > On Wed, Nov 29,

Re: [DISCUSS] KIP-228 Negative record timestamp support

I'm getting to this a little late, but as for the missing timestamp semantics, it's a +1 from me for using Long.MIN_VALUE for missing timestamps for the reasons outlined by Matthias previously. Thanks, Bill On Wed, Dec 6, 2017 at 2:05 AM, Dong Lin wrote: > Sounds good. I don't think there is co

Build failed in Jenkins: kafka-trunk-jdk8 #2259

See Changes: [ismael] KAFKA-6174; Add methods in AdminClient Options classes to restore binary -- [...truncated 3.38 MB...] kafka.utils.CoreUtilsTest > testCsvList STARTE

Build failed in Jenkins: kafka-trunk-jdk9 #238

See Changes: [ismael] KAFKA-6174; Add methods in AdminClient Options classes to restore binary -- [...truncated 1.44 MB...] kafka.zk.KafkaZkClientTest > testSetGetAndDele

Re: Comparing Pulsar and Kafka: unified queuing and streaming

In terms of building out the Apache Pulsar ecosystem, Landoop is working on porting our Kafka Connect Connectors to Pulsars framework, We already have a Kafka to Pulsar source and sink. On 05/12/2017, 19:59, "Jason Gustafson" wrote: > I believe a lot of users are using the kafka high level

Requesting permission to publish KIP

Hi Kafka Owners, Could you please add me to the Confluence users group so I can publish my KIP for https://issues.apache.org/jira/browse/KAFKA-5722? Thanks, Viktor

[jira] [Created] (KAFKA-6317) Maven artifact for kafka should not depend on log4j

Ismael Juma created KAFKA-6317: -- Summary: Maven artifact for kafka should not depend on log4j Key: KAFKA-6317 URL: https://issues.apache.org/jira/browse/KAFKA-6317 Project: Kafka Issue Type: Imp

[GitHub] kafka pull request #4297: KAFKA-6317: Maven artifact for kafka should not de...

GitHub user ijuma opened a pull request: https://github.com/apache/kafka/pull/4297 KAFKA-6317: Maven artifact for kafka should not depend on log4j It should only depend on slf4j-api (like kafka-clients). The release tarball still includes log4j and slf4j-log4j12. Manuall

Re: [DISCUSS] KIP-233: Simplify StreamsBuilder#addGlobalStore

Hi. I wanted to double check. If I understand the proposal, it would replace the explicit name with a name that is dynamically generated using the AtomicInteger index. Would this affect the naming of any internally generated topics? On Wed, Dec 6, 2017 at 7:59 AM Panuwat Anawatmongkhon < panuwat.

[GitHub] kafka pull request #4298: Remove ignore-case grep option from kafka-server-s...

GitHub user sm4rk0 opened a pull request: https://github.com/apache/kafka/pull/4298 Remove ignore-case grep option from kafka-server-stop.sh In order to avoid killing kafka-manager running on the same machine, because it will match these strings from command line: lib/org.apache

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

> > Thinking about this again. I do see the reason that we want to have a epoch > to avoid out of order registration of the interested set. But I am > wondering if the following semantic would meet what we want better: > - Session Id: the id assigned to a single client for life long time. i.e > it

Re: [DISCUSS] KIP-228 Negative record timestamp support

I agree with Matthias that keeping -1 special will be prone to errors. We should accept this is mistake resulting from lack of foresight on our part when adding timestamps in the first place and correct it. Using deltas will probably cause lots of headaches. It means we have to figure out the impl

Re: Requesting permission to publish KIP

Hi, Viktor, Just gave you the wiki permission. Thanks, Jun On Wed, Dec 6, 2017 at 8:51 AM, Viktor Somogyi wrote: > Hi Kafka Owners, > > Could you please add me to the Confluence users group so I can publish my > KIP for https://issues.apache.org/jira/browse/KAFKA-5722? > > Thanks, > Viktor >

[GitHub] kafka pull request #4299: KAFKA-6300 SelectorTest may fail with ConcurrentMo...

GitHub user tedyu opened a pull request: https://github.com/apache/kafka/pull/4299 KAFKA-6300 SelectorTest may fail with ConcurrentModificationException Synchronization is added w.r.t. sockets ArrayList to avoid ConcurrentModificationException ### Committer Checklist (exclu

[GitHub] kafka pull request #4288: KAFKA-6300 SelectorTest may fail with ConcurrentMo...

Github user tedyu closed the pull request at: https://github.com/apache/kafka/pull/4288 ---

Build failed in Jenkins: kafka-trunk-jdk9 #239

See Changes: [ismael] MINOR: Add maybeThrow method to ZooKeeperClient AsyncResponse -- [...truncated 1.45 MB...] kafka.zk.KafkaZkClientTest > testSetGetAndDeletePartitionR

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

Hey Colin, How about this approach? We have a tunable for minimum eviction time > (default 2 minutes). We cannot evict a client before this timeout has > expired. We also have a tunable for total number of cache slots. We > never cache more than this number of incremental fetch sessions. I t

Re: [VOTE] KIP-225 - Use tags for consumer “records.lag” metrics

Hi Charly, Personally I prefer emitting both and deprecate old one. This does not block on the 2.0 release and we don't need to worry about more users picking up the old metric in 1.1 release. Thanks, Jiangjie (Becket) Qin On Tue, Dec 5, 2017 at 4:08 AM, charly molter wrote: > Thanks Jun and

Re: [VOTE] KIP-223 - Add per-topic min lead and per-partition lead metrics to KafkaConsumer

+1, Thanks for the KIP. One thing is that we might want to make this in consistent with what proposed in KIP-225. On Wed, Dec 6, 2017 at 6:40 AM, Guozhang Wang wrote: > Hi Hu Xi, > > As I said before, it is only a clarification question for its internal > implementation; it is not related to th

Re: [VOTE] KIP-225 - Use tags for consumer “records.lag” metrics

Sounds good I'll update the KIP On Wed, Dec 6, 2017 at 6:04 PM, Becket Qin wrote: > Hi Charly, > > Personally I prefer emitting both and deprecate old one. This does not > block on the 2.0 release and we don't need to worry about more users > picking up the old metric in 1.1 release. > > Thanks,

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

On Wed, Dec 6, 2017, at 09:32, Jason Gustafson wrote: > > > > Thinking about this again. I do see the reason that we want to have a epoch > > to avoid out of order registration of the interested set. But I am > > wondering if the following semantic would meet what we want better: > > - Session Id:

Re: [DISCUSS] KIP-228 Negative record timestamp support

bq. Using deltas will probably cause lots of headaches The downside of my initial proposal for using delta is that it didn't keep the natural ordering among negative timestamps. Here is variation of using delta : -1 is still kept as how it is treated today. The delta is calculated based on: 1/

Interested in becoming committer - Where to fork ?

Hi All!, I am interested to contribute code changes in Kafka but couldn't find enough information regarding forking the codebase on https://kafka.apache.org/contributing. As per https://kafka.apache.org/code , one can use https://git-wip-us.apache.org/repos/asf/kafka.git kafka to clone the repo.

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

> > There is already a way in the existing proposal for clients to change > the set of partitions they are interested in, while re-using their same > session and session ID. We don't need to change how sequence ID works > in order to do this. There is some inconsistency in the KIP about this, so

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

On Wed, Dec 6, 2017, at 10:02, Jason Gustafson wrote: > Hey Colin, > > How about this approach? We have a tunable for minimum eviction time > > (default 2 minutes). We cannot evict a client before this timeout has > > expired. We also have a tunable for total number of cache slots. We > > neve

Re: Interested in becoming committer - Where to fork ?

NVM found this https://cwiki.apache.org/confluence/display/KAFKA/Contributing+Code+Changes On Wed, Dec 6, 2017 at 7:47 PM, coding aurora wrote: > Hi All!, > > I am interested to contribute code changes in Kafka but couldn't find > enough information regarding forking the codebase on > https://ka

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

On Wed, Dec 6, 2017, at 10:49, Jason Gustafson wrote: > > > > There is already a way in the existing proposal for clients to change > > the set of partitions they are interested in, while re-using their same > > session and session ID. We don't need to change how sequence ID works > > in order to

Re: [DISCUSS] KIP-232: Detect outdated metadata by adding ControllerMetadataEpoch field

Bump up the thread. It will be great to have more comments on whether we should do it or whether there is better way to address the motivation of this KIP. On Mon, Dec 4, 2017 at 3:09 PM, Dong Lin wrote: > I don't have an interesting rejected alternative solution to put in the > KIP. If there i

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

On Wed, Dec 6, 2017, at 10:49, Jason Gustafson wrote: > > > > There is already a way in the existing proposal for clients to change > > the set of partitions they are interested in, while re-using their same > > session and session ID. We don't need to change how sequence ID works > > in order to

Re: Flaky healthcheck when trying to check Kafka Stream processing app status

Hi Patrice, Which version of Kafka are you using for this demo app? Guozhang On Wed, Dec 6, 2017 at 8:04 AM, Patrice Chalcol wrote: > Hi Bill, > > Thanks, I understand. Let me know if you need further information. > > Regards, > Patrice > > 2017-12-06 16:03 GMT+01:00 Bill Bejeck : > > > Hi Pa

Re: Interested in becoming committer - Where to fork ?

Thanks for your interest in Kafka! Looking forward to your first PR! :) Development happens via GitHub. Thus, you should clone https://github.com/apache/kafka on GitHub and also open PR on GitHub targeting https://github.com/apache/kafka The Apache repo is the source of truth and it gets mirrored

Re: Interested in becoming committer - Where to fork ?

Hello, You can find the the guidelines on how to clone / fork and submit a PR in https://cwiki.apache.org/confluence/display/KAFKA/Contributing+Code+Changes. Guozhang On Wed, Dec 6, 2017 at 10:47 AM, coding aurora wrote: > Hi All!, > > I am interested to contribute code changes in Kafka but

Re: [VOTE] KIP-210: Provide for custom error handling when Kafka Streams fails to produce

+1 On 12/6/17 7:54 AM, Bill Bejeck wrote: > +1 > > On Wed, Dec 6, 2017 at 9:54 AM, Matt Farmer wrote: > >> Bumping this thread so it’s visible given that the conversation on KIP-210 >> has converged again. >> >> Current tally is 2 binding +1s, and 2 non-binding +1s. >> >> On November 8, 2017

Re: [DISCUSS] KIP-234: add support for getting topic defaults from AdminClient

Rajini, that was not my intent, the intent was to give a user of this api an insight in to what their topic will look like once created. as things stand now a user is unable to (easily) have any knowledge of what their topic configs will be before doing a `CREATE_TOPICS`. as i mentioned in the KIP

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

Hi Becket, Thanks for the ideas. It's interesting to think about how we could further optimize fetches. For now, I think sending offsets in the FetchRequest is still important, because of how we do replication. The leader considers followers caught up with an offset when they fetch the followin

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

Hi Colin, >A full fetch request will certainly avoid any ambiguity here. But now >we're back to sending full fetch requests whenever there are network >issues, which is worse than the current proposal. And has the >congestion collapse problem I talked about earlier when the network is >wobbling.

[jira] [Created] (KAFKA-6318) StreamsResetter should return non-zero return code on error

Matthias J. Sax created KAFKA-6318: -- Summary: StreamsResetter should return non-zero return code on error Key: KAFKA-6318 URL: https://issues.apache.org/jira/browse/KAFKA-6318 Project: Kafka

[jira] [Resolved] (KAFKA-5520) Extend Consumer Group Reset Offset tool for Stream Applications

[ https://issues.apache.org/jira/browse/KAFKA-5520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-5520. -- Resolution: Fixed Issue resolved by pull request 4159 [https://github.com/apache/kafka/pull/4159

[GitHub] kafka pull request #4159: KAFKA-5520: KIP-171 - Extend Consumer Group Reset ...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/4159 ---

[GitHub] kafka pull request #4299: KAFKA-6300 SelectorTest may fail with ConcurrentMo...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/4299 ---

[jira] [Resolved] (KAFKA-6300) SelectorTest may fail with ConcurrentModificationException

[ https://issues.apache.org/jira/browse/KAFKA-6300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajini Sivaram resolved KAFKA-6300. --- Resolution: Fixed Fix Version/s: 1.0.1 1.1.0 Issue resolved by pull

Re: [DISCUSS] KIP-234: add support for getting topic defaults from AdminClient

Hey Dan, I think you are saying that, if user can read the default config before creating the topic, then this user can make better decision in what configs need to be overwritten. The question here is, how user is going to use this config to make the decision. In my understanding, user will comp

Re: [DISCUSS] KIP-231: Improve the Required ACL of ListGroups API

Hey Vahid, Ismael, Good point regarding the similar filter in OffsetFetchResponse and TopicMetadataResponse. This addresses my concern. Thanks, Dong On Tue, Dec 5, 2017 at 2:42 PM, Vahid S Hashemian wrote: > Hi Dong, > > Thanks for your message. > > My driving point for this KIP was a -seeming

[GitHub] kafka pull request #4283: KAFKA-6193: Only delete reassign_partitions znode ...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/4283 ---

Jenkins build is back to normal : kafka-trunk-jdk9 #240

See

[DISCUSS] KIP-236 Interruptible Partition Reassignment

Hi, This is still very new, but I wanted some quick feedback on a preliminary KIP which could, I think, help with providing an AdminClient API for partition reassignment. https://cwiki.apache.org/confluence/display/KAFKA/KIP-236%3A+Interruptible+Partition+Reassignment I wasn't sure whether to st

[jira] [Created] (KAFKA-6319) kafka-acls regression for comma characters (and maybe other characters as well)

Jordan Mcmillan created KAFKA-6319: -- Summary: kafka-acls regression for comma characters (and maybe other characters as well) Key: KAFKA-6319 URL: https://issues.apache.org/jira/browse/KAFKA-6319 Pro

[jira] [Created] (KAFKA-6320) move ZK metrics in KafkaHealthCheck to ZookeeperClient

Jun Rao created KAFKA-6320: -- Summary: move ZK metrics in KafkaHealthCheck to ZookeeperClient Key: KAFKA-6320 URL: https://issues.apache.org/jira/browse/KAFKA-6320 Project: Kafka Issue Type: Sub-task

[GitHub] kafka pull request #4300: KAFKA-6269[WIP] DO NOT MERGE attempts to recreate ...

GitHub user bbejeck opened a pull request: https://github.com/apache/kafka/pull/4300 KAFKA-6269[WIP] DO NOT MERGE attempts to recreate original error *More detailed description of your change, if necessary. The PR title and PR message become the squashed commit message, so us

[jira] [Created] (KAFKA-6321) ConsumerGroupCommand should use the new consumer to query the log end offsets.

Jiangjie Qin created KAFKA-6321: --- Summary: ConsumerGroupCommand should use the new consumer to query the log end offsets. Key: KAFKA-6321 URL: https://issues.apache.org/jira/browse/KAFKA-6321 Project: K

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

>For now, I think sending offsets in the FetchRequest is still important, >because of how we do replication. The leader considers followers caught >up with an offset when they fetch the following offset. I think we >should hold off on changing how replication works in this KIP, and >consider thes

Build failed in Jenkins: kafka-trunk-jdk8 #2260

See Changes: [ismael] MINOR: Add maybeThrow method to ZooKeeperClient AsyncResponse -- [...truncated 3.11 MB...] kafka.zk.ZKPathTest > testCreateEphemeralPathExists PASSED

[jira] [Created] (KAFKA-6322) Error deleting log for topic, all log dirs failed.

dongyan li created KAFKA-6322: - Summary: Error deleting log for topic, all log dirs failed. Key: KAFKA-6322 URL: https://issues.apache.org/jira/browse/KAFKA-6322 Project: Kafka Issue Type: Bug

[GitHub] kafka pull request #4265: KAFKA-6065: Latency metric for KafkaZkClient

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/4265 ---

[jira] [Resolved] (KAFKA-6065) Add ZooKeeperRequestLatencyMs to KafkaZkClient

[ https://issues.apache.org/jira/browse/KAFKA-6065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-6065. Resolution: Fixed > Add ZooKeeperRequestLatencyMs to KafkaZkClient > ---

Re: [DISCUSS] KIP 226 - Dynamic Broker Configuration

Hi, Rajini, Thanks for the kip. Looks good overall. A few comments below. 1. "num.replica.fetchers: Affinity of partitions to threads will be preserved for ordering." Does that mean the new fetcher threads won't be used until new partitions are added? This may be limiting. 2. I am wondering how

Re: [DISCUSS] KIP 226 - Dynamic Broker Configuration

A couple more things. 6. For the SSL/SASL configurations with the listener prefix, do we need another level in config_source since it's neither topic nor broker? 7. For include_synonyms in DescribeConfigs, the name makes sense for the topic level configs. Not sure if it makes sense for other hier

[GitHub] kafka pull request #4261: MINOR: Avoid intermediate strings when parsing/dec...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/4261 ---

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

Hi, Becket, Yes, I agree that it's rare to have the fetch size smaller than index interval. It's just that we still need additional code to handle the rare case. If you go this far, a more general approach (i.e., without returning at the index boundary) is the following. We can cache the followin

Jenkins build is back to normal : kafka-trunk-jdk8 #2261

See

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

Hi, Collin, Thanks for the KIP. A few comments below. 20. Not sure that I fully understand session ID and session epoch. Is session ID tied to a socket connection? That is if the client creates a new connection, will a new session ID be created? 21. "The partition becomes dirty when". It seems t

Build failed in Jenkins: kafka-trunk-jdk7 #3023

See Changes: [ismael] KAFKA-6065; Latency metric for KafkaZkClient -- [...truncated 1.87 MB...] org.apache.kafka.streams.processor.internals.assignment.StickyTaskAssignor

Build failed in Jenkins: kafka-trunk-jdk9 #243

See Changes: [ismael] MINOR: Avoid intermediate strings when parsing/decoding ZK JSON -- [...truncated 1.44 MB...] kafka.server.epoch.LeaderEpochFileCacheTest > shouldDrop

[DISCUSS] KIP-237: More Controller Health Metrics

Hi all, I have created KIP-237: More Controller Health Metrics https://cwiki.apache.org/confluence/display/KAFKA/KIP-237%3A+More+Controller+Health+Metrics . The KIP proposes to add a few more metrics to help monitor Kafka Controller health. Feedback and suggestions are welcome! Thanks, Dong

[jira] [Resolved] (KAFKA-5878) Add sensor for queue size of the controller-event-thread

[ https://issues.apache.org/jira/browse/KAFKA-5878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Lin resolved KAFKA-5878. - Resolution: Duplicate This metric will be added in https://issues.apache.org/jira/browse/KAFKA-3473. > Ad

Re: [DISCUSS]KIP-216: IQ should throw different exceptions for different errors

Thank you for your attention to this KIP and assistance. --- Vito On Wed, Dec 6, 2017 at 2:35 PM, Matthias J. Sax wrote: > Thanks for the info. There is no hurry. Was just curious :) > > -Matthias > > > On 12/5/17 7:34 PM, vito jeng wrote: > > Matthias, > > > > Still in progress. I've been bu

Re: [DISCUSS] KIP-237: More Controller Health Metrics

Thanks for the KIP. Can you add description for EventRateAndQueueTimeMs ? Cheers On Wed, Dec 6, 2017 at 6:21 PM, Dong Lin wrote: > Hi all, > > I have created KIP-237: More Controller Health Metrics > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > 237%3A+More+Controller+Health+Metrics

Re: [DISCUSS] KIP-232: Detect outdated metadata by adding ControllerMetadataEpoch field

Hi, Dong, Great finding on the issue. It's a real problem. A few comments about the KIP. (1) I am not sure about updating controller_metadata_epoch on every UpdateMetadataRequest. Currently, the controller can send UpdateMetadataRequest when there is no actual metadata change. Doing this may requi

Re: [DISCUSS] KIP-237: More Controller Health Metrics

Thanks for the comment. I have updated the KIP to explain the value of these metrics. On Wed, Dec 6, 2017 at 6:38 PM, Ted Yu wrote: > Thanks for the KIP. > > Can you add description for EventRateAndQueueTimeMs ? > > Cheers > > On Wed, Dec 6, 2017 at 6:21 PM, Dong Lin wrote: > > > Hi all, > > >

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

Hi Jun, Yes, we still need to handle the corner case. And you are right, it is all about trade-off between simplicity and the performance gain. I was thinking that the brokers always return at least log.index.interval.bytes per partition to the consumer, just like we will return at least one mess

Build failed in Jenkins: kafka-trunk-jdk7 #3024

See Changes: [ismael] MINOR: Avoid intermediate strings when parsing/decoding ZK JSON -- [...truncated 1.87 MB...] org.apache.kafka.streams.processor.internals.assignment

[jira] [Created] (KAFKA-6323) punctuate with WALL_CLOCK_TIME triggered immediately

Frederic Arno created KAFKA-6323: Summary: punctuate with WALL_CLOCK_TIME triggered immediately Key: KAFKA-6323 URL: https://issues.apache.org/jira/browse/KAFKA-6323 Project: Kafka Issue Type

Re: Comparing Pulsar and Kafka: unified queuing and streaming

Andrew, Thank you! Is there any estimation on when I can try out Kafka Connect with Pulsar? Can you also point me when I can find the Kafka-to-Pulsar source and sink? - KN On Wed, Dec 6, 2017 at 2:48 AM, Andrew Stevenson wrote: > In terms of building out the Apache Pulsar ecosystem, Landoop i

Pulsar and Kafka - Segment Centric vs Partition Centric

Hi Kafka Community, Has anyone taken a look at this blog post, comparing pulsar and kafka from architectural view? I am wondering how you guys think about segment centric vs partition centric. https://streaml.io/blog/pulsar-segment-based-architecture/ - KN