[jira] [Resolved] (KAFKA-1977) Make logEndOffset available in the Zookeeper consumer

[ https://issues.apache.org/jira/browse/KAFKA-1977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-1977. -- Resolution: Auto Closed Closing this as the Scala consumers have been deprecated and no further work is

[jira] [Resolved] (KAFKA-6199) Single broker with fast growing heap usage

[ https://issues.apache.org/jira/browse/KAFKA-6199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Tweedie resolved KAFKA-6199. -- Resolution: Fixed Fix Version/s: 1.1.0 We upgraded this single broker to 1.1.0 (keeping t

Re: [DISCUSS] KIP-298: Error Handling in Connect

Ewen, Thanks a lot for your comments! 1. For errors.retry.delay.max.ms, yes we plan to use exponential backoffs with an fixed initial value. Updated the KIP to say this. 2. A failed operation will be retried (potentially multiple times). If all the retries fail, we declare this to be an error. T

Re: [DISCUSS] KIP-290: Support for wildcard suffixed ACLs

Hi Andy, I still have some minor changes left to the KIP. I'll make them in the morning. I'm sorry I got caught up in some other things today. But that would still give us 72 hours before the deadline :) Thanks Piyush Vijay On Thu, May 17, 2018 at 1:27 PM, Andy Coates wrote: > Hey Piyush - m

Re: [VOTE] KIP-277 - Fine Grained ACL for CreateTopics API

bumping again as this KIP is still missing one binding vote ... voting so far : binding: Guozhang, Rajini non-binding: Ted, Stephane, Vahid, Colin, Manikumar, Matt Edo On 17 May 2018 at 03:15, Matt Farmer wrote: > +1 (non-binding) > > On Tue, May 15, 2018 at 4:26 AM, Edoardo Comar wrote: > >>

[jira] [Resolved] (KAFKA-3180) issue for extracting JSON from https web page

[ https://issues.apache.org/jira/browse/KAFKA-3180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-3180. -- Resolution: Not A Problem  I suggest to post these kind of queries to  [[us...@kafka.apache.org|mailto:u

[jira] [Resolved] (KAFKA-3620) Clean up Protocol class.

[ https://issues.apache.org/jira/browse/KAFKA-3620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-3620. -- Resolution: Fixed Closing this as some cleanup done in newer versions.  Please reopen if you think othe

[jira] [Resolved] (KAFKA-3843) Endless consuming messages in Kafka cluster

[ https://issues.apache.org/jira/browse/KAFKA-3843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-3843. -- Resolution: Auto Closed The Scala consumers have been deprecated and no further work is planned, please

[jira] [Created] (KAFKA-6916) AdminClient does not refresh metadata on broker failure

Rajini Sivaram created KAFKA-6916: - Summary: AdminClient does not refresh metadata on broker failure Key: KAFKA-6916 URL: https://issues.apache.org/jira/browse/KAFKA-6916 Project: Kafka Issue

[jira] [Resolved] (KAFKA-4091) Unable to produce or consume on any topic

[ https://issues.apache.org/jira/browse/KAFKA-4091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-4091. -- Resolution: Cannot Reproduce {color:#00}Closing inactive issue. Please reopen if you think the issue

Re: [VOTE] #2 KIP-248: Create New ConfigCommand That Uses The New AdminClient

Hi Colin, Ok, let's drop it for now and I'll do it as you suggested: merging in the ConfigCommand and leave the AC unchanged. I'll update the KIP with this and the other stuff we agreed on (I think it was mostly the SCRAM tool but will revisit the thread). Thank you very much for your contribution

Re: [VOTE] #2 KIP-248: Create New ConfigCommand That Uses The New AdminClient

Ok, ignore my previous mail (except the last sentence), gmail didn't update me about your last email :/. > I think we should probably just create a flag for alterConfigs which marks it > as incremental, like we discussed earlier, and do this as a compatible change > that is needed for the shell

[jira] [Created] (KAFKA-6917) Request handler deadlocks attempting to acquire group metadata lock

Rajini Sivaram created KAFKA-6917: - Summary: Request handler deadlocks attempting to acquire group metadata lock Key: KAFKA-6917 URL: https://issues.apache.org/jira/browse/KAFKA-6917 Project: Kafka

Re: [VOTE] KIP-297: Externalizing Secrets for Connect Configurations

Hi Robert, A couple of questions: 1. Since we always expand config values, don't we also need a way to include values that never get expanded? I may want to use "${vault:mypassword}" as my literal password without a lookup. Since we allow only level of indirection, perhaps all we nee

Re: [DISCUSS] KIP-304: Connect runtime mode improvements for container platforms

Hi, thanks for the feedback, you have some valid points that I did not consider thoroughly. Now that I think about it, my current proposal is indeed a bit hacky, namely the dichotomy between running "light"/"heavy" connectors in standalone/distributed mode: indeed there should be no difference, o

Fails Kafka core tests

[INFO:2018-05-18 13:32:08,904]: starting test run with session id 2018-05-18--006... [INFO:2018-05-18 13:32:08,904]: running 17 tests... [INFO:2018-05-18 13:32:08,905]: Triggering test 1 of 17... [INFO:2018-05-18 13:32:09,046]: RunnerClient: Loading test {'directory': '/home/vmukha/proj/kafka-1.0.1

Re: [DISCUSS] KIP-273 Kafka to support using ETCD beside Zookeeper

Hi Jakub, Yes I tried it and as I saw in various forums it works well for Kafka as well, but because it mimics the Zookeeper API it creates quite many metadata related value. Kafka uses only a subset from the Zookeeper API, so those values can be omitted by implementing Kafka Zookeeper calls in a

Re: [VOTE] KIP-297: Externalizing Secrets for Connect Configurations

Thanks Ewen! > * Let's list in the KIP what package the ConfigProvider, > ConfigChangeCallback, ConfigData and ConfigTransformer interfaces are > defined in. Does org.apache.kafka.common.config work for people? > Also, I think ConfigData is left out of the list of new interfaces by accident Goo

Re: [VOTE] #2 KIP-248: Create New ConfigCommand That Uses The New AdminClient

Updated KIP-248: https://cwiki.apache.org/confluence/display/KAFKA/KIP-248+-+Create+New+ConfigCommand+That+Uses+The+New+AdminClient I'd like to ask project members, committers and contributors to vote as this would be a useful improvement in Kafka. Sections changed: - Public interfaces: added the

Re: [VOTE] KIP-297: Externalizing Secrets for Connect Configurations

HI Rajini, Good questions. First, if no ConfigProviders are configured, then config values of the form ${vault:mypassword} will remain as is. Second, I mention in the KIP that if a provider does not have a value for a given key, the variable will remain unresolved and the final value will be of

[jira] [Created] (KAFKA-6918) Kafka server fails to start with IBM JAVA

Nayana Thorat created KAFKA-6918: Summary: Kafka server fails to start with IBM JAVA Key: KAFKA-6918 URL: https://issues.apache.org/jira/browse/KAFKA-6918 Project: Kafka Issue Type: Bug A

Re: [VOTE] KIP-297: Externalizing Secrets for Connect Configurations

Thanks, Robert! Sounds good. And thanks for the KIP. +1 (binding) Regards, Rajini On Fri, May 18, 2018 at 4:17 PM, Robert Yokota wrote: > HI Rajini, > > Good questions. > > First, if no ConfigProviders are configured, then config values of the form > ${vault:mypassword} will remain as is. > >

[jira] [Resolved] (KAFKA-5728) Stopping consumer thread cause loosing message in the partition

[ https://issues.apache.org/jira/browse/KAFKA-5728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-5728. -- Resolution: Auto Closed looks like this is related to spring kafka config issue. must be related to com

Re: [DISCUSS] KIP-293 Add new metrics for consumer/replication fetch requests

Hello, There haven't been many comments regarding this KIP (not surprising as the code change is ~10 lines long). Would it be okay to start the vote? https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=80452537 Best regards, Adam Kotwasinski On Wed, May 2, 2018 at 6:49 AM, Adam Kot

Re: [VOTE] KIP-285: Connect Rest Extension Plugin

+1 (non-binding) - Konstantine On Thu, May 17, 2018 at 10:05 PM, Ewen Cheslack-Postava wrote: > +1 (binding) > > Thanks, > Ewen > > On Thu, May 17, 2018 at 12:16 PM Ted Yu wrote: > > > +1 > > Original message From: Gwen Shapira > > Date: 5/17/18 12:02 PM (GMT-08:00) To: dev

Re: [DISCUSS] KIP-301 Schema Inferencing in JsonConverters

There are lots of potential details/gotchas here, with no obvious defaults. Even if we did have optional configs for the type(s) to use for empty arrays and null values, how likely is it that those would apply for all occurrences? Two fields might have empty arrays, but the best schema for each mig

Re: [VOTE] KIP-285: Connect Rest Extension Plugin

+1 (non-binding) Regards, Randall On Fri, May 18, 2018 at 11:32 AM, Konstantine Karantasis < konstant...@confluent.io> wrote: > +1 (non-binding) > > - Konstantine > > On Thu, May 17, 2018 at 10:05 PM, Ewen Cheslack-Postava > > wrote: > > > +1 (binding) > > > > Thanks, > > Ewen > > > > On Thu, M

Re: [VOTE] KIP-297: Externalizing Secrets for Connect Configurations

Looks great. +1 (non-binding) Regards, Randall On Fri, May 18, 2018 at 10:23 AM, Rajini Sivaram wrote: > Thanks, Robert! Sounds good. And thanks for the KIP. > > +1 (binding) > > Regards, > > Rajini > > On Fri, May 18, 2018 at 4:17 PM, Robert Yokota wrote: > > > HI Rajini, > > > > Good questi

Build failed in Jenkins: kafka-trunk-jdk8 #2653

See Changes: [wangguoz] KAFKA-5697: revert wakeup-based impl (#5035) -- [...truncated 423.14 KB...] kafka.api.SaslClientsWithInvalidCredentialsTest > testConsumerGroupSe

Jenkins build is back to normal : kafka-trunk-jdk10 #113

See

[jira] [Resolved] (KAFKA-6566) SourceTask#stop() not called after exception raised in poll()

[ https://issues.apache.org/jira/browse/KAFKA-6566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewen Cheslack-Postava resolved KAFKA-6566. -- Resolution: Fixed > SourceTask#stop() not called after exception raised in poll(

Re: [DISCUSS] KIP-293 Add new metrics for consumer/replication fetch requests

Hey Adam, Thanks for the KIP. We currently already have the per-topic byte-out-rate (not including replication traffic) with MBean path kafka.server:name=BytesOutPerSec,type=BrokerTopicMetrics,topic=*. Though this is not the FetchRequest rate, it seems to address the motivation of the KIP by telli

Re: [DISCUSS] KIP-293 Add new metrics for consumer/replication fetch requests

Hi Dong, Unfortunately this does not work when there are no messages, but consumers are still running. BytesOutPerSec increases only when the messages are being delivered to the consumer. But you could have a situation when consumer is already at the end of log & noone is going to produce any more

Build failed in Jenkins: kafka-0.10.2-jdk7 #210

See Changes: [me] KAFKA-6566: Improve Connect Resource Cleanup -- [...truncated 159.29 KB...] kafka.coordinator.GroupCoordinatorResponseTest > testSyncGroupNotCoordinator

Re: [DISCUSS] KIP-293 Add new metrics for consumer/replication fetch requests

Hey Adam, Got it. Thanks for the explanation. Could you also explain the use-case for knowing whether a topic with no new messages has active consumers? Just want to make sure that the use-case is reasonable and can not be accomplished with the existing available approach. Thanks, Dong On Fri,

Re: [DISCUSS] KIP-305: Add Connect primitive number converters

Thanks, Ewen. You make several good points, and I've updated the KIP to hopefully address your comments. I think the symmetry with the Kafka serdes is useful, so I've kept all 5 converters in the KIP. Interestingly, perhaps the short and int converters (with the reduced ranges) are not necessaril

Build failed in Jenkins: kafka-1.0-jdk7 #191

See Changes: [me] KAFKA-6566: Improve Connect Resource Cleanup -- [...truncated 373.52 KB...] kafka.server.epoch.LeaderEpochFileCacheTest > shouldReturnUnsupportedIfNoEpochR

Re: [DISCUSS] KIP-293 Add new metrics for consumer/replication fetch requests

Hey Dong, I have observed two usecases: a) monitoring (if infrastructure does not have any control over clients) - if I only provide Kafka brokers, but I am not aware of clients, I cannot say if anyone would even consume the data - this can be somehow compared to listing existing consumers/connect

Re: [DISCUSS] KIP-290: Support for wildcard suffixed ACLs

Hey Piyush, How are you getting on updating the KIP? Can we offer any support? We're starting to fly really close to the required 72 hours cut off for KIPs. This doesn't leave much room for resolving any issues any committers find. Also, we now require at least three committers to review this KIP

Re: [DISCUSS] KIP-290: Support for wildcard suffixed ACLs

On it, Andy. It should be out in next 30 mins. On Fri, May 18, 2018 at 12:17 PM Andy Coates wrote: > Hey Piyush, > > How are you getting on updating the KIP? Can we offer any support? We're > starting to fly really close to the required 72 hours cut off for KIPs. > This doesn't leave much room f

Re: [DISCUSS] KIP-293 Add new metrics for consumer/replication fetch requests

Hey Adam, Regarding a), if the goal is to monitor the load in the system, then the existing FetchRequest rate (including those sent by the broker) would probably be sufficient. Anyway, this new metric for monitoring system load seems useful with little overhead in code or operation. Just trying to

Build failed in Jenkins: kafka-trunk-jdk10 #114

See Changes: [me] KAFKA-6566: Improve Connect Resource Cleanup [github] MINOR: Remove unused class (#5037) -- [...truncated 1.50 MB...] kafka.network.SocketServerTest > t

Re: [DISCUSS] KIP-290: Support for wildcard suffixed ACLs

Andy, 1. Updated the KIP about need for getMatchingAcls(). Basically, it's required to add an inspection method without breaking compatibility. getAcls() only looks at a single location. 2. Nothing will change from user's perspective. they will add / delete/ list ACLs as earlier. 3. Good point a

Re: [DISCUSS] KIP-290: Support for wildcard suffixed ACLs

Ready to review. Let me know if something looks missing or not clear. Thanks Piyush Vijay On Fri, May 18, 2018 at 12:54 PM, Piyush Vijay wrote: > Andy, > > 1. Updated the KIP about need for getMatchingAcls(). Basically, it's > required to add an inspection method without breaking compatibilit

Build failed in Jenkins: kafka-0.11.0-jdk7 #369

See Changes: [me] KAFKA-6566: Improve Connect Resource Cleanup -- [...truncated 2.45 MB...] org.apache.kafka.streams.integration.JoinIntegrationTest > testLeftKStreamKStr

[jira] [Resolved] (KAFKA-6917) Request handler deadlocks attempting to acquire group metadata lock

[ https://issues.apache.org/jira/browse/KAFKA-6917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson resolved KAFKA-6917. Resolution: Fixed > Request handler deadlocks attempting to acquire group metadata lock > --

Re: [DISCUSS] KIP-290: Support for wildcard suffixed ACLs

Hey Piyush, Thanks for getting this in! :D About to read now. But just quickly... 1. I'll read up on the need for getMatchingAcls - but just playing devils advocate for a moment - if a current caller of getAcls() expects it to return the full set of ACLs for a given resource, would post this cha

Re: [DISCUSS] KIP-293 Add new metrics for consumer/replication fetch requests

Hey Dong, Re a) - not really, it's only the corner case when you actually have consumers but no one is producing (I had that in my environments, when producer services were shut down weeks before consumers). However, it is absolutely possible to argue that a simple gauge answering a question `when

Re: [DISCUSS] KIP-290: Support for wildcard suffixed ACLs

OK I've read it now. 1. I see you have an example: > For example: If I want to fetch all ACLs that match ’topicA*’, it’s not possible without introducing new API AND maintaining backwards compatibility. getAcls takes a Resource, right, which would be either a full resource name or 'ALL', i.e. '*',

Re: [DISCUSS] KIP-305: Add Connect primitive number converters

Yeah, the usefulness of short seems questionable, but int is probably a large enough range for some identifiers (e.g. we use an int in schema registry). But yeah, I don't really have a problem with having Converters for each of the existing serdes just for symmetry and since presumably somebody fin

Jenkins build is back to normal : kafka-trunk-jdk8 #2654

See

Re: [DISCUSS] KIP-298: Error Handling in Connect

The updated version of the KIP that uses the dead-letter-queue only for sink records and only to store the raw record data looks better and easier to understand. I think it's moving to the right direction. No further comments from my side. Thanks Arjun! - Konstantine On Fri, May 18, 2018 at 1:0

Re: [VOTE] KIP-298: Error Handling in Connect kafka

+1 (non-binding) - Konstantine On Thu, May 17, 2018 at 12:58 AM, Arjun Satish wrote: > All, > > Many thanks for all the feedback on KIP-298. Highly appreciate the time and > effort you all put into it. > > I've updated the KIP accordingly, and would like to start to start a vote > on it. > > KI

Re: [DISCUSS] KIP-298: Error Handling in Connect

Thanks, Konstantine! On Fri, May 18, 2018 at 2:36 PM, Konstantine Karantasis < konstant...@confluent.io> wrote: > The updated version of the KIP that uses the dead-letter-queue only for > sink records and only to store the raw record data looks better and easier > to understand. > I think it's mo

Re: [DISCUSS] KIP-298: Error Handling in Connect

Arjun, Thanks for all the updates. I think it looks great and I don't have any other concerns. Thanks Magesh On Fri, May 18, 2018 at 2:36 PM, Konstantine Karantasis < konstant...@confluent.io> wrote: > The updated version of the KIP that uses the dead-letter-queue only for > sink records and on

Re: [DISCUSS] KIP-298: Error Handling in Connect

Super! Thanks, Magesh! On Fri, May 18, 2018 at 2:53 PM, Magesh Nandakumar wrote: > Arjun, > > Thanks for all the updates. I think it looks great and I don't have any > other concerns. > > Thanks > Magesh > > On Fri, May 18, 2018 at 2:36 PM, Konstantine Karantasis < > konstant...@confluent.io> wr

Re: [VOTE] KIP-298: Error Handling in Connect kafka

+1 (non-binding) Thanks, Magesh On Fri, May 18, 2018 at 2:38 PM, Konstantine Karantasis < konstant...@confluent.io> wrote: > +1 (non-binding) > > - Konstantine > > On Thu, May 17, 2018 at 12:58 AM, Arjun Satish > wrote: > > > All, > > > > Many thanks for all the feedback on KIP-298. Highly appr

Build failed in Jenkins: kafka-1.1-jdk7 #134

See Changes: [jason] KAFKA-6917; Process txn completion asynchronously to avoid deadlock -- [...truncated 416.64 KB...] kafka.server.epoch.LeaderEpochFileCacheTest > should

Re: [VOTE] KIP-298: Error Handling in Connect kafka

+1 Thank you! Error handling in Connect will be a huge improvement. On Thu, May 17, 2018, 1:58 AM Arjun Satish wrote: > All, > > Many thanks for all the feedback on KIP-298. Highly appreciate the time and > effort you all put into it. > > I've updated the KIP accordingly, and would like to star

Build failed in Jenkins: kafka-trunk-jdk8 #2655

See Changes: [jason] KAFKA-6917; Process txn completion asynchronously to avoid deadlock -- [...truncated 423.08 KB...] kafka.admin.ResetConsumerGroupOffsetTest > testRese

Jenkins build is back to normal : kafka-1.0-jdk7 #192

See

Re: [VOTE] #2 KIP-248: Create New ConfigCommand That Uses The New AdminClient

Hi Viktor, Thanks, this looks good. The boolean should default to false if not set, to ensure that existing clients continue to work as-is, right? Might be good to add a note specifying that. +1 (non-binding) best, Colin On Fri, May 18, 2018, at 08:16, Viktor Somogyi wrote: > Updated KIP-248

Re: [DISCUSS] KIP-285: Connect Rest Extension Plugin

Hi Magesh, Thanks for the KIP. It's definitely useful to have a pluggable auth layer, as we have for the kafka brokers. I was a bit unclear why we needed to expose all this health information in the context though. Since it is the bigger part of the API, I was hoping to see a little more motivatio

[jira] [Resolved] (KAFKA-6849) Add transformValues() method to KTable

[ https://issues.apache.org/jira/browse/KAFKA-6849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-6849. -- Resolution: Fixed Fix Version/s: 2.0.0 > Add transformValues() method to KTable > --

Build failed in Jenkins: kafka-trunk-jdk8 #2656

See Changes: [wangguoz] KAFKA-6849: add transformValues methods to KTable. (#4959) -- [...truncated 423.07 KB...] kafka.admin.ResetConsumerGroupOffsetTest > testResetOffs

Build failed in Jenkins: kafka-trunk-jdk7 #3439

See Changes: [wangguoz] KAFKA-6849: add transformValues methods to KTable. (#4959) -- [...truncated 420.12 KB...] kafka.admin.ResetConsumerGroupOffsetTest > testResetOffse

Re: [DISCUSS] KIP-290: Support for wildcard suffixed ACLs

Hi Piyush, We're fast approaching the KIP deadline. Are you actively working on this? If you're not I can take over. Thanks, Andy On 18 May 2018 at 14:25, Andy Coates wrote: > OK I've read it now. > > 1. I see you have an example: > > For example: If I want to fetch all ACLs that match ’topic

Re: [DISCUSS] KIP-290: Support for wildcard suffixed ACLs

Piyush Vijay On Fri, May 18, 2018 at 1:55 PM, Andy Coates wrote: > Hey Piyush, > > Thanks for getting this in! :D > > About to read now. But just quickly... > > 1. I'll read up on the need for getMatchingAcls - but just playing devils > advocate for a moment - if a current caller of getAcls() ex

Jenkins build is back to normal : kafka-trunk-jdk10 #116

See

Re: [VOTE] KIP-255: OAuth Authentication via SASL/OAUTHBEARER

Hi committers. KIP 255 still needs 1 more binding vote. Currently there are two binding + 1 votes, from Rajini and Jun, and three non-binding +1 votes, from Mickael, Manikumar, and myself. Please vote by the Monday deadline. Ron On Thu, May 17, 2018 at 10:59 AM, Ron Dagostino wrote: > Hi Jun

Re: [DISCUSS] KIP-290: Support for wildcard suffixed ACLs

Updated the KIP. 1. New enum field 'ResourceNameType' in Resource and ResourceFilter classes. 2. modify getAcls() and rely on ResourceNameType' field in Resource to return either exact matches or all matches based on wildcard-suffix. 3. CLI changes to identify if resource name is literal or wildca

[VOTE] KIP-290: Support for wildcard suffixed ACLs

Hi everyone, I would like to start a vote for https://cwiki.apache.org/confluence/display/KAFKA/KIP-290%3A+Support+for+wildcard+suffixed+ACLs . The KIP proposes a way to support wildcard-suffixed resource names in Kafka ACLs. The main challenge was to support it in a backward compatible way beca

[VOTE] KIP-290: Support for wildcard suffixed ACLs

Hi everyone, I would like to start a vote for https://cwiki.apache.org/ confluence/display/KAFKA/KIP-290%3A+Support+for+wildcard+suffixed+ACLs. The KIP proposes a way to support wildcard-suffixed resource names in Kafka ACLs. The main challenge was to support it in a backward compatible way beca