Jenkins build is back to normal : kafka-trunk-jdk10 #108

See

Re: [DISCUSS] KIP-298: Error Handling in Connect

Magesh, Just to add to your point about retriable exceptions: the producer can throw retriable exceptions which we are handling it here: https://github.com/apache/kafka/blob/trunk/connect/ runtime/src/main/java/org/apache/kafka/connect/runtime/ WorkerSourceTask.java#L275 BTW, exceptions like Tim

Build failed in Jenkins: kafka-trunk-jdk8 #2647

See Changes: [github] MINOR: doc change for deprecate removal (#5006) -- [...truncated 423.06 KB...] kafka.admin.ResetConsumerGroupOffsetTest > testResetOffsetsShiftPlus

Re: [DISCUSS] KIP-303: Add Dynamic Routing in Streams Sink

Just my 2 cents: I am fine with `KeyValueMapper` (+1 for code reusage) -- the JavaDocs will explain what the `KeyValueMapper` is supposed to do, ie, extract and return the sink topic name from the key-value pair. A side remark though: do we think that accessing key/value is sufficient? Or should

Re: [DISCUSS] KIP-298: Error Handling in Connect

Magesh, Thanks for the feedback! Really appreciate your comments. 1. I updated the KIP to state that only the configs of the failed operation will be emitted. Thank you! The purpose of bundling the configs of the failed operation along with the error context is to have a single place to find eve

Re: [VOTE] KIP-255: OAuth Authentication via SASL/OAUTHBEARER

Hi Jun. I think you are getting at the fact that OAuth 2 is a flexible framework that allows different installations to do things differently. It is true that the principal name in Kafka could come from any claim in the token. Most of the time it would come from the 'sub' claim, but it could cer

Re: [DISCUSS] KIP-297: Externalizing Secrets for Connect Configurations

Thanks, Ron! I will take a look. Regards, Robert On Tue, May 15, 2018 at 5:59 PM, Ron Dagostino wrote: > Hi Robert. Regarding your comment "use the lease duration to schedule a > configuration reload in the future", you might be interested in the code > that does refresh for OAuth Bearer Toke

Re: [DISCUSS] KIP-297: Externalizing Secrets for Connect Configurations

Hi Robert. Regarding your comment "use the lease duration to schedule a configuration reload in the future", you might be interested in the code that does refresh for OAuth Bearer Tokens in KIP-255; specifically, the class org.apache.kafka.common.security.oauthbearer.internal.expiring.ExpiringCred

Re: [DISCUSS] KIP-303: Add Dynamic Routing in Streams Sink

Thanks for the KIP Guozhang, it's a +1 for me. As for re-using the KeyValueMapper for choosing the topic, I am on the fence, a more explicitly named class would be more clear, but I'm not sure it's worth a new class that will primarily perform the same actions as the KeyValueMapper. Thanks, Bill

Re: [DISCUSS] KIP-298: Error Handling in Connect

Matt, Thanks so much for your comments. Really appreciate it! 1. Good point about the acronym. I can use deadletterqueue instead of dlq (using all lowercase to be consistent with the other configs in Kafka). What do you think? 2. Could you please tell us what errors caused these tasks to fail? W

Build failed in Jenkins: kafka-trunk-jdk10 #107

See Changes: [wangguoz] KAFKA-6896: Add producer metrics exporting in KafkaStreams (#4998) -- [...truncated 1.50 MB...] kafka.admin.ResetConsumerGroupOffsetTest > testRese

Re: [DISCUSS] KIP-297: Externalizing Secrets for Connect Configurations

Hi Colin, Thanks for the feedback! > The KIP says that "Vault is very popular and has been described as 'the current gold standard in secret management and provisioning'." I think this might be a bit too much detail -- we don't really need to > favorites, right? :) I've removed this line :)

Re: [DISCUSS] KIP-297: Externalizing Secrets for Connect Configurations

Hi Robert, Thanks for posting this. In the past we've been kind of reluctant to add more complexity to configuration. I think Connect does have a clear need for this kind of functionality, though. As you mention, Connect integrates with external systems, which are very likely to have passwor

[jira] [Created] (KAFKA-6906) Kafka Streams does not commit transactions if data is produced via wall-clock punctuation

Matthias J. Sax created KAFKA-6906: -- Summary: Kafka Streams does not commit transactions if data is produced via wall-clock punctuation Key: KAFKA-6906 URL: https://issues.apache.org/jira/browse/KAFKA-6906

Re: [VOTE] KIP-295: Add Streams Configuration Allowing for Optional Topology Optimization

+1 Original message From: Guozhang Wang Date: 5/15/18 2:34 PM (GMT-08:00) To: dev@kafka.apache.org Subject: Re: [VOTE] KIP-295: Add Streams Configuration Allowing for Optional Topology Optimization +1 (binding). On Tue, May 15, 2018 at 2:16 PM, Matthias J. Sax wrote: > +1

Re: [DISCUSS] KIP-303: Add Dynamic Routing in Streams Sink

Hello John: * As for the type superclass, it is mainly for allowing super class serdes. More details can be found here: https://cwiki.apache.org/confluence/display/KAFKA/KIP-100+-+Relax+Type+constraints+in+Kafka+Streams+API * I may have slight preference on reusing existing classes but I think mo

Re: [DISCUSS] KIP-303: Add Dynamic Routing in Streams Sink

Thanks for the KIP. +1 @John: compare https://cwiki.apache.org/confluence/display/KAFKA/KIP-100+-+Relax+Type+constraints+in+Kafka+Streams+API about the generics -Matthias On 5/15/18 11:19 AM, John Roesler wrote: > Thanks for the KIP, Guozhang. > > It looks good overall to me; I just have one

Re: [DISCUSS] KIP-301 Schema Inferencing in JsonConverters

I've went through several iterations of back-and-forth with @rhauch on the PR and on Confluent's Slack Community. The current thinking is that assuming an empty array is a String array is not necessarily the best option, nor is assuming that all null values in a JSON node is a String. We might be

Re: [VOTE] KIP-295: Add Streams Configuration Allowing for Optional Topology Optimization

+1 (binding). On Tue, May 15, 2018 at 2:16 PM, Matthias J. Sax wrote: > +1 (binding) > > > On 5/15/18 1:45 PM, Bill Bejeck wrote: > > Hi all, > > > > I'd like to start a vote on KIP-295: Add Streams Configuration Allowing > for > > Optional Topology Optimization. > > > > KIP wiki page: > > https

[jira] [Resolved] (KAFKA-6896) add producer metrics exporting in KafkaStreams.java

[ https://issues.apache.org/jira/browse/KAFKA-6896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-6896. -- Resolution: Fixed > add producer metrics exporting in KafkaStreams.java > --

Re: [EXTERNAL] Kafka Connect: New Kafka Source Connector

Hi Team, Would someone be able to provide me with Confluence permission in order to write a KIP for the below code. User: https://cwiki.apache.org/confluence/display/~mccaig Cheers, Rhys On May 11, 2018, at 4:45 PM, McCaig, Rhys mailto:rhys_mcc...@comcast.com>> wrote: Hi there, Over at Comc

Re: [VOTE] KIP-295: Add Streams Configuration Allowing for Optional Topology Optimization

+1 (binding) On 5/15/18 1:45 PM, Bill Bejeck wrote: > Hi all, > > I'd like to start a vote on KIP-295: Add Streams Configuration Allowing for > Optional Topology Optimization. > > KIP wiki page: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-295%3A+Add+Streams+Configuration+Allowing+fo

[VOTE] KIP-295: Add Streams Configuration Allowing for Optional Topology Optimization

Hi all, I'd like to start a vote on KIP-295: Add Streams Configuration Allowing for Optional Topology Optimization. KIP wiki page: https://cwiki.apache.org/confluence/display/KAFKA/KIP-295%3A+Add+Streams+Configuration+Allowing+for+Optional+Topology+Optimization Discussion thread: https://www.mai

Re: [DISCUSS] KIP-295: Add Streams Configuration Allowing for Optional Topology Optimization

At this point, I think the overall discussion on the new config is completed and I'll start a voting thread. Thanks, Bill On Tue, May 15, 2018 at 4:37 PM, Bill Bejeck wrote: > Thanks for the comments, John. I've updated the config name on the KIP.. > > -Bill > > > On Tue, May 15, 2018 at 2:2

Re: [DISCUSS] KIP-295: Add Streams Configuration Allowing for Optional Topology Optimization

Thanks for the comments, John. I've updated the config name on the KIP.. -Bill On Tue, May 15, 2018 at 2:22 PM, John Roesler wrote: > Hi Bill, > > Thanks for the KIP. Now that we're using strings describing the "set of > optimizations", such as "none" and "all", should we change the config na

Re: [DISCUSS] KIP-298: Error Handling in Connect

Hi Arjun, I think this a great KIP and would be a great addition to have in connect. Had a couple of minor questions: 1. What would be the value in logging the connector config using errors.log.include.configs for every message? 2. Not being picky on format here but it might be clearer if the beh

[jira] [Created] (KAFKA-6905) Document that Processor objects can be reused

David Glasser created KAFKA-6905: Summary: Document that Processor objects can be reused Key: KAFKA-6905 URL: https://issues.apache.org/jira/browse/KAFKA-6905 Project: Kafka Issue Type: Impro

Re: [DISCUSS] KIP-295: Add Streams Configuration Allowing for Optional Topology Optimization

Hi Bill, Thanks for the KIP. Now that we're using strings describing the "set of optimizations", such as "none" and "all", should we change the config name to just "topology.optimizations"? The "enable." feels like a holdover from the boolean-valued config. Thanks, -John On Tue, May 8, 2018 at

Re: [DISCUSS] KIP-303: Add Dynamic Routing in Streams Sink

Thanks for the KIP, Guozhang. It looks good overall to me; I just have one question: * Why do we bound the generics of KVMapper in KStream to be superclass-of-K and superclass-of-V instead of exactly K and V, as in Topology? I might be thinking about it wrong, but that seems backwards to me. If an

Re: [DISCUSS] KIP-290: Support for wildcard suffixed ACLs

Hi Piyush, It is possible to configure PrincipalBuilder for SASL ( https://cwiki.apache.org/confluence/display/KAFKA/KIP-189%3A+Improve+principal+builder+interface+and+add+support+for+SASL). If that satisfies your requirements, perhaps we can move wildcarded principals out of this KIP and focus on

Re: [DISCUSS] KIP-290: Support for wildcard suffixed ACLs

Hi Colin, Escaping at this level is making sense to me but let me think more and get back to you. But should we not just get rid of one of AclBinding or AclBindingFilter then? Is there a reason to keep both given that AclBindingFilter and AclBinding look exact copy of each other after this change

Re: [VOTE] KIP-282: Add the listener name to the authentication context

Thanks for the KIP, +1 (binding). Ismael On Wed, Apr 25, 2018 at 1:52 AM Mickael Maison wrote: > Hi, > > There has been no objections in the DISCUSS thread so I'd like to > start a vote on KIP-282: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-282%3A+Add+the+listener+name+to+the+aut

Re: [VOTE] KIP-283: Efficient Memory Usage for Down-Conversion

Thanks for the KIP Dhruvil, this is a welcome improvement! My understanding is that you have done some work to validate that the change has the desired effect, it would be good to include that information in the "Testing Strategy" section. +1 (binding) Ismael On Wed, May 2, 2018 at 9:27 AM Dhruv

Re: [VOTE] KIP-244: Add Record Header support to Kafka Streams

Thanks for the KIP! +1 -Bill On Tue, May 15, 2018 at 1:47 PM, Damian Guy wrote: > Thanks. +1 (binding) > > On Tue, 15 May 2018 at 01:04 Jorge Esteban Quilcate Otoya < > quilcate.jo...@gmail.com> wrote: > > > @Guozhang added. Thanks! > > > > El mar., 15 may. 2018 a las 5:50, Matthias J. Sax (<

Re: [VOTE] KIP-244: Add Record Header support to Kafka Streams

Thanks. +1 (binding) On Tue, 15 May 2018 at 01:04 Jorge Esteban Quilcate Otoya < quilcate.jo...@gmail.com> wrote: > @Guozhang added. Thanks! > > El mar., 15 may. 2018 a las 5:50, Matthias J. Sax ( >) > escribió: > > > +1 (binding) > > > > Thanks a lot for the KIP! > > > > -Matthias > > > > On 5/1

Re: [VOTE] KIP-255: OAuth Authentication via SASL/OAUTHBEARER

Hi, Ron, Thanks for the reply. I understood your answers to #2 and #3. For #1, will the server map all clients' principal name to the value associated with "sub" claim? How do we support mapping different clients to different principal names? Jun On Mon, May 14, 2018 at 7:02 PM, Ron Dagostino

use orther encoding

hello! I want to use kafka to produce some message in GBK encoding. what should i do? wuy...@boco.com.cn 发件人：吴芋努

Re: [DISCUSS] KIP-285: Connect Rest Extension Plugin

Randall- I think I have addressed all the comments. Let me know if we can take this to Vote. Thanks Magesh On Tue, May 8, 2018 at 10:12 PM, Magesh Nandakumar wrote: > Hi All, > > I have updated the KIP to reflect changes based on the PR > https://github.com/apache/kafka/pull/4931. Its mostly ha

Re: [DISCUSS] KIP-290: Support for wildcard suffixed ACLs

Hi Piyush, I think AclBinding should operate the same way as AclBindingFilter. So you should be able to do something like this: > AclBindingFilter filter = new AclBindingFiler(new > ResourceFilter(ResourceType.GROUP, "foo*")) > AclBinding binding = new AclBinding(new Resource(ResourceType.GROUP,

Re: [DISCUSS] KIP-302 - Enable Kafka clients to use all DNS resolved IP addresses

Hi Edo, I agree that KIP-235 and KIP-302 address different scenarios. And I agree that each one is not sufficient in itself to address both the scenarios. But I also think that they conflict and hence they need to be looked at together and perhaps use a single config. As an example: If I run: f

Kafka Elasticsearch Connector

Hi Team, Am new to this tech. I need to connect Kafka to Elastic search using Windows(To consume the data from Kafka with the help of Elastic search). I have tried so many methods but I can't finish this one. Is there any way to do this in windows or direct me to the respective place. Thanks in

Re: [DISCUSS] Apache Kafka 2.0.0 Release Plan

Hi all, This is just a reminder that KIP freeze for 2.0.0 release is in a week's time and we still have a lot of KIPs in progress. KIPs that are currently being discussed should start the voting process soon to get voting complete by 22nd of May. Please participate in discussions and votes to enab

Jenkins build is back to normal : kafka-trunk-jdk8 #2645

See

Re: [VOTE] KIP-277 - Fine Grained ACL for CreateTopics API

Hi, bumping the thread as the current vote count for this KIP is 2 binding +1 5 non-binding +1 thanks, Edo On 8 May 2018 at 16:14, Edoardo Comar wrote: > Hi, > bumping the thread as the current vote count for this KIP is > 2 binding +1 > 5 non-binding +1 > > so still missing a binding vote pleas

Re: [VOTE] KIP-244: Add Record Header support to Kafka Streams

@Guozhang added. Thanks! El mar., 15 may. 2018 a las 5:50, Matthias J. Sax () escribió: > +1 (binding) > > Thanks a lot for the KIP! > > -Matthias > > On 5/14/18 10:17 AM, Guozhang Wang wrote: > > +1 from me > > > > One more comment on the wiki: while reviewing the PR I realized that in ` > > Moc

Jenkins build is back to normal : kafka-trunk-jdk10 #106

See