Re: [DISCUSS] KIP-507: Securing Internal Connect REST Endpoints

2019-08-28 Thread Greg Harris
Hey Chris, The KIP makes sense to me, and I think it's a very natural way to solve the issue at hand. I had two questions about the automatic rollout logic. Does this roll-out ratchet the protocol version irreversibly? What is the expected behavior when the feature has been enabled, and a worker

Request to be added as a contributor

2019-09-09 Thread Greg Harris
Hi, I'd like to contribute to Kafka, and would like permissions for the following accounts: ASF Jira: gharris1727 Confluence: gharris1727 Thanks! Greg Harris

Re: [VOTE] KIP-949: Add flag to enable the usage of topic separator in MM2 DefaultReplicationPolicy

2023-07-25 Thread Greg Harris
Hey Omnia, Thanks for the KIP! I think that MM2 is responsible for providing an upgrade path for users, even if it isn't backwards-compatible by default due to a mistake. The non-configuration-based strategies I could think of aren't viable due to the danger of inferring the incorrect topic name,

Re: [VOTE] KIP-959 Add BooleanConverter to Kafka Connect

2023-07-26 Thread Greg Harris
Hey Hector, Thanks for the straightforward and clear KIP! +1 (binding) Thanks, Greg On Wed, Jul 26, 2023 at 5:16 AM Chris Egerton wrote: > > +1 (binding) > > Thanks Hector! > > On Wed, Jul 26, 2023 at 3:18 AM Kamal Chandraprakash < > kamal.chandraprak...@gmail.com> wrote: > > > +1 (non-binding)

Re: Flaky tests need attention (clients, Connect, Mirror Maker, Streams, etc.)

2023-08-08 Thread Greg Harris
Hey all, MirrorConnectorsIntegration*Test#testOffsetTranslationBehindReplicationFlow() flakiness should be addressed by https://github.com/apache/kafka/pull/14156 which is currently in review. There is other flakiness in the suite which this PR does not address and needs further investigation. Th

Re: [VOTE] KIP-953: partition method to be overloaded to accept headers as well.

2023-08-23 Thread Greg Harris
Hey Jack, The design of this KIP is also consistent with the way header support was added to Connect: https://cwiki.apache.org/confluence/display/KAFKA/KIP-440%3A+Extend+Connect+Converter+to+support+headers I think making argument for precedent here is reasonable. Hi Ismael, Can you expand what

Re: Please provide permission to add reviewer in github PR

2023-08-25 Thread Greg Harris
Hi Arpit, Contributors aren't typically given permissions to tag reviewers in the kafka repository. Instead, you should @-mention potential reviewers in a github comment. Hope this helps! Greg On Fri, Aug 25, 2023 at 9:15 AM Arpit Goyal wrote: > > Hi All, > I just generated a patch for one of t

Re: Please provide permission to add reviewer in github PR

2023-08-25 Thread Greg Harris
ri, Aug 25, 2023 at 9:28 AM Arpit Goyal wrote: > > Sure I tagged in the jira , will also comment in GitHub. > Do we maintain a list of issues which beginner can start picking from? > > On Fri, Aug 25, 2023, 21:49 Greg Harris > wrote: > > > Hi Arpit, > > > > C

Re: [VOTE] KIP-953: partition method to be overloaded to accept headers as well.

2023-08-26 Thread Greg Harris
it easy to add new fields to `NewTopic` or `CreateTopicsOptions`. > > Ismael > > On Wed, Aug 23, 2023 at 11:48 AM Greg Harris > wrote: > > > Hey Jack, > > > > The design of this KIP is also consistent with the way header support > > was added to Conn

Re: [VOTE] KIP-953: partition method to be overloaded to accept headers as well.

2023-08-26 Thread Greg Harris
fka/commit/1d22b0d70686aef5689b775ea2ea7610a37f3e8c > > Ismael > > On Sat, Aug 26, 2023 at 8:29 AM Greg Harris > wrote: > > > Hey Ismael, > > > > Thank you for clarifying where the DTO pattern is used already, I did > > not have the admin methods i

Re: Disabling Test: org.apache.kafka.trogdor.coordinator.CoordinatorTest.testTaskRequestWithOldStartMsGetsUpdated()

2023-08-28 Thread Greg Harris
Hey Sagar, The JIRA for this flaky test is here: https://issues.apache.org/jira/browse/KAFKA-8115 Rather than disabling the test, I think we should look into the cause of the flakiness. Thanks! Greg On Mon, Aug 28, 2023 at 2:49 AM Sagar wrote: > > Hi All, > > Should we disable this test: > org

Re: [VOTE] KIP-965: Support disaster recovery between clusters by MirrorMaker

2023-08-29 Thread Greg Harris
Hi hudeqi, Thanks for the KIP! I think the original behavior (removing WRITE permissions during the sync) is a good default, but is not acceptable in every situation. I think providing a configuration for this behavior is the right idea. I had a few questions: 1. Is this configuration only relev

Re: Unable to start the Kafka with Kraft in Windows 11

2023-09-05 Thread Greg Harris
Hey Sumanshu, Thanks for trying out Kraft! I hope that you can get it working :) I am not familiar with Kraft or Windows, but the error appears to mention that the file is already in use by another process so maybe we can start there. 1. Have you verified that no other Kafka processes are runnin

Re: Apache Kafka 3.6.0 release

2023-09-06 Thread Greg Harris
Hey all, A user just experienced this problem that has already been reported for several versions and has an open PR: https://issues.apache.org/jira/browse/KAFKA-14273 https://github.com/apache/kafka/pull/12763 Impact: In KRaft mode on Windows, Kafka crashes on startup with an IOException. Does

Re: [VOTE] KIP-970: Deprecate and remove Connect's redundant task configurations endpoint

2023-09-06 Thread Greg Harris
Hey Yash, +1(binding) Thanks for the KIP! Greg On Wed, Sep 6, 2023 at 6:59 AM Yash Mayya wrote: > > Hi all, > > I just wanted to bump up this vote thread. Thanks to everyone who's voted > so far - we have 1 binding +1 vote and 3 non-binding +1 votes so far. > > Thanks, > Yash > > On Wed, Aug 30

Re: Unable to start the Kafka with Kraft in Windows 11

2023-09-06 Thread Greg Harris
l/12763 > > -- > Ziming > > > On Sep 6, 2023, at 07:25, Greg Harris wrote: > > > > Hey Sumanshu, > > > > Thanks for trying out Kraft! I hope that you can get it working :) > > > > I am not familiar with Kraft or Windows, but the error appears

Re: Apache Kafka 3.6.0 release

2023-09-18 Thread Greg Harris
Hi Satish, While validating 3.6.0-rc0, I noticed this regression as compared to 3.5.1: https://issues.apache.org/jira/browse/KAFKA-15473 Impact: The `connector-plugins` endpoint lists duplicates which may cause confusion for users, or poor behavior in clients. Using the other REST API endpoints a

Re: [VOTE] 3.6.0 RC0

2023-09-18 Thread Greg Harris
Hey all, I noticed this regression in RC0: https://issues.apache.org/jira/browse/KAFKA-15473 I've mentioned it in the release thread, and I'm working on a fix. I'm -1 (non-binding) until we determine if this regression is a blocker. Thanks! On Mon, Sep 18, 2023 at 10:56 AM Josep Prat wrote: >

Re: Apache Kafka 3.6.0 release

2023-09-18 Thread Greg Harris
pened a PR here: https://github.com/apache/kafka/pull/14398 and I'll work to get it merged promptly. Thanks! On Mon, Sep 18, 2023 at 11:54 AM Greg Harris wrote: > > Hi Satish, > > While validating 3.6.0-rc0, I noticed this regression as compared to > 3.5.1: https://issues

Re: [kafka-clients] [VOTE] 3.6.0 RC1

2023-09-20 Thread Greg Harris
Hi all, I verified the functionality of KIP-898 and the recent fix for KAFKA-15473 with the following steps: 1. I started a 3.5.1 broker, and a 3.5.1 worker with most (>400) publicly available plugins installed 2. I captured the output of /connector-plugins 3. I upgraded the worker to 3.6.0-rc1 4

Re: [ANNOUNCE] New committer: Yash Mayya

2023-09-21 Thread Greg Harris
Congratulations Yash! Glad to have you :) On Thu, Sep 21, 2023 at 8:44 AM Boudjelda Mohamed Said wrote: > > Congratulations Yash Mayya > > > On Thu 21 Sep 2023 at 17:28, Bruno Cadonna wrote: > > > Hi all, > > > > The PMC of Apache Kafka is pleased to announce a new Kafka committer > > Yash Mayya

Re: [ANNOUNCE] New Kafka PMC Member: Justine Olshan

2023-09-23 Thread Greg Harris
Congratulations Justine! On Sat, Sep 23, 2023 at 5:49 AM Boudjelda Mohamed Said wrote: > > Congrats Justin ! > > On Sat 23 Sep 2023 at 14:44, Randall Hauch wrote: > > > Congratulations, Justine! > > > > On Sat, Sep 23, 2023 at 4:25 AM Kamal Chandraprakash < > > kamal.chandraprak...@gmail.com> wr

[DISCUSS] KIP-986: Cross-Cluster Replication

2023-10-02 Thread Greg Harris
desirable, but I don't have an implementation in mind that meets these requirements. If you have additional requirements, an alternative UX in mind, or wish to propose some implementation details, please edit the KIP with your contributions. Thanks everyone! Greg Harris Aiven, Inc

Re: Re: [DISCUSS] Cluster Linking / Cross-Cluster Replication - Call for Interest & Co-Authors

2023-10-02 Thread Greg Harris
this feature, and make edits to the KIP directly when you feel it is appropriate. Thanks! Greg Harris On Mon, May 22, 2023 at 4:59 AM Andrew Otto wrote: > > Hello! > > The Wikimedia Foundation would be very interested in this feature. It > would make deployment of multi DC/region

Re: [VOTE] 3.6.0 RC2

2023-10-02 Thread Greg Harris
Hey Satish, I verified KIP-898 functionality and the KAFKA-15473 patch. +1 (non-binding) Thanks! On Mon, Oct 2, 2023 at 1:28 PM Justine Olshan wrote: > > Hey all -- I noticed we still have the system tests as something that will > be updated. Did we get a run for this RC? > > On Mon, Oct 2, 202

Re: [DISCUSS] KIP-986: Cross-Cluster Replication

2023-10-02 Thread Greg Harris
ng? If the topic ids can differ then what is > responsible for the mapping and rewriting of metadata records which include > topic ids? > D2. "The network path between Kafka clusters is assumed to be less reliable > than the intra-cluster network," we should be explicit about wh

Re: [DISCUSS] KIP-986: Cross-Cluster Replication

2023-10-03 Thread Greg Harris
> servers to a different cluster. A layer above Kafka or a proxy can solve > this by abstracting away the cluster itself. It could force out a metadata > refresh and from that point on clients can fetch from the other cluster. Is > this problem within the scope of this KIP or not? &g

Re: Permission to Create KIP

2023-10-03 Thread Greg Harris
Hey Colt, You should be all set. Looking forward to the KIP! Greg On Tue, Oct 3, 2023 at 11:37 AM Colt McNealy wrote: > > Hello there, > > Could I please have access to create a Wiki page? A team member and I would > like to jointly propose a small KIP. > > JIRA id: coltmcnealy-lh > > Thank you

Re: [DISCUSS] KIP-986: Cross-Cluster Replication

2023-10-04 Thread Greg Harris
e bigger coordination between clusters than in > pre-TS Kafka. Do you think we should take this into consideration in the > design and in the UX? > > Thanks, > Viktor > > On Tue, Oct 3, 2023 at 6:30 PM Greg Harris > wrote: > > > Hi Viktor, > > > > Than

Re: [DISCUSS] KIP-986: Cross-Cluster Replication

2023-10-05 Thread Greg Harris
collects > this information. It could also serve as an admin endpoint (swagger maybe?) > for managing flows and configuration. With this you could instruct clusters > to create/pause/delete replications. What do you think? > > Thanks, > Viktor > > > > On Wed, Oct

[DISCUSS] KIP-987: Connect Static Assignments

2023-10-05 Thread Greg Harris
tems other than Strimzi. Find the proposal here: https://github.com/strimzi/proposals/pull/96 Thanks! Greg Harris

Re: [VOTE] KIP-976: Cluster-wide dynamic log adjustment for Kafka Connect

2023-10-06 Thread Greg Harris
Hey Chris, Thanks for the KIP! I think that preserving the ephemeral nature of the logging change is the right choice here, and using the config topic for intra-cluster broadcast is better than REST forwarding. +1 (binding) Thanks, Greg On Fri, Oct 6, 2023 at 9:05 AM Chris Egerton wrote: > > H

Re: Re: [DISCUSS] KIP-986: Cross-Cluster Replication

2023-10-07 Thread Greg Harris
Hello hudeqi, I apologize if the KIP and discussion have diverged, as I've been trying to add detail rather than propose changes. > Why can't we use the follower fetch protocol? What you've described sounds like a very reasonable implementation of CCR. I purposely have not specified any implemen

Re: Re: [DISCUSS] Cluster Linking / Cross-Cluster Replication - Call for Interest & Co-Authors

2023-10-09 Thread Greg Harris
Hey all, I think it may be helpful to discuss the KIP draft more informally on the ASF slack #kafka channel. I've invited those of you who responded above, and for anyone else who wants an invite please just let me know on this thread. Thanks! Greg On Mon, Oct 2, 2023 at 1:39 PM Greg H

Re: [VOTE] KIP-980: Allow creating connectors in a stopped state

2023-10-09 Thread Greg Harris
Thanks Yash for the well written KIP! And thank you for finally adding JSON support to the standalone mode that isn't file extension sensitive. That will be very useful. +1 (binding) On Mon, Oct 9, 2023 at 10:45 AM Knowles Atchison Jr wrote: > > This is super useful for pipeline setup! > > +1 (

Re: [DISCUSS] KIP-987: Connect Static Assignments

2023-10-10 Thread Greg Harris
tasks. So without deeper knowledge of a particular connector it's > > not clear to the user/operator how to configure their static workers and > > static assignments. > > 5. Is there a lurking assumption that task indices are stable? E.g. that > > the task with index 3 will always

Re: [DISCUSS] KIP-987: Connect Static Assignments

2023-10-10 Thread Greg Harris
n a future KIP. As for this proposal, because the management layer is responsible for placing workers and placing tasks on those workers, it would also be responsible for implementing rack-awareness in isolated JVMs. Thanks a lot! Greg On Tue, Oct 10, 2023 at 10:08 AM Greg Harris wrote: > > H

Re: Requesting to be added to Kafka project

2023-10-12 Thread Greg Harris
Hey Alyssa, Thanks for considering opening a KIP! I've given you the necessary permissions to create a KIP, let me know if you have any more issues. Thanks! Greg On Thu, Oct 12, 2023 at 3:22 PM Alyssa Huang wrote: > > Can I also have edit/write permission for Confluence to start a KIP? My > use

Re: [DISCUSS] KIP-987: Connect Static Assignments

2023-10-18 Thread Greg Harris
e, what happens when one of the workers is restarted or is being > rolled? Would the assignment eventually be sticky when the worker comes > back? Does this new protocol also abide by the scheduled rebalance delays? > Maybe this has been explained in the KIP but it wasn't clear to me when

Re: [DISCUSS] KIP-987: Connect Static Assignments

2023-10-20 Thread Greg Harris
rs by the administrators as > they also operate the management system. > > When a connector is created, the runtime tries to place tasks on the > available brokers by matching the placement and tags. If no suitable > workers are found, the tasks stay in unassigned state and the runtim

Re: [DISCUSS] KIP-987: Connect Static Assignments

2023-10-20 Thread Greg Harris
f context would prevent a custom assignor from solving the resource utilization problem adequately. Thanks! Greg On Fri, Oct 20, 2023 at 9:58 AM Greg Harris wrote: > > Mickael, > > Thank you for discussing that rejected alternative, I was almost going > to propose it. > > > I

Development with Git Worktree

2023-10-23 Thread Greg Harris
Hey Kafka Developers, This is a small announcement that the gradle build now supports the git-worktree subcommand [1] on the 3.4, 3.5, 3.6, and trunk branches [2]. If you've needed to check out multiple copies of Kafka concurrently, you previously needed to manage multiple full clones of the repo

Re: [VOTE] KIP-967: Support custom SSL configuration for Kafka Connect RestServer

2023-10-23 Thread Greg Harris
Hey Taras, Thanks for the KIP! The design you propose follows the conventions started in KIP-519, and should feel natural to operators familiar with the broker feature. I also like that we're able to clean up some connect-specific functionality and make the codebase more consistent. +1 (binding)

Re: KIP-993: Allow restricting files accessed by File and Directory ConfigProviders

2023-11-08 Thread Greg Harris
Hey Tina, Thanks for the KIP! Unrestricted file system access over a REST API is an unfortunate anti-pattern, so I'm glad that you're trying to change it. I had a few questions, mostly from the Connect perspective. 1. In the past Connect removed the FileStream connectors in order to prevent a RES

Re: [DISCUSS] Apache Kafka 3.5.2 release

2023-11-10 Thread Greg Harris
Hey Luke, I merged and backported KAFKA-15800 to 3.5 and 3.6, sorry for the delay. Thanks for running the release! Greg On Thu, Nov 9, 2023 at 6:54 PM Luke Chen wrote: > > Hi all, > > Greg found a regression issue in Kafka connect: > https://issues.apache.org/jira/browse/KAFKA-15800 > I'll wait

[DISCUSS] KIP-1006: Remove SecurityManager Support

2023-11-20 Thread Greg Harris
27;re using it, and how you're planning around its removal. Thanks! Greg Harris

Re: [DISCUSS] KIP-1006: Remove SecurityManager Support

2023-11-20 Thread Greg Harris
ache Kafka. > When we drop support for JDK 17, will we set the “-target” option of Javac > such that the resulting JARs will not load in JVMs which are lesser than or > equal to that version ? > > Thanks, > Ashwin > > > On Tue, Nov 21, 2023 at 6:18 AM Greg Harris > wr

Re: [VOTE] KIP-477: Add PATCH method for connector config in Connect REST API

2024-05-01 Thread Greg Harris
Hi Ivan, Thank you for the KIP! I think PATCH using the same return type as PUT benefits the clients, even though the "created" field will always be false. +1 (binding) Thanks, Greg On Fri, Apr 12, 2024 at 4:31 AM Yash Mayya wrote: > > Hi Ivan, > > Thanks for reviving this KIP, I think it will

Re: DISCUSS KIP-984 Add pluggable compression interface to Kafka

2024-05-01 Thread Greg Harris
for example. > > > > 4. Passing the class name as metadata is one way to have the producer > > talk to the broker about which plugin to use. However there could be > > other implementation > >where you could set every thing to know about the topic using topic >

Re: [DISCUSS] KIP-1042 support for wildcard when creating new acls

2024-05-06 Thread Greg Harris
Hi Murali, Thanks for the KIP! I think I understand the motivation for this KIP in situations where there are a "cross product" of topics for two or more variables X and Y, and want to write ACLs for each of the variable axes. If you format your topics "X-Y-suffix", it's not easy to write rules t

[DISCUSS] KIP-891: Running multiple versions of Connector plugins

2024-05-09 Thread Greg Harris
converters, transforms, and predicates) not just connectors & tasks 2. Specifying a range of versions instead of an exact match 3. New metrics to observe what versions are in-use Thanks to Snehashis for the original KIP idea! Thanks, Greg On Tue, Jan 2, 2024 at 11:49 AM Greg Harris wrote: &g

Re: [DISCUSS] KIP-891: Running multiple versions of Connector plugins

2024-05-10 Thread Greg Harris
t; priorities. If it's okay with you, I would still like to get this > implemented myself, including the additional scope. > > Thanks and regards > Snehashis > > On Fri, May 10, 2024 at 12:45 AM Greg Harris > wrote: > > > Hi all, > > > > I'd li

Re: DISCUSS KIP-984 Add pluggable compression interface to Kafka

2024-05-13 Thread Greg Harris
arification on this topic. > Again, thank you for sharing your concerns about balancing this proposal > against the impact to the ecosystem. We think the additional performance that > this could provide along with the improved flexibility to add or accelerate > compression codecs outweigh

Re: [DISCUSS] KIP-891: Running multiple versions of Connector plugins

2024-05-14 Thread Greg Harris
gured constraints.) > > 6) (Nit) It'd be nice to add a link to the TestPlugins class or somewhere > in its neighborhood to the testing plan; unfamiliar readers probably won't > get much out of what's there right now. > > [1] - > https://cwiki.apache.org/confluence

Re: [DISCUSS] KIP-891: Running multiple versions of Connector plugins

2024-05-15 Thread Greg Harris
t heinous, but will it be possible to configure > a connector to use two instances of the same transform or predicate, but > with different versions for each? (I don't think this is worth significant > design/implementation effort, so if it would inflate either of those, > please don

Re: please let me subscribe

2024-05-16 Thread Greg Harris
Hi Harry, Thanks for your interest in Apache Kafka. You can see how to subscribe to the mailing list(s) here: https://kafka.apache.org/contact Please note that the email for subscribing/unsubscribing is different from the one used for sending and receiving emails once you're on the list. Also, sub

Re: [DISCUSS] KIP-1042 support for wildcard when creating new acls

2024-05-21 Thread Greg Harris
the estimation can be skipped and the cardinality value of each > > Bloom filter can be used instead. You can then check the smaller filters > > against the larger ones and find where one candidate is contained within > > another. > > > > If you want to know if they

Re: [DISCUSS] KIP-891: Running multiple versions of Connector plugins

2024-05-22 Thread Greg Harris
is information somewhere directly accessible without > having to provide a full connector config. FWIW I'd be fine with GET > /connector-plugins//versions as a first-class endpoint either > instead of or in addition to adding recommended values for all plugin > versions. > &

Action requested: Changes to CI for JDK 11 & 17 builds on Pull Requests

2024-05-27 Thread Greg Harris
Bees) should no longer be present. Thanks everyone, Greg Harris

Re: [DISCUSS] KIP-891: Running multiple versions of Connector plugins

2024-05-29 Thread Greg Harris
ugins, discover the set of available versions by > examining the recommended values for the "version" property after hitting > the GET GET /connector-plugins//config endpoint > > I don't think this is necessarily optimal since plucking the "version" > property out

Re: standalone race condition?

2024-05-29 Thread Greg Harris
Hi John, Thanks for your question! I wasn't able to identify anything obviously wrong from your logfile and visible configurations. REST timeouts are something that we have ongoing problems with, and they're usually caused by blocking plugins. You can capture some additional diagnostics with `js

Re: [DISCUSS] KIP-1054: Support external schemas in JSONConverter

2024-06-07 Thread Greg Harris
Hi Priyanka, Thanks for the KIP! I think you captured the motivation well: The Converter interface on its own implies a fairly large raw message size, and the ecosystem's strategies for deduplicating schema information are complex. I did have some questions/concerns. 1. Have you done a comparison

Re: [VOTE] KIP-1040: Improve handling of nullable values in InsertField, ExtractField, and other transformations

2024-06-11 Thread Greg Harris
Hi Mario, Thanks for the KIP. I'm on-board with this KIP, I just wanted to verify a discrepancy I noticed. I checked all of the call-sites of Struct#get(Field) and Struct#get(String) in Kafka, and noticed there are some call-sites which are not included in the KIP. 1. The Flatten transformation s

Re: [VOTE] KIP-1040: Improve handling of nullable values in InsertField, ExtractField, and other transformations

2024-06-12 Thread Greg Harris
e KIP to also other > potential affected SMTs. > > As of now the PR scope, instead, is only for InsertField and ExtractField > SMTs. > > Thanks, > Mario. > > > On Wed, Jun 12, 2024 at 12:08 AM Greg Harris > > wrote: > > > Hi Mario, > > > > Th

Re: [DISCUSS] KIP-1056: Remove `default.` prefix for exception handler StreamsConfig

2024-06-12 Thread Greg Harris
Hi Murali, Thanks for the KIP! I'm not familiar with Streams so I'll pose a general question, open for anyone to answer: The configs that are being changed don't currently accept in-place overwrites in the code, so the "default.*" prefix doesn't make sense. Could there be a KIP to accept in-plac

Re: Subscribe for discussion on Kafka(R) development

2024-06-12 Thread Greg Harris
Hi Punsak, Please find the instructions for subscribing to this mailing list here: https://kafka.apache.org/contact and here: https://www.apache.org/foundation/mailinglists Thanks, Greg On Wed, Jun 12, 2024 at 7:11 AM Punsak Incham wrote: > I want to subscribe for discussion on Kafka® developm

Re: [VOTE] KIP-1040: Improve handling of nullable values in InsertField, ExtractField, and other transformations

2024-06-13 Thread Greg Harris
fore closing > the vote? > > Good catch. I have just updated the KIP. > > I think that we can close the voting and mark it as approved, right? > > Thank you all. > > On Wed, Jun 12, 2024 at 7:21 PM Greg Harris > wrote: > > > Hi Mario, > > > > Than

Re: [DISCUSS] Apache Kafka 3.8.0 release

2024-06-13 Thread Greg Harris
Hi Sophie and Justine, I share your concerns about delaying 3.8 in order to give the Kraft KIPs more time for implementation. I raised them in the discussion for KIP-1012 [1]: > I think there is a > risk that features that are on-time and eligible for a 3.8 release > could be delayed by some KIPs

Re: [DISCUSS] Apache Kafka 3.8.0 release

2024-06-13 Thread Greg Harris
X Bogan > wrote: > > > > is this where I ask > > to join? > > > > On Thu, Jun 13, 2024 at 1:20 PM Greg Harris > > > wrote: > > > > > Hi Sophie and Justine, > > > > > > I share your concerns about delaying 3.8 in order t

Re: [VOTE] KIP-1017: A health check endpoint for Kafka Connect

2024-06-14 Thread Greg Harris
Hey Chris, Thanks for the KIP. I think demonstrating the health of the tick thread through a longer request is going to be more reliable than caching the health and status in order to respond quickly to requests. It should also be a natural negative feedback loop, so that when the herder tick thr

Re: Herder#requestTaskReconfiguration

2024-06-24 Thread Greg Harris
Hi John, Thanks for the question! That method is used when the Connector notifies the runtime it observed an event that would require a new set of task configs. For example, the MirrorSourceConnector calls ConnectorContext#requestTaskReconfiguration() when a new topic appears in the input cluster

Re: [DISCUSS] KIP-891: Running multiple versions of Connector plugins

2024-07-01 Thread Greg Harris
> returned from the GET /connector-plugins//config endpoint. > > > > With those two points settled, I think the only unresolved item is the > > small change to version parsing added to the KIP (where raw version > numbers > > are treated as an exact match, instead of a b

Re: [DISCUSS] KIP-1006: Remove SecurityManager Support

2024-07-01 Thread Greg Harris
g the timelines and that we only drop support for Java > versions in major Kafka releases, I think the proposed approach of > detecting the APIs to use makes sense. > > Thanks, > Mickael > > On Tue, Nov 21, 2023 at 8:38 AM Greg Harris > wrote: > > > > Hey Ashwin,

Re: [DISCUSS] KIP-1006: Remove SecurityManager Support

2024-07-03 Thread Greg Harris
hanks, Greg On Wed, Jul 3, 2024 at 9:39 AM Frédérik Rouleau wrote: > Hi all, > > When this KIP is intended to be implemented? As KIP-1013 is deprecating > Java 11 in AK 3.7 and removes its support in AK 4.0, maybe the KIP needs an > update. > > Regards, > > On Mon, Ju

Re: [DISCUSS] KIP-1006: Remove SecurityManager Support

2024-07-03 Thread Greg Harris
anks for the KIP. I'm not totally clear on why we need a KIP. Can we not > use the SecurityManager when it's available and fallback when it's not? If > so, then it would mean that whether SecurityManager is used or not depends > on the JDK and its configuration. > >

Re: [DISCUSS] KIP-1006: Remove SecurityManager Support

2024-07-05 Thread Greg Harris
lback when it's not? If >> so, then it would mean that whether SecurityManager is used or not depends >> on the JDK and its configuration. >> >> Ismael >> >> On Mon, Nov 20, 2023 at 4:48 PM Greg Harris > > >> wrote: >> >> > Hi all,

Re: [DISCUSS] KIP-1006: Remove SecurityManager Support

2024-07-05 Thread Greg Harris
Also, I've opened a PR with the proposed reflective shim, for anyone that is interested: https://github.com/apache/kafka/pull/16522 Thanks, Greg On Fri, Jul 5, 2024 at 3:17 PM Greg Harris wrote: > Hi all, > > I've added Ismael's kip-less idea as a rejected alterna

Re: [DISCUSS] KIP-1032: Upgrade to Jakarta and JavaEE 9 in Kafka 4.0

2024-07-08 Thread Greg Harris
gt; > credit that person as you mentioned. If I work on it I would probably > start > > over with a new branch and just use the old PR as a guide and then maybe > > figure out a way to credit the original author. There's always that > > co-author tag that could be

[VOTE]: KIP-1006: Remove SecurityManager Support

2024-07-08 Thread Greg Harris
Hi all, I'd like to call a vote on https://cwiki.apache.org/confluence/display/KAFKA/KIP-1006%3A+Remove+SecurityManager+Support Voting for this KIP would signal our intention to remove SecurityManager support in the future when the project's minimum supported version is Java 21. Th

Re: [VOTE]: KIP-1032: Upgrade to Jakarta and JavaEE 10 in Kafka 4.0

2024-07-09 Thread Greg Harris
While breaking compatibility is always undesirable, this is unfortunately a way in which one of our dependencies "leaks through" our API, and makes this a necessity. Connect and MirrorMaker2 can follow the broker's lead on setting Java 17 as the minimum version. +1 (binding) Thanks! Greg On Tue,

Re: [DISCUSS] Apache Kafka 3.8.0 release

2024-07-10 Thread Greg Harris
Hi Josep, A contributor just raised a regression [1] that I think should be addressed in 3.8.0 prior to the release. Summary: This change [2] causes multiple ERROR logs to appear during worker startup when operators install other, unrelated plugins that package the Jackson library. Severity: Work

Re: [DISCUSS] KIP-1059: Enable the Producer flush() method to clear the latest send() error

2024-07-12 Thread Greg Harris
Hi all, Alieh, thanks for the KIP! And everyone else, thanks for the robust discussion. I understand that there are situations in which users desire that the pipeline "just keep working" and skip errors. However, I question whether it is appropriate to support/encourage this behavior via inclusio

Re: [DISCUSS] KIP-1059: Enable the Producer flush() method to clear the latest send() error

2024-07-15 Thread Greg Harris
n impact. I'll let KStream folks > comment on whether an implementation that requires storing records in > memory actually violates the computational complexity on the main code > path, I just wanted to make the point that we shouldn't necessarily dismiss > API changes

Re: [DISCUSS] KIP-1059: Enable the Producer flush() method to clear the latest send() error

2024-07-15 Thread Greg Harris
. In fact, if the number of users regularly encountering RTLEs is sufficiently small, I would strongly advocate for an application-specific workaround instead of trying to fix this in Streams, or make memory buffering an optional feature in streams. Thanks, Greg On Mon, Jul 15, 2024 at 1:29 PM

Re: [DISCUSS] KIP-1059: Enable the Producer flush() method to clear the latest send() error

2024-07-15 Thread Greg Harris
it interval is 1 > > hour, I expect 3600MB of additional heap needed ... > > > > Agree, that would be ideal. On the other hand, the effort to prove that > > keeping all records in memory won't break some scenarios (and generally > > breaking on

Re: [VOTE] 3.8.0 RC1

2024-07-16 Thread Greg Harris
Hi Josep, I found this blocker regression: https://issues.apache.org/jira/browse/KAFKA-17150 Summary: Connector configurations that specified converters with aliases (e.g. JsonConverter instead of org.apache.kafka.connect.json.JsonConverter) previously worked, and in this RC they throw validation

Re: [DISCUSS] Apache Kafka 3.8.0 release

2024-07-17 Thread Greg Harris
eutschland GmbH > Alexanderufer 3-7, 10117 Berlin > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > Amtsgericht Charlottenburg, HRB 209739 B > > On Wed, Jul 10, 2024, 22:55 Greg Harris > wrote: > > > Hi Josep, > > > > A contributor just raised a regression [1]

Re: [DISCUSS] KIP-1059: Enable the Producer flush() method to clear the latest send() error

2024-07-19 Thread Greg Harris
t; >void send(ProducerRecord record, Callback callback, boolean > > throwImmediately) > > > > and if throwImmediately=false, then we behave the same way as now (return > > errors via Future and poison transaction) and if throwImmediately=true > then > > we just

Re: [DISCUSS] KIP-1059: Enable the Producer flush() method to clear the latest send() error

2024-07-22 Thread Greg Harris
send` in scenario 1 is different from the one in scenario > 2, since a part of the second one 's job has been done during > `prepare()`ing. > > > Cheers, > > Alieh > > On Sat, Jul 20, 2024 at 1:20 AM Greg Harris > wrote: > > > Hi Artem and Matthias, > >

Re: [VOTE] 3.8.0 RC3

2024-07-23 Thread Greg Harris
Hi Josep, +1 (binding) I performed the following validations: 1. I formatted a new log directory and verified that the stray log message is no longer present (KAFKA-17148) 2. I configured a connect worker with a custom plugin path and plugin.discovery=service_load and verified that plugin scannin

Re: New release branch 3.9

2024-07-30 Thread Greg Harris
Hi all, I agree that we are not yet ready for breaking changes on trunk, so I opened a PR to bump to 3.10.0-SNAPSHOT: https://github.com/apache/kafka/pull/16732 When KIP-853 is feature complete, we can bump to 4.0.0-SNAPSHOT. Thanks, Greg On Tue, Jul 30, 2024 at 10:01 AM Josep Prat wrote: > H

Re: [DISCUSS] KIP-1074: Make the replication of internal topics configurable

2024-07-30 Thread Greg Harris
Hi Patrik, Thanks for the KIP! Your motivation for this KIP is reasonable, because it is definitely possible for the ".internal" suffix to collide with real topics. It would have been nice if the original design included some mm2-specific namespace like "mm2.internal" to lessen the likelihood of

Re: New release branch 3.9

2024-07-30 Thread Greg Harris
Hi all, I'd like to clarify my understanding of the path forward, the one I voted for in KIP-1012 and what I understood to be the consensus in the 3.8.0 release thread. 1. If KIP-853 is feature-complete before October, Kafka 3.9 can be released ASAP with KIP-853. There will be no 3.10 release, an

Re: [kafka-clients] [ANNOUNCE] Apache Kafka 3.8.0

2024-07-30 Thread Greg Harris
t; Afshin > > > >> Moazami, Ahmed Najiub, Ahmed Sobeh, Akhilesh Chaganti, Almog Gavra, > > > Alok > > > >> Thatikunta, Alyssa Huang, Anatoly Popov, Andras Katona, Andrew > > > >> Schofield, Anna Sophie Blee-Goldman, Antoine Pourchet, Anton > Agest

Re: New release branch 3.9

2024-07-30 Thread Greg Harris
sense to me, > and we should not have a 3.9 release branch, and trunk should stay on > 3.9-SNAPSHOT for the time being... > > > -Matthias > > On 7/30/24 3:03 PM, Greg Harris wrote: > > Hi all, > > > > I'd like to clarify my understanding of the path forw

[DISCUSS] Mirroring the Repartition Topic: MirrorMaker2 and Streams interaction

2024-08-12 Thread Greg Harris
Hi all, We were recently working to set up a MirrorMaker2 flow to migrate a Streams job between an existing Kafka cluster and a new Kafka cluster. We noticed the following behavior: * Streams sets infinite retention on the repartition topic, and uses the AdminClient deleteRecords call to implemen

Re: [DISCUSS] KIP-891: Running multiple versions of Connector plugins

2024-08-19 Thread Greg Harris
e KIP (LMK if you want me to > take that up). > > Regards > Snehashis > > On Tue, Jul 2, 2024 at 12:08 AM Greg Harris > wrote: > > > Hey Snehashis, > > > > Sorry for the late reply, and thanks for helping close out the > discussion. > > > >

Re: [VOTE]: KIP-1006: Remove SecurityManager Support

2024-08-19 Thread Greg Harris
ecated) > already receive a warning message on startup." > Maybe we can remove it or add more words at the end? > > > Anyway, +1 (binding) from me. > > Thank you. > Luke > > On Tue, Jul 9, 2024 at 2:26 AM Greg Harris > wrote: > > > Hi all, &g

Re: Question about config.action.reload

2024-08-20 Thread Greg Harris
Hi Ran, The config.action.reload, subscribe, unsubscribe, and ConfigChangeCallback features are partially-implemented. As far as I can tell if you're using only AK connect-runtime and config providers, none of it is functional. If you have a config provider that implements TTLs, the runtime will

Re: [DISCUSS] KIP-967: Support custom SSL configuration for Kafka Connect RestServer

2023-12-04 Thread Greg Harris
Hi Chris, Thank you for your comments above. I disagree with your recommendation for a new SslEngineFactory variant/hierarchy. 1. A superinterface could be more confusing to users. Since this is an interface in `clients`, the connect-specific interface would also need to be in clients, despite be

  1   2   3   4   5   >