Re: [DISCUSS] KIP-1147: Improve consistency of command-line arguments

2025-07-09 Thread Chris Egerton
Hi Andrew, Thanks for the KIP. The mixed dash and dot syntax in --producer.config and --consumer.config has always vexed me. I just have one thought: To be honest, like Kirk, I don't love --command-config. While it is the most common flag name for specifying client properties, it seems both inacc

Re: [ANNOUNCE] New Kafka Committer: Omnia Ibrahim

2025-06-25 Thread Chris Egerton
Congrats, Omnia! On Wed, Jun 25, 2025, 08:23 Justine Olshan wrote: > Hi, > > The Project Management Committee (PMC) for Apache Kafka is pleased to > announce Omnia Ibrahim as a new Kafka committer. > > Omnia has been involved in the community since 2021 working on MirrorMaker > and clients, as w

Re: [DISCUSS] ConfigDef.ConfigKey builder and addition of deprecated information.

2025-05-29 Thread Chris Egerton
Hi Claude, As a general improvement to our config API, this sounds fine (I'm perhaps a little iffy on first-class support for deprecation instead of just adding a note to the docstring, but that's low-level enough that it can and should wait for a KIP before being discussed). However, if we're ta

Re: [DISCUSS] KIP-1173: Connect Storage Topics Sharing Across Clusters

2025-05-03 Thread Chris Egerton
Hi Pritam, I think Greg's concerns about read amplification are still unaddressed, no? I'd guess that the config topic is unlikely to experience an increase in load large enough to cause problems, but what about the status topic (especially if there's, e.g. a K8s operator constantly restarting fai

Re: Custom Kafka Connect REST Extension for Token Authentication

2025-04-22 Thread Chris Egerton
Hi Jamie, This was an issue we ran into several years ago with the basic auth extension that comes OOTB with Connect. TL;DR: there are currently two endpoints that Connect uses for inter-worker, but intra-communication (i.e., REST requests that are made spontaneously from one worker to another, in

Re: [DISCUSS] KIP-1158: Improve Connect SourceTask commit()

2025-04-08 Thread Chris Egerton
way > AbstractWorkerSourceTask invokes "commit()" currently. > > However, tasks must fail if commit() throws an exception -> +1. > Should we deal with the breaking change in another ticket (targeting the > next major release) OR should we try to include the changes in this

Re: [DISCUSS] KIP-1158: Improve Connect SourceTask commit()

2025-04-08 Thread Chris Egerton
Hi Sudesh, Thanks for the KIP. The existing commit method is... not great. I think the word "futile" you chose in the KIP is putting it lightly. The new interface looks clean, implementation seems extremely straightforward, and everything's backwards compatible. The only comment I'll leave is th

Re: [VOTE] KIP-1126: Serialize changes to Kafka with a build queue

2025-02-13 Thread Chris Egerton
Hi David, Isn't it a requirement that we leave KIP vote threads open for a minimum of 72 hours before closing them? Best, Chris On Thu, Feb 13, 2025, 10:16 David Arthur wrote: > Thanks for taking the time to vote on this KIP. Since we have the required > +1s, I'll go ahead and close the vote.

Re: [DISCUSS] KIP-1126: Serialize changes to Kafka with a build queue

2025-01-23 Thread Chris Egerton
Hi David, Thanks for the KIP, and for your ongoing work to improve Kafka CI. Two small thoughts: 1) Today, there's a convention of waiting for a green (or at least completed) CI build before merging PRs, which includes running the full test suite. This can be relaxed on a case-by-case basis but

Re: KIPs being posted to the wrong section on the wiki

2025-01-15 Thread Chris Egerton
Hi Chris, Yes, this is fine--have at it. Cheers, Other Chris On Wed, Jan 15, 2025, 19:26 Christopher Shannon < christopher.l.shan...@gmail.com> wrote: > Hey Everyone, > > Recently I've noticed a trend of new KIPs that are under discussion or > still being worked on (in draft) being posted to t

Re: [VOTE] KIP-1054: Support external schemas in JSONConverter

2024-10-15 Thread Chris Egerton
Hi Priyanka, Sorry for the delay! +1 (binding), thanks for the KIP. Cheers, Chris On Wed, Sep 4, 2024, 01:35 Priyanka K U wrote: > Hi Everyone, > > Please go through the proposal and requesting everyone to support with > your votes : > https://cwiki.apache.org/confluence/display/KAFKA/KIP-10

Re: [VOTE] KIP-891: Running multiple versions of Connector plugins

2024-10-07 Thread Chris Egerton
Hi Snehashis, Thanks for the KIP. I'm +1 (binding) but have one more non-blocking thought I wanted to share. On the off chance that an existing plugin is designed to accept a "version" property, could we either 1) keep passing that property to plugins instead of stripping it, or 2) rename our new

Re: [VOTE] KIP-1089: Allow disabling heartbeats replication in MirrorSourceConnector

2024-10-02 Thread Chris Egerton
+1 (binding), thanks for the KIP! On Wed, Oct 2, 2024, 06:47 Viktor Somogyi-Vass wrote: > +1 (binding) > > Thanks for this improvement. If you have a PR, you can ping me for review. > > On Fri, Sep 27, 2024 at 4:07 PM Dániel Urbán > wrote: > > > Hi everyone, > > > > I would like to start a vote

Re: [ANNOUNCE] New committer: Kamal Chandraprakash

2024-09-30 Thread Chris Egerton
Congrats! -- Forwarded message - From: Satish Duggana Date: Mon, Sep 30, 2024, 09:01 Subject: Re: [ANNOUNCE] New committer: Kamal Chandraprakash To: Congratulations Kamal! On Mon, 30 Sept 2024 at 18:18, Josep Prat wrote: > Congrats Kamal! > > On Mon, Sep 30, 2024 at 2:46 PM

Re: So long, Jenkins 👋

2024-09-25 Thread Chris Egerton
David, you're a legend. Thanks for spearheading this effort! On Wed, Sep 25, 2024, 20:51 David Arthur wrote: > Today, we disabled the Jenkins build on trunk. With this change, we should > now be expecting all green status checks on PRs before merging. Of course, > flaky tests still exist, but ge

Re: [DISCUSS] Apache Kafka 4.0.0 release

2024-09-23 Thread Chris Egerton
Thanks David! +1 On Mon, Sep 23, 2024, 13:07 José Armando García Sancio wrote: > +1, thanks for volunteering David! > > On Mon, Sep 23, 2024 at 11:56 AM David Arthur wrote: > > > > +1, thanks David! > > > > On Mon, Sep 23, 2024 at 11:48 AM Satish Duggana < > satish.dugg...@gmail.com> > > wrote:

[jira] [Resolved] (KAFKA-17557) Test123

2024-09-19 Thread Chris Egerton (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-17557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton resolved KAFKA-17557. --- Resolution: Invalid > Test123 > --- > > Key:

Re: [VOTE] KIP-1082: Require Client-Generated IDs over the ConsumerGroupHeartbeat RPC

2024-09-18 Thread Chris Egerton
Thanks for the KIP! +1 (binding) On Tue, Sep 17, 2024 at 12:45 PM Andrew Schofield < andrew_schofield_j...@outlook.com> wrote: > +1 (non-binding) > > > From: 吳岱儒 > Sent: 17 September 2024 04:24 > To: dev@kafka.apache.org > Subject: Re: [VOTE] KIP-1082: R

Re: [VOTE] KIP-1074: Allow the replication of user internal topics

2024-09-09 Thread Chris Egerton
Hi Patrik, Sorry for the delay, I've been on PTO. LGTM! +1 (binding) Cheers, Chris On Tue, Sep 3, 2024 at 5:10 AM Viktor Somogyi-Vass wrote: > Hi Patrik, > > Thanks for working on this. +1 from me (binding). > > Best, > Viktor > > On Wed, Aug 28, 2024 at 1:34 PM Patrik Marton > > wrote: > >

[ANNOUNCE] New Kafka PMC Member: Josep Prat

2024-09-06 Thread Chris Egerton
Hi all, Josep has been a Kafka committer since December 2022. He has remained very active and instructive in the community since then, and it's my pleasure to announce that he has accepted our invitation to become a member of the Kafka PMC. Congratulations Josep! Enjoy voting on those release can

[jira] [Created] (KAFKA-17493) Sink connector-related OffsetsApiIntegrationTest suite test cases failing more frequently with new consumer/group coordinator

2024-09-06 Thread Chris Egerton (Jira)
Chris Egerton created KAFKA-17493: - Summary: Sink connector-related OffsetsApiIntegrationTest suite test cases failing more frequently with new consumer/group coordinator Key: KAFKA-17493 URL: https

[jira] [Resolved] (KAFKA-15935) Flaky test: testRestartReplication() – org.apache.kafka.connect.mirror.integration.IdentityReplicationIntegrationTest

2024-08-06 Thread Chris Egerton (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton resolved KAFKA-15935. --- Fix Version/s: 4.0.0 Resolution: Fixed > Flaky test: testRestartReplicat

[jira] [Resolved] (KAFKA-16919) Flaky test testNoCheckpointsIfNoRecordsAreMirrored() – org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationSSLTest

2024-08-06 Thread Chris Egerton (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton resolved KAFKA-16919. --- Fix Version/s: 4.0.0 Resolution: Fixed > Flaky t

[jira] [Resolved] (KAFKA-15933) Flaky test: testRestartReplication() – org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationTransactionsTest

2024-08-06 Thread Chris Egerton (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton resolved KAFKA-15933. --- Fix Version/s: 4.0.0 Resolution: Fixed > Flaky test: testRestartReplicat

[jira] [Resolved] (KAFKA-10812) Unclean worker shutdown in MirrorConnectorsIntegrationTest

2024-08-06 Thread Chris Egerton (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-10812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton resolved KAFKA-10812. --- Fix Version/s: 4.0.0 Resolution: Fixed > Unclean worker shutdown

[jira] [Resolved] (KAFKA-15524) Flaky test org.apache.kafka.connect.integration.OffsetsApiIntegrationTest.testResetSinkConnectorOffsetsZombieSinkTasks

2024-08-06 Thread Chris Egerton (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton resolved KAFKA-15524. --- Fix Version/s: 4.0.0 Resolution: Fixed > Flaky t

[jira] [Resolved] (KAFKA-15892) Flaky test: testAlterSinkConnectorOffsets – org.apache.kafka.connect.integration.OffsetsApiIntegrationTest

2024-08-06 Thread Chris Egerton (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton resolved KAFKA-15892. --- Fix Version/s: 4.0.0 Resolution: Fixed > Flaky test: testAlterSinkConnectorOffs

[jira] [Resolved] (KAFKA-15914) Flaky test - testAlterSinkConnectorOffsetsOverriddenConsumerGroupId - OffsetsApiIntegrationTest

2024-08-06 Thread Chris Egerton (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton resolved KAFKA-15914. --- Fix Version/s: 4.0.0 Resolution: Fixed > Flaky t

[jira] [Resolved] (KAFKA-15918) Flaky test - OffsetsApiIntegrationTest.testResetSinkConnectorOffsets

2024-08-06 Thread Chris Egerton (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton resolved KAFKA-15918. --- Fix Version/s: 4.0.0 Resolution: Fixed > Flaky t

[jira] [Resolved] (KAFKA-15197) Flaky test MirrorConnectorsIntegrationExactlyOnceTest.testOffsetTranslationBehindReplicationFlow()

2024-08-06 Thread Chris Egerton (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton resolved KAFKA-15197. --- Fix Version/s: 4.0.0 (was: 3.9.0) Resolution: Fixed > Fl

[jira] [Resolved] (KAFKA-15292) Flaky test IdentityReplicationIntegrationTest#testReplicateSourceDefault()

2024-08-06 Thread Chris Egerton (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton resolved KAFKA-15292. --- Fix Version/s: 4.0.0 Resolution: Fixed > Flaky t

[jira] [Resolved] (KAFKA-15072) Flaky test MirrorConnectorsIntegrationExactlyOnceTest.testReplicationWithEmptyPartition

2024-08-06 Thread Chris Egerton (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton resolved KAFKA-15072. --- Fix Version/s: 4.0.0 Resolution: Fixed > Flaky t

[jira] [Resolved] (KAFKA-14453) Flaky test suite MirrorConnectorsWithCustomForwardingAdminIntegrationTest

2024-08-06 Thread Chris Egerton (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-14453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton resolved KAFKA-14453. --- Fix Version/s: 4.0.0 Resolution: Fixed > Flaky test su

[jira] [Resolved] (KAFKA-15945) Flaky test - testSyncTopicConfigs() – org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationBaseTest

2024-08-06 Thread Chris Egerton (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton resolved KAFKA-15945. --- Fix Version/s: 4.0.0 Resolution: Fixed > Flaky test - testSyncTopicConf

Re: Re: [DISCUSS] KIP-1074: Make the replication of internal topics configurable

2024-08-06 Thread Chris Egerton
y 1 and this would bypass the filters and > the replication policy. Since forcing is usually not the normal way of > doing things in general, users may be more willing to accept unintended > side effects. > > What do you all think? > (If they aren't better than the original

[jira] [Resolved] (KAFKA-4400) Prefix for sink task consumer groups should be configurable

2024-08-06 Thread Chris Egerton (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-4400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton resolved KAFKA-4400. -- Resolution: Won't Do > Prefix for sink task consumer groups should be conf

Re: Re: [DISCUSS] KIP-1074: Make the replication of internal topics configurable

2024-08-05 Thread Chris Egerton
ried to come up with a bit different regex based solution, and modified > the KIP based on that. I think it is a more straightforward way to get the > desired behavior. > > Let me know your thoughts. > > Thanks, > Patrik > > > > On 2024/07/30 18:57:18 Chris Egerton

[jira] [Created] (KAFKA-17252) Forwarded source task zombie fencings may fail when leader has just started

2024-08-02 Thread Chris Egerton (Jira)
Chris Egerton created KAFKA-17252: - Summary: Forwarded source task zombie fencings may fail when leader has just started Key: KAFKA-17252 URL: https://issues.apache.org/jira/browse/KAFKA-17252

Re: [DISCUSS] Graduation steps for Features

2024-07-31 Thread Chris Egerton
barrier to entry is fair, but I think this really only > >> applies to > >> a handful of large KIPs which take several releases to come to fruition. > >> I’m sure we didn’t know when KRaft began which release would be the > first > >> in which it would be

Re: New release branch 3.9

2024-07-31 Thread Chris Egerton
Hi Colin, Would it be alright if I cherry-picked https://github.com/apache/kafka/pull/16678 onto 3.9? This isn't a regression but it's a low-risk fix that I'd like to get into the next release if possible. Cheers, Chris On Wed, Jul 31, 2024 at 12:29 AM Ismael Juma wrote: > I would recommend a

Re: [DISCUSS] Graduation steps for Features

2024-07-31 Thread Chris Egerton
I'd be hesitant to add too much to the KIP template. We should be careful to keep the barrier to entry low for new users. On Wed, Jul 31, 2024, 13:24 Kirk True wrote: > Hi Josep, > > Thanks for starting this conversation! > > Yes, for KIPs that span multiple releases, clearly defining the KIP’s

Re: [DISCUSS] KIP-1074: Make the replication of internal topics configurable

2024-07-30 Thread Chris Egerton
Hi Patrick, I share Greg's concerns with the feature as it's currently proposed. I don't think I could vote for something unless it made replication of genuinely internal topics and replication cycles impossible, or at least significantly less likely. Best, Chris On Tue, Jul 30, 2024, 14:51 Gre

[jira] [Created] (KAFKA-17221) Flaky test DedicatedMirrorIntegrationTest::testMultiNodeCluster

2024-07-30 Thread Chris Egerton (Jira)
Chris Egerton created KAFKA-17221: - Summary: Flaky test DedicatedMirrorIntegrationTest::testMultiNodeCluster Key: KAFKA-17221 URL: https://issues.apache.org/jira/browse/KAFKA-17221 Project: Kafka

[jira] [Resolved] (KAFKA-17044) Connector deletion can lead to resource leak during a long running connector startup

2024-07-30 Thread Chris Egerton (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-17044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton resolved KAFKA-17044. --- Resolution: Won't Fix > Connector deletion can lead to resource leak during a long

[jira] [Resolved] (KAFKA-15522) Flaky test org.apache.kafka.connect.mirror.integration.MirrorConnectorsIntegrationExactlyOnceTest.testOneWayReplicationWithFrequentOffsetSyncs

2024-07-29 Thread Chris Egerton (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton resolved KAFKA-15522. --- Resolution: Fixed > Flaky t

Re: [kafka-clients] [ANNOUNCE] Apache Kafka 3.8.0

2024-07-29 Thread Chris Egerton
Schofield, Anna Sophie Blee-Goldman, Antoine Pourchet, Anton Agestam, > Anton Liauchuk, Anuj Sharma, Apoorv Mittal, Arnout Engelen, Arpit Goyal, > Artem Livshits, Ashwin Pankaj, Ayoub Omari, Bruno Cadonna, Calvin Liu, > Cameron Redpath, charliecheng630, Cheng-Kai, Zhang, Cheryl Simmons, Chia &

Fwd: [VOTE] 3.8.0 RC3

2024-07-23 Thread Chris Egerton
Forwarding my response to the other mailing list threads; apologies for missing the reply-all the first time! -- Forwarded message - From: Chris Egerton Date: Tue, Jul 23, 2024 at 3:45 PM Subject: Re: [VOTE] 3.8.0 RC3 To: Hi Josep, Thanks for running this release! I'

[jira] [Resolved] (KAFKA-16068) Use TestPlugins in ConnectorValidationIntegrationTest to silence plugin scanning errors

2024-07-23 Thread Chris Egerton (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16068?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton resolved KAFKA-16068. --- Fix Version/s: 3.9.0 Resolution: Fixed > Use TestPlugins

[jira] [Resolved] (KAFKA-17105) Unnecessary connector restarts after being newly created

2024-07-17 Thread Chris Egerton (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-17105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton resolved KAFKA-17105. --- Resolution: Fixed > Unnecessary connector restarts after being newly crea

[jira] [Created] (KAFKA-17155) Redundant rebalances triggered after connector creation/deletion and task config updates

2024-07-17 Thread Chris Egerton (Jira)
Chris Egerton created KAFKA-17155: - Summary: Redundant rebalances triggered after connector creation/deletion and task config updates Key: KAFKA-17155 URL: https://issues.apache.org/jira/browse/KAFKA-17155

[jira] [Resolved] (KAFKA-16383) fix flaky test IdentityReplicationIntegrationTest.testReplicateFromLatest()

2024-07-16 Thread Chris Egerton (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton resolved KAFKA-16383. --- Resolution: Fixed > fix flaky t

[jira] [Resolved] (KAFKA-17145) Reinstate Utils.join() method in org.apache.kafka.common.utils

2024-07-16 Thread Chris Egerton (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-17145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton resolved KAFKA-17145. --- Resolution: Won't Fix > Reinstate Utils.join() method in org.apache.kafka.comm

[jira] [Resolved] (KAFKA-14401) Connector/Tasks reading offsets can get stuck if underneath WorkThread dies

2024-07-15 Thread Chris Egerton (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-14401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton resolved KAFKA-14401. --- Fix Version/s: 3.9.0 Resolution: Fixed > Connector/Tasks reading offsets can

Re: [VOTE] KIP-1059: Enable the Producer flush() method to clear the latest send() error

2024-07-15 Thread Chris Egerton
+0 Alieh, Matthias, Andrew--I know this isn't what you were hoping for, and I want to acknowledge the significant time and effort you've put into this KIP. I do believe it solves a real problem for Kafka Streams, but I don't believe the solution as presented is worth the fine print and potential f

[jira] [Created] (KAFKA-17130) Connect workers do not properly ensure group membership before responding to health checks

2024-07-12 Thread Chris Egerton (Jira)
Chris Egerton created KAFKA-17130: - Summary: Connect workers do not properly ensure group membership before responding to health checks Key: KAFKA-17130 URL: https://issues.apache.org/jira/browse/KAFKA-17130

[jira] [Created] (KAFKA-17115) Closing newly-created consumers during rebalance can cause rebalances to hang

2024-07-10 Thread Chris Egerton (Jira)
Chris Egerton created KAFKA-17115: - Summary: Closing newly-created consumers during rebalance can cause rebalances to hang Key: KAFKA-17115 URL: https://issues.apache.org/jira/browse/KAFKA-17115

Re: [VOTE] KIP-1054: Support external schemas in JSONConverter

2024-07-10 Thread Chris Egerton
rties file, > as well as embedding an inline schema directly within the connector > properties file . > > Thanks, > Priyanka > > From: Chris Egerton > Date: Friday, 5 July 2024 at 5:55 PM > To: dev@kafka.apache.org > Subject: [EXTERNAL] Re: [VOTE] KIP-1054: Support e

[jira] [Created] (KAFKA-17105) Unnecessary connector restarts after being newly created

2024-07-09 Thread Chris Egerton (Jira)
Chris Egerton created KAFKA-17105: - Summary: Unnecessary connector restarts after being newly created Key: KAFKA-17105 URL: https://issues.apache.org/jira/browse/KAFKA-17105 Project: Kafka

Re: [VOTE]: KIP-1032: Upgrade to Jakarta and JavaEE 10 in Kafka 4.0

2024-07-09 Thread Chris Egerton
+1 (binding), thanks for the KIP! On Tue, Jul 9, 2024 at 11:35 AM Greg Harris wrote: > While breaking compatibility is always undesirable, this is unfortunately a > way in which one of our dependencies "leaks through" our API, and makes > this a necessity. > Connect and MirrorMaker2 can follow t

Re: [DISCUSS] KIP-1032: Upgrade to Jakarta and JavaEE 9 in Kafka 4.0

2024-07-09 Thread Chris Egerton
hanges. For number 2 that was a holdover back when we > were discussing 11 vs 12 and whether or not JDK 17 would be used, but we > are going with 12 since 11 is deprecated so I will fix that. > > > On Tue, Jul 9, 2024 at 2:54 PM Chris Egerton > wrote: > > > Hi Chris, &g

Re: [DISCUSS] KIP-1032: Upgrade to Jakarta and JavaEE 9 in Kafka 4.0

2024-07-09 Thread Chris Egerton
Hi Chris, Thanks for the KIP! It's a bit of a drastic change to rip the bandaid off like this and require users running Connect to upgrade to JDK 17, but I think it's the best out of the options that are available to us. Two small nits on the KIP: 1. It'd be nice to link to KIP-1013 in the motiva

Re: [DISCUSS] KIP-1059: Enable the Producer flush() method to clear the latest send() error

2024-07-08 Thread Chris Egerton
s good. What do you think? I am totally against enumerating > errors in Javadocs since these sort of errors can be changing during > time. More > over, have you ever seen any list of recoverable or irrecoverable errors > somewhere so far? > > > Bests, > > Alieh > >

Re: [VOTE] KIP-1054: Support external schemas in JSONConverter

2024-07-05 Thread Chris Egerton
es. However, your point about the ergonomics is true - as this > needs the schema to be provided in a single line, it would be difficult to > read unless the schema is very small and simple. I think using a config > provider would be the answer for this. > > > Thanks > Priyanka > &

[jira] [Created] (KAFKA-17075) Use health check endpoint to verify Connect worker readiness in system tests

2024-07-03 Thread Chris Egerton (Jira)
Chris Egerton created KAFKA-17075: - Summary: Use health check endpoint to verify Connect worker readiness in system tests Key: KAFKA-17075 URL: https://issues.apache.org/jira/browse/KAFKA-17075

[jira] [Resolved] (KAFKA-10816) Connect REST API should have a resource that can be used as a readiness probe

2024-07-03 Thread Chris Egerton (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-10816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton resolved KAFKA-10816. --- Fix Version/s: 3.9.0 Resolution: Done > Connect REST API should have a resource t

Re: [DISCUSS] KIP-1059: Enable the Producer flush() method to clear the latest send() error

2024-07-03 Thread Chris Egerton
so hopefully Alieh can expand upon this. > > Justine > > On Wed, Jul 3, 2024 at 8:28 AM Chris Egerton > wrote: > > > Hi Alieh, > > > > I don't love defining the changes for this KIP in terms of a catch clause > > in the KafkaProducer class, f

Re: [DISCUSS] KIP-1059: Enable the Producer flush() method to clear the latest send() error

2024-07-03 Thread Chris Egerton
stine > > > > > > > > On Tue, Jul 2, 2024 at 10:51 AM Alieh Saeedi > > > > > > > > > > > wrote: > > > > > > > > > Hey Chris, > > > > > thanks for sharing your concerns. > > >

[jira] [Reopened] (KAFKA-15524) Flaky test org.apache.kafka.connect.integration.OffsetsApiIntegrationTest.testResetSinkConnectorOffsetsZombieSinkTasks

2024-07-02 Thread Chris Egerton (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton reopened KAFKA-15524: --- > Flaky t

[jira] [Resolved] (KAFKA-15524) Flaky test org.apache.kafka.connect.integration.OffsetsApiIntegrationTest.testResetSinkConnectorOffsetsZombieSinkTasks

2024-07-02 Thread Chris Egerton (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton resolved KAFKA-15524. --- Resolution: Later > Flaky t

Re: [DISCUSS] KIP-1059: Enable the Producer flush() method to clear the latest send() error

2024-07-02 Thread Chris Egerton
us from > > handling more in the future in an intuitive way? > > > > I think yes. This is all we want. Other sorts of errors such as having > > problem with partition addition, producer fenced exception, etc seem to > be > > more serious issues. The intention was to

Re: [VOTE] KIP-1025: Optionally URL-encode clientID and clientSecret in authorization header

2024-06-29 Thread Chris Egerton
Hi Nelson, Thank you for your patience! I like the plan for 4.0.0 and agree it'd be nice to land this KIP in time for 3.9.0. +1 (binding) Cheers, Chris On Wed, Jun 26, 2024 at 8:44 PM Nelson B. wrote: > Hi all, > > I want to bring up this thread once more. > > I am hoping to include this KIP

Re: [VOTE] KIP-1054: Support external schemas in JSONConverter

2024-06-29 Thread Chris Egerton
Hi Priyanka, I think allowing a raw schema string, instead of introducing a property that gives users access to the file system on the worker, is a good idea. I have two small thoughts: 1) The KIP provides an example of schema.content:${directory:/schema/schema.json} for how a config provider mig

Re: [DISCUSS] KIP-1059: Enable the Producer flush() method to clear the latest send() error

2024-06-28 Thread Chris Egerton
ll we want to handle, and will it prevent us from handling more in the future in an intuitive way), though. On Fri, Jun 28, 2024 at 8:57 PM Chris Egerton wrote: > Hi Alieh, > > This KIP has evolved a lot since I last looked at it, but the changes seem > well thought-out both in seman

Re: [DISCUSS] KIP-1059: Enable the Producer flush() method to clear the latest send() error

2024-06-28 Thread Chris Egerton
Hi Alieh, This KIP has evolved a lot since I last looked at it, but the changes seem well thought-out both in semantics and API. One clarifying question I have is that it looks based on the draft PR that we've narrowed the scope from any error that might take place with producing a record to Kafka

[jira] [Resolved] (KAFKA-16949) System test test_dynamic_logging in connect_distributed_test is failing

2024-06-25 Thread Chris Egerton (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton resolved KAFKA-16949. --- Fix Version/s: 3.9.0 Resolution: Fixed > System test test_dynamic_logging

Re: [DISCUSS] KIP-1059: Enable the Producer flush() method to clear the latest send() error

2024-06-25 Thread Chris Egerton
- the simple things should be really simple, any failure > during send or commit should abort send + commit sequence without special > handling. > > -Artem > > On Mon, Jun 24, 2024 at 6:37 PM Chris Egerton > wrote: > > > One quick thought: if a user invokes Producer::a

Re: [DISCUSS] KIP-1059: Enable the Producer flush() method to clear the latest send() error

2024-06-24 Thread Chris Egerton
// at this point, we know that all Future are completed and all > >> Callbacks are executed, and we can assume that all user code checking > for > >> error did execute, before `commitTx` is called > >>> > >>> // I consider this option as safe > >>&

Re: [DISCUSS] KIP-1059: Enable the Producer flush() method to clear the latest send() error

2024-06-24 Thread Chris Egerton
estion… > >>> > >>> Imagine the user’s application is writing information to a database > >> instead of Kafka. If there’s a table with a CHAR(1) column and this SQL > >> statement was attempted, what should happen? > >>> > >>>

Re: [DISCUSS] KIP-1059: Enable the Producer flush() method to clear the latest send() error

2024-06-21 Thread Chris Egerton
t;> flush() method. We already have too many configs. But I totally > >>>>>> understand > >>>>>>> the problem that you’re trying to solve and some of the other > >>>>> suggestions > >>>>>>> in this thread seem neater.

Re: [VOTE] KIP-1017: A health check endpoint for Kafka Connect

2024-06-20 Thread Chris Egerton
: Re: [VOTE] KIP-1017: A health check endpoint for Kafka > Connect > > > > > > > > Hi Chris, > > > > > > > > +1 (binding) > > > > > > > > Thanks, > > > > Mickael > > > > > > > > On Fri,

Re: [DISCUSS] KIP-1059: Enable the Producer flush() method to clear the latest send() error

2024-06-19 Thread Chris Egerton
Hi Matthias, I like the alternatives you've listed. One more that might help is if, instead of overloading flush(), we overloaded commitTransaction() to something like commitTransaction(boolean tolerateRecordErrors). This seems slightly cleaner in that it takes the behavioral change we want, which

Re: [DISCUSS] KIP-1038: Add Custom Error Handler to Producer

2024-06-17 Thread Chris Egerton
> SWALLOW. Therefore, RETRY will be interpreted and executed as FAIL. > > > > > > Why do we need this javadoc note? I think it's not possible to return > > > RETRY in the current form. > > > > > > When we talk about swallowing in the default i

Re: [DISCUSS] KIP-1017: A health check endpoint for Kafka Connect

2024-06-15 Thread Chris Egerton
the connectors. > would you agree? > > On Fri, 14 Jun 2024 at 15:44, Chris Egerton > wrote: > > > > Thanks Viktor! Seems as good a time as any to kick off the vote thread. > > > > On Fri, Jun 14, 2024, 10:43 Viktor Somogyi-Vass > > wrote: > > > &g

[VOTE] KIP-1017: A health check endpoint for Kafka Connect

2024-06-14 Thread Chris Egerton
Hi all, Happy Friday! I'd like to kick off a vote on KIP-1017. Design doc: https://cwiki.apache.org/confluence/display/KAFKA/KIP-1017%3A+Health+check+endpoint+for+Kafka+Connect Discussion thread: https://lists.apache.org/thread/95nto84wtogdgk3v97rxcwxr258wnjnt Cheers, Chris

Re: [DISCUSS] KIP-1017: A health check endpoint for Kafka Connect

2024-06-14 Thread Chris Egerton
x endpoints. The second solution seems complicated and > maybe we shouldn't overcomplicate it. > 2) Unfortunately I can't commit to that hypothetical KIP but I'll make a > mental note :) > > I'm +1 on the KIP. > > Thanks, > Viktor > > On Tue, Jun 11, 20

Re: [VOTE] KIP-1040: Improve handling of nullable values in InsertField, ExtractField, and other transformations

2024-06-13 Thread Chris Egerton
ect only ExtractField and > InsertField) > > > On 2024/06/13 17:01:49 Chris Egerton wrote: > > Hi Punsak, > > > > If nobody has signaled their intent to contribute that work yet (which I > > believe is the case), you are welcome to take it on yourself! > > > >

Re: [VOTE] KIP-1040: Improve handling of nullable values in InsertField, ExtractField, and other transformations

2024-06-13 Thread Chris Egerton
re one or more follow-up KIPs to apply the exact same > > > > changes > > > > > > to the SMTs we missed the first time. > > > > > > > > > > However, the KIP still omits the MaskField and ValueToKey > > > > transformations. > > > >

[jira] [Reopened] (KAFKA-16935) Automatically wait for cluster startup in embedded Connect integration tests

2024-06-12 Thread Chris Egerton (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton reopened KAFKA-16935: --- > Automatically wait for cluster startup in embedded Connect integration te

[jira] [Resolved] (KAFKA-16935) Automatically wait for cluster startup in embedded Connect integration tests

2024-06-12 Thread Chris Egerton (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton resolved KAFKA-16935. --- Resolution: Fixed > Automatically wait for cluster startup in embedded Connect integrat

[jira] [Created] (KAFKA-16943) Synchronously verify Connect worker startup failure in InternalTopicsIntegrationTest

2024-06-12 Thread Chris Egerton (Jira)
Chris Egerton created KAFKA-16943: - Summary: Synchronously verify Connect worker startup failure in InternalTopicsIntegrationTest Key: KAFKA-16943 URL: https://issues.apache.org/jira/browse/KAFKA-16943

Re: [DISCUSS] KIP-1017: A health check endpoint for Kafka Connect

2024-06-11 Thread Chris Egerton
if the rest server isn't available, it would serve in > itself as a log about the workers' behavior. I understand if you think it's > such a scope change that it should be an improvement KIP, but would like to > gauge what you think about this. > > Regards, > Vik

[jira] [Created] (KAFKA-16935) Automatically wait for cluster startup in embedded Connect integration tests

2024-06-11 Thread Chris Egerton (Jira)
Chris Egerton created KAFKA-16935: - Summary: Automatically wait for cluster startup in embedded Connect integration tests Key: KAFKA-16935 URL: https://issues.apache.org/jira/browse/KAFKA-16935

Re: [DISCUSS] KIP-1017: A health check endpoint for Kafka Connect

2024-06-11 Thread Chris Egerton
ling? Perhaps by providing a > richer JSON response? Something like this would help us adopt the health > check, as we could start by paging someone for all failures, then over time > (as we gained confidence) transition more of the conditions over to being > handled by automati

[jira] [Resolved] (KAFKA-9228) Reconfigured converters and clients may not be propagated to connector tasks

2024-06-10 Thread Chris Egerton (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-9228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton resolved KAFKA-9228. -- Fix Version/s: 3.9.0 Resolution: Fixed > Reconfigured converters and clients may not

Re: [DISCUSS] KIP-1017: A health check endpoint for Kafka Connect

2024-06-10 Thread Chris Egerton
rly like the explanation of how the > result will specifically > check the worker health in ways that have previously caused trouble. > > Thanks, > Andrew > > > On 7 Jun 2024, at 16:18, Mickael Maison > wrote: > > > > Hi Chris, > > > > Happy Friday! Th

Re: [VOTE] KIP-877: Mechanism for plugins and connectors to register metrics

2024-06-10 Thread Chris Egerton
+1 (binding), thanks Mickael! On Mon, Jun 10, 2024, 04:24 Mickael Maison wrote: > Hi, > > Following the feedback in the DISCUSS thread, I made significant > changes to the proposal. So I'd like to restart a vote for KIP-877: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-877%3A+Mechan

Re: Request permission to Create KIP

2024-06-06 Thread Chris Egerton
Hi Eric, I don't think you need to log into Pony Mail to start a discussion thread. You can just send an email to the dev list with [DISCUSS] in the title and that should be sufficient. Cheers, Chris On Thu, Jun 6, 2024 at 12:01 PM Eric Lu wrote: > Hi, > > Thanks for setting my account up. I

[jira] [Resolved] (KAFKA-16838) Kafka Connect loads old tasks from removed connectors

2024-06-04 Thread Chris Egerton (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton resolved KAFKA-16838. --- Fix Version/s: 3.9.0 Resolution: Fixed > Kafka Connect loads old tasks from remo

[jira] [Resolved] (KAFKA-16837) Kafka Connect fails on update connector for incorrect previous Config Provider tasks

2024-06-04 Thread Chris Egerton (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton resolved KAFKA-16837. --- Fix Version/s: 3.9.0 Resolution: Fixed > Kafka Connect fails on update connector

Re: [DISCUSS] KIP-877: Mechanism for plugins and connectors to register metrics

2024-05-31 Thread Chris Egerton
in the > imperative mood > > > > > >>>>> for their method name. Monitorable doesn't, which initially > seemed a bit > > > > > >>>>> inconsistent to me, but I think your intention is to allow > plugins to > > > > > >>>>> merely ret

  1   2   3   4   5   6   7   8   >