Re: [ANNOUNCE] New committer: Andrew Schofield

2024-12-02 Thread Yash Mayya
Congrats Andrew! On Mon, 2 Dec, 2024, 20:26 PoAn Yang, wrote: > Congratulations Andrew! > > PoAn > > > On Dec 2, 2024, at 10:36 PM, Rajini Sivaram > wrote: > > > > Congratulations, Andrew! Well deserved! > > > > Regards, > > > > Rajini > > > > On Mon, Dec 2, 2024 at 2:31 PM Kamal Chandraprakash

Re: [ANNOUNCE] New committer: Kamal Chandraprakash

2024-09-30 Thread Yash Mayya
Congratulations Kamal! On Mon, 30 Sept, 2024, 18:07 Luke Chen, wrote: > Hi all, > > The PMC of Apache Kafka is pleased to announce a new Kafka committer, Kamal > Chandraprakash. > > Kamal has been a Kafka contributor since May 2017. He has made significant > contributions to the tiered storage f

Re: [ANNOUNCE] New Kafka PMC Member: Josep Prat

2024-09-06 Thread Yash Mayya
Congratulations Josep! On Fri, 6 Sept, 2024, 21:55 Chris Egerton, wrote: > Hi all, > > Josep has been a Kafka committer since December 2022. He has remained very > active and instructive in the community since then, and it's my pleasure to > announce that he has accepted our invitation to become

Re: [ANNOUNCE] New committer: Lianet Magrans

2024-08-28 Thread Yash Mayya
Congratulations Lianet! On Wed, Aug 28, 2024 at 9:05 PM David Jacot wrote: > Hi all, > > The PMC of Apache Kafka is pleased to announce a new Kafka committer, > Lianet Magrans. > > Lianet has been a Kafka contributor since June 2023. In addition to > being a regular contributor and reviewer, she

Re: [VOTE] KIP-1040: Improve handling of nullable values in InsertField, ExtractField, and other transformations

2024-06-11 Thread Yash Mayya
Thanks for the KIP! +1 (binding) On Tue, Jun 11, 2024 at 7:40 PM Mario Fiore Vitale wrote: > Hi all, > > Just wanted to bump up this thread for visibility. > > Thanks! > > On Wed, May 29, 2024 at 5:31 PM Mickael Maison > wrote: > > > Hi Mario, > > > > +1 (binding) > > Thanks for the KIP! > > >

Re: Permission to contribute to Apache Kafka

2024-05-22 Thread Yash Mayya
Hi Frédérik, I've granted you the necessary permissions. Let me know if something doesn't work as expected. Cheers, Yash On Wed, May 22, 2024 at 1:38 PM Frédérik Rouleau wrote: > Hi, > As I now have my wiki Id: frouleau and my Jira Id: fred-ro, can I have the > permission to contribute to KIP

Re: [ANNOUNCE] New committer: Igor Soarez

2024-04-24 Thread Yash Mayya
Congratulations Igor! On Wed, 24 Apr, 2024, 23:36 Colin McCabe, wrote: > Hi all, > > The PMC of Apache Kafka is pleased to announce a new Kafka committer, Igor > Soarez. > > Igor has been a Kafka contributor since 2019. In addition to being a > regular contributor and reviewer, he has made signi

Re: [ANNOUNCE] New Kafka PMC Member: Greg Harris

2024-04-13 Thread Yash Mayya
Congrats Greg! On Sun, 14 Apr, 2024, 05:56 Randall Hauch, wrote: > Congratulations, Greg! > > On Sat, Apr 13, 2024 at 6:36 PM Luke Chen wrote: > > > Congrats, Greg! > > > > On Sun, Apr 14, 2024 at 7:05 AM Viktor Somogyi-Vass > > wrote: > > > > > Congrats Greg! :) > > > > > > On Sun, Apr 14, 20

Re: [VOTE] KIP-477: Add PATCH method for connector config in Connect REST API

2024-04-12 Thread Yash Mayya
Hi Ivan, Thanks for reviving this KIP, I think it will be a useful addition to Connect! +1 (binding) Cheers, Yash On Tue, Apr 9, 2024 at 4:23 AM Knowles Atchison Jr wrote: > +1 (non binding) > > On Mon, Apr 8, 2024, 3:30 PM Chris Egerton > wrote: > > > Thanks Ivan! +1 (binding) from me. > >

Re: Permission to assign tickets in Jira

2024-03-27 Thread Yash Mayya
Hi Pavel, I've granted you the necessary permissions. Thanks for your interest in contributing to the Apache Kafka project! Cheers, Yash On Tue, Mar 26, 2024 at 11:32 PM Pavel Pozdeev wrote: > > Hi Team, > > Would it be possible to get a permission to assign tickets in Jira? > I've got a Jira

Re: [ANNOUNCE] New committer: Christo Lolov

2024-03-26 Thread Yash Mayya
Congratulations Christo! On Tue, Mar 26, 2024 at 5:34 PM Luke Chen wrote: > Hi, Everyone, > > The PMC of Apache Kafka is pleased to announce a new Kafka committer: > Christo Lolov. > > Christo has been a Kafka contributor since 2021. He has made over 50 > commits. He authored KIP-902, KIP-963, a

Re: [VOTE] KIP-995: Allow users to specify initial offsets while creating connectors

2024-03-05 Thread Yash Mayya
ctor implements alterOffsets or not. This also means > that > > if initial_offsets is set in the ConnectorCreate request, we will return > a > > new REST entity (ConnectorInfoWithInitialOffsetsResponse ?) which will > be a > > child class of ConnectorInfo. > > > > (

Re: [VOTE] KIP-910: Update Source offsets for Source Connectors without producing records

2024-01-30 Thread Yash Mayya
Hi Sagar, Thanks for the KIP and apologies for the extremely long delay here! I think we could do with some wordsmithing on the Javadoc for the new `SourceTask::updateOffsets` method but that can be taken care of in the PR. +1 (binding) Thanks, Yash On Wed, Nov 15, 2023 at 11:43 PM Sagar wrote

[jira] [Created] (KAFKA-16196) Cast transform doesn't handle invalid whole value casts gracefully

2024-01-25 Thread Yash Mayya (Jira)
Yash Mayya created KAFKA-16196: -- Summary: Cast transform doesn't handle invalid whole value casts gracefully Key: KAFKA-16196 URL: https://issues.apache.org/jira/browse/KAFKA-16196 Project:

Re: [VOTE] KIP-995: Allow users to specify initial offsets while creating connectors

2024-01-17 Thread Yash Mayya
Hi Ashwin, Thanks for the KIP. > If Connect runtime encounters an error in any of these steps, > it will cleanup (if required) and return an error response Could you please elaborate on the cleanup steps? For instance, if we encounter an error after wiping existing offsets but before writing the

Re: [VOTE] KIP-1004: Enforce tasks.max property in Kafka Connect

2024-01-03 Thread Yash Mayya
Hi Chris, +1 (binding), thanks for the KIP. Based on discussion in other threads, it looks like the community is aligned with having a 3.8 release before the 4.0 release so we should be able to remove the 'tasks.max.enforce' connector property in 4.0 (we'd discussed potentially having to live wit

Re: [ANNOUNCE] New Kafka PMC Member: Divij Vaidya

2023-12-28 Thread Yash Mayya
Congratulations Divij! On Wed, Dec 27, 2023 at 5:15 PM Luke Chen wrote: > Hi, Everyone, > > Divij has been a Kafka committer since June, 2023. He has remained very > active and instructive in the community since becoming a committer. It's my > pleasure to announce that Divij is now a member of K

Re: Requesting permission to contribute to Apache Kafka

2023-12-14 Thread Yash Mayya
Hi Roman, I've granted the required permissions to your accounts. Cheers, Yash On Fri, Dec 15, 2023 at 12:12 PM Роман Бондарь wrote: > Hi all, > > Please add me as a contributor to kafka project > > JIRA username: rbond > Wiki username: rbond > GitHub username: gitrbond > > Thank you, > Roman

Re: [DISCUSS] KIP-1004: Enforce tasks.max property in Kafka Connect

2023-11-22 Thread Yash Mayya
Hi Chris, Thanks for the well written and comprehensive KIP! Given that we're already past the KIP freeze deadline for 3.7.0 ( https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.7.0) and there may not be a 3.8.0 release before the 4.0.0 release, would we then be forced to punt the re

[jira] [Created] (KAFKA-15888) DistributedHerder log context should not use the same client ID for each Connect worker by default

2023-11-22 Thread Yash Mayya (Jira)
Yash Mayya created KAFKA-15888: -- Summary: DistributedHerder log context should not use the same client ID for each Connect worker by default Key: KAFKA-15888 URL: https://issues.apache.org/jira/browse/KAFKA-15888

Re: Requesting permission to contribute to Apache Kafka.

2023-11-07 Thread Yash Mayya
Hi Afshin, I've granted you the necessary permissions. Thanks for your interest in contributing to Apache Kafka! Cheers, Yash On Tue, Nov 7, 2023 at 3:52 PM Afshin Moazami wrote: > Hi, > I would like to request permission to contribute to Apache Kafka. > wiki ID: amoazami > Jira ID: afshing >

Re: Requesting permission to contribute to Apache Kafka

2023-11-02 Thread Yash Mayya
Hi Vedarth, I've granted you the necessary permissions. Thanks for your interest in contributing to Apache Kafka! Cheers, Yash On Thu, Nov 2, 2023 at 1:22 PM Vedarth Sharma wrote: > Hi, > > Please grant me permission to contribute to KIPs and assign jira tickets to > myself. > > wiki id and ji

Re: [VOTE] KIP-980: Allow creating connectors in a stopped state

2023-10-16 Thread Yash Mayya
Hi all, Thanks for participating in the discussion and voting! KIP-980 has been accepted with the following +1 votes: - Chris Egerton (binding) - Knowles Atchison Jr (non-binding) - Greg Harris (binding) - Mickael Maison (binding) - Yash Mayya (binding) The target release for

Re: [VOTE] KIP-980: Allow creating connectors in a stopped state

2023-10-16 Thread Yash Mayya
, 2023, 7:57 AM Chris Egerton > wrote: > > > > > Thanks for the KIP, Yash! > > > > > > +1 (binding) > > > > > > On Mon, Oct 9, 2023, 01:12 Yash Mayya wrote: > > > > > > > Hi all, > > > > > &g

[jira] [Created] (KAFKA-15570) Add unit tests for MemoryConfigBackingStore

2023-10-10 Thread Yash Mayya (Jira)
Yash Mayya created KAFKA-15570: -- Summary: Add unit tests for MemoryConfigBackingStore Key: KAFKA-15570 URL: https://issues.apache.org/jira/browse/KAFKA-15570 Project: Kafka Issue Type: Test

[VOTE] KIP-980: Allow creating connectors in a stopped state

2023-10-08 Thread Yash Mayya
Hi all, I'd like to start a vote on KIP-980 which proposes allowing the creation of connectors in a stopped (or paused) state. KIP - https://cwiki.apache.org/confluence/display/KAFKA/KIP-980%3A+Allow+creating+connectors+in+a+stopped+state Discussion Thread - https://lists.apache.org/thread/om803

Re: [VOTE] KIP-976: Cluster-wide dynamic log adjustment for Kafka Connect

2023-10-08 Thread Yash Mayya
Hi Chris, Thanks for the KIP! +1 (binding) Yash On Fri, Oct 6, 2023 at 9:54 PM Greg Harris wrote: > Hey Chris, > > Thanks for the KIP! > I think that preserving the ephemeral nature of the logging change is > the right choice here, and using the config topic for intra-cluster > broadcast is be

Re: [DISCUSS] KIP-980: Allow creating connectors in a stopped state

2023-10-06 Thread Yash Mayya
t 10:49 PM Chris Egerton wrote: > Hi Yash, > > Yeah, I think just hardcoding with JSON-first, properties-second is fine. > IMO it's worth calling this out explicitly in the KIP. > > Apart from that, no further comments. LGTM, thanks for the KIP! > > Cheers, > > Chris

Re: [DISCUSS] KIP-980: Allow creating connectors in a stopped state

2023-10-05 Thread Yash Mayya
something else? > > 4. (Addressed) Thanks! Looks great. > > 6. (Addressed) Awesome, great to hear. The point about laggy connector > startup is very convincing; my paranoia is satiated. > > > Cheers, > > Chris > > On Wed, Oct 4, 2023 at 5:35 AM Yash Mayya wrote:

[jira] [Resolved] (KAFKA-15547) Thread leak in MirrorMakerConfigTest#testClientConfigProperties

2023-10-04 Thread Yash Mayya (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yash Mayya resolved KAFKA-15547. Fix Version/s: 3.7.0 Resolution: Fixed > Thread leak in MirrorMakerConfigT

Re: [DISCUSS] KIP-980: Allow creating connectors in a stopped state

2023-10-04 Thread Yash Mayya
ors in the stopped state, or add any other > special logic besides noting the new state in the config topic? Or is it > sufficient to write a non-running target state to the config topic and then > rely on existing logic to simply refuse to generate task configs for the > newly-created

Re: [DISCUSS] KIP-980: Allow creating connectors in a stopped state

2023-10-03 Thread Yash Mayya
more. > > 5. (NIt): We can link to our own Javadocs [2] instead of javadoc.io > > > [1] - > > https://github.com/apache/kafka/blob/dcd8c7d05f2f22f2d815405e7ab3ad7439669239/connect/runtime/src/main/java/org/apache/kafka/connect/storage/KafkaConfigBackingStore.java#L234-L236 >

[jira] [Resolved] (KAFKA-15177) MirrorMaker 2 should implement the alterOffsets KIP-875 API

2023-09-27 Thread Yash Mayya (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yash Mayya resolved KAFKA-15177. Fix Version/s: 3.6.0 Resolution: Fixed > MirrorMaker 2 should implement the alterOffs

Re: [DISCUSS] KIP-980: Allow creating connectors in a stopped state

2023-09-21 Thread Yash Mayya
the case of a downgrade, how will Connect worker handle the optional > “state” field in config topic ? > > Thanks, > Ashwin > > > > > On Sun, Sep 17, 2023 at 11:09 PM Yash Mayya wrote: > > > Hi all, > > > > I'd like to begin discussion on a KIP

[DISCUSS] KIP-980: Allow creating connectors in a stopped state

2023-09-17 Thread Yash Mayya
Hi all, I'd like to begin discussion on a KIP to allow creating connectors in a stopped state - https://cwiki.apache.org/confluence/display/KAFKA/KIP-980%3A+Allow+creating+connectors+in+a+stopped+state Thanks, Yash

[jira] [Created] (KAFKA-15470) Allow creating connectors in a stopped state

2023-09-15 Thread Yash Mayya (Jira)
Yash Mayya created KAFKA-15470: -- Summary: Allow creating connectors in a stopped state Key: KAFKA-15470 URL: https://issues.apache.org/jira/browse/KAFKA-15470 Project: Kafka Issue Type: New

[jira] [Resolved] (KAFKA-14067) Sink connector override.consumer.group.id can conflict with worker group.id

2023-09-12 Thread Yash Mayya (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-14067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yash Mayya resolved KAFKA-14067. Resolution: Fixed > Sink connector override.consumer.group.id can conflict with worker group

Re: [VOTE] KIP-970: Deprecate and remove Connect's redundant task configurations endpoint

2023-09-08 Thread Yash Mayya
for the KIP! > > Mickael > > On Wed, Sep 6, 2023 at 6:05 PM Greg Harris > wrote: > > > > Hey Yash, > > > > +1(binding) > > > > Thanks for the KIP! > > Greg > > > > On Wed, Sep 6, 2023 at 6:59 AM Yash Mayya wrote: > > > > > &

Re: [VOTE] KIP-970: Deprecate and remove Connect's redundant task configurations endpoint

2023-09-06 Thread Yash Mayya
Hi all, I just wanted to bump up this vote thread. Thanks to everyone who's voted so far - we have 1 binding +1 vote and 3 non-binding +1 votes so far. Thanks, Yash On Wed, Aug 30, 2023 at 11:14 PM Sagar wrote: > +1 (non - binding). > > Thanks ! > Sagar. > > On Wed, 30 Aug 2023 at 11:09 PM, Ch

Re: KIP-976: Cluster-wide dynamic log adjustment for Kafka Connect

2023-09-06 Thread Yash Mayya
Hi Chris, Thanks for the clarification on the last modified timestamp tracking here and on the KIP, things look good to me now. On the persistence front, I hadn't considered the interplay between levels set by the log level REST APIs and those set by the log4j configuration files, and the user co

Re: KIP-976: Cluster-wide dynamic log adjustment for Kafka Connect

2023-09-03 Thread Yash Mayya
27;t account for the fact that at worker startup we'll still have loggers with non-null log levels - the root logger and any other named loggers which have explicit log levels configured in the log4j properties. On Mon, Sep 4, 2023 at 12:00 PM Yash Mayya wrote: > Hi Chris, > > Thank

Re: KIP-976: Cluster-wide dynamic log adjustment for Kafka Connect

2023-09-03 Thread Yash Mayya
Hi Chris, Thanks for the KIP, this looks like a really useful addition to Kafka Connect's log level REST APIs! I have a few questions and comments: > If no modifications to the namespace have > been made since the worker was started, > they will be null If no modifications to a logging namespace

[VOTE] KIP-970: Deprecate and remove Connect's redundant task configurations endpoint

2023-08-29 Thread Yash Mayya
Hi all, This is the vote thread for KIP-970 which proposes deprecating (in the Apache Kafka 3.7 release) and eventually removing (in the next major Apache Kafka release - 4.0) Connect's redundant task configurations endpoint. KIP - https://cwiki.apache.org/confluence/display/KAFKA/KIP-970%3A+Depr

Re: [DISCUSS] KIP-910: Update Source offsets for Source Connectors without producing records

2023-08-29 Thread Yash Mayya
Hi Sagar, > The size of offsets topic can be controlled by > setting appropriate topic retention values and > that is a standard practice in Kafka Kafka Connect enforces the `cleanup.policy` configuration for the offsets topic to be "compact" only (references - [1], [2]), so the topic retention r

Re: [DISCUSS] KIP-910: Update Source offsets for Source Connectors without producing records

2023-08-22 Thread Yash Mayya
erent thread, ordering > guarantees might be harder to ensure if we do it from the other thread. The > current mechanism proposed, even though gets invoked multiple times, keeps > things simpler to reason about. > > Let me know how things look now. If it's all looking ok, I would go a

[DISCUSS] KIP-970: Deprecate and remove Connect's redundant task configurations endpoint

2023-08-22 Thread Yash Mayya
Hi all, I'd like to start a discussion thread for this KIP - https://cwiki.apache.org/confluence/display/KAFKA/KIP-970%3A+Deprecate+and+remove+Connect%27s+redundant+task+configurations+endpoint . It proposes the deprecation and eventual removal of Kafka Connect's redundant task configurations end

[jira] [Created] (KAFKA-15387) Deprecate and remove Connect's duplicate task configurations retrieval endpoint

2023-08-21 Thread Yash Mayya (Jira)
Yash Mayya created KAFKA-15387: -- Summary: Deprecate and remove Connect's duplicate task configurations retrieval endpoint Key: KAFKA-15387 URL: https://issues.apache.org/jira/browse/KAFKA-15387 Pr

[jira] [Created] (KAFKA-15377) GET /connectors/{connector}/tasks-config endpoint exposes externalized secret values

2023-08-18 Thread Yash Mayya (Jira)
Yash Mayya created KAFKA-15377: -- Summary: GET /connectors/{connector}/tasks-config endpoint exposes externalized secret values Key: KAFKA-15377 URL: https://issues.apache.org/jira/browse/KAFKA-15377

Re: Apache Kafka 3.6.0 release

2023-07-26 Thread Yash Mayya
Hi Hector, KIP-959 actually still requires 2 more binding votes to be accepted ( https://cwiki.apache.org/confluence/display/KAFKA/Bylaws#Bylaws-Approvals). The non-binding votes from people who aren't committers (including myself) don't count towards the required lazy majority. Thanks, Yash On

Re: [VOTE] KIP-959 Add BooleanConverter to Kafka Connect

2023-07-25 Thread Yash Mayya
Hi Hector, Thanks for the KIP! +1 (non-binding) Thanks, Yash On Tue, Jul 25, 2023 at 11:01 PM Andrew Schofield < andrew_schofield_j...@outlook.com> wrote: > Thanks for the KIP. As you say, not that controversial. > > +1 (non-binding) > > Thanks, > Andrew > > > On 25 Jul 2023, at 18:22, Hector

Re: [DISCUSS] KIP-910: Update Source offsets for Source Connectors without producing records

2023-07-25 Thread Yash Mayya
e > >> reading the "Proposed Changes" section. I'd bet that this is also why we > >> use that ordering in the KIP template. > >> > >> 2. Why are we invoking SourceTask::updateOffsets so frequently when > >> exactly-once support is disabled?

[jira] [Created] (KAFKA-15238) Connect workers can be disabled by DLQ related stuck admin client calls

2023-07-23 Thread Yash Mayya (Jira)
Yash Mayya created KAFKA-15238: -- Summary: Connect workers can be disabled by DLQ related stuck admin client calls Key: KAFKA-15238 URL: https://issues.apache.org/jira/browse/KAFKA-15238 Project: Kafka

[jira] [Created] (KAFKA-15216) InternalSinkRecord::newRecord method ignores the headers argument

2023-07-18 Thread Yash Mayya (Jira)
Yash Mayya created KAFKA-15216: -- Summary: InternalSinkRecord::newRecord method ignores the headers argument Key: KAFKA-15216 URL: https://issues.apache.org/jira/browse/KAFKA-15216 Project: Kafka

[jira] [Created] (KAFKA-15182) Normalize offsets before invoking SourceConnector::alterOffsets

2023-07-12 Thread Yash Mayya (Jira)
Yash Mayya created KAFKA-15182: -- Summary: Normalize offsets before invoking SourceConnector::alterOffsets Key: KAFKA-15182 URL: https://issues.apache.org/jira/browse/KAFKA-15182 Project: Kafka

[jira] [Created] (KAFKA-15179) Add integration tests for the FileStream Sink and Source connectors

2023-07-11 Thread Yash Mayya (Jira)
Yash Mayya created KAFKA-15179: -- Summary: Add integration tests for the FileStream Sink and Source connectors Key: KAFKA-15179 URL: https://issues.apache.org/jira/browse/KAFKA-15179 Project: Kafka

[jira] [Created] (KAFKA-15177) MirrorMaker 2 should implement the alterOffsets KIP-875 API

2023-07-11 Thread Yash Mayya (Jira)
Yash Mayya created KAFKA-15177: -- Summary: MirrorMaker 2 should implement the alterOffsets KIP-875 API Key: KAFKA-15177 URL: https://issues.apache.org/jira/browse/KAFKA-15177 Project: Kafka

Re: [ANNOUNCE] New committer: Greg Harris

2023-07-10 Thread Yash Mayya
Congrats Greg! On Mon, Jul 10, 2023 at 9:15 PM Chris Egerton wrote: > Hi all, > > The PMC for Apache Kafka has invited Greg Harris to become a committer, and > we are happy to announce that he has accepted! > > Greg has been contributing to Kafka since 2019. He has made over 50 commits > mostly

Re: [VOTE] KIP-882: Kafka Connect REST API timeout improvements

2023-07-10 Thread Yash Mayya
continue with a public discussion and vote. Thanks, Yash On Wed, Mar 1, 2023 at 6:32 PM Yash Mayya wrote: > Hi all, > > I'd like to call for a vote on the (hopefully) straightforward KIP-882 > which adds support for configuring request timeouts on Kafka Connect REST > API

Re: Apache Kafka 3.6.0 release

2023-07-06 Thread Yash Mayya
Hi Satish, KIP-793 [1] just passed voting and we should be able to wrap up the implementation in time for the 3.6.0 feature freeze. Could we add it to the release plan? [1] - https://cwiki.apache.org/confluence/display/KAFKA/KIP-793%3A+Allow+sink+connectors+to+be+used+with+topic-mutating+SMTs Th

Re: [VOTE] KIP-793: Allow sink connectors to be used with topic-mutating SMTs

2023-07-06 Thread Yash Mayya
; > > > On Mon, Jul 3, 2023 at 7:19 AM Chris Egerton > > wrote: > > > > > > > > Hi Yash, > > > > > > > > Thanks for the KIP! +1 (binding) > > > > > > > > Cheers, > > > > > > > > Chris > > >

Re: [DISCUSS] KIP-943: Add independent "offset.storage.segment.bytes" for connect-distributed.properties

2023-07-06 Thread Yash Mayya
" (via the existing KIP-605 based mechanism). On Thu, Jul 6, 2023 at 6:04 PM Yash Mayya wrote: > Hi hudeqi, > > Thanks for the KIP! Just to clarify - since KIP-605 ( > https://cwiki.apache.org/confluence/display/KAFKA/KIP-605%3A+Expand+Connect+Worker+Internal+Topic+Settings) &

Re: [DISCUSS] KIP-943: Add independent "offset.storage.segment.bytes" for connect-distributed.properties

2023-07-06 Thread Yash Mayya
Hi hudeqi, Thanks for the KIP! Just to clarify - since KIP-605 ( https://cwiki.apache.org/confluence/display/KAFKA/KIP-605%3A+Expand+Connect+Worker+Internal+Topic+Settings) already allows configuring "segment.bytes" for the Connect cluster's offsets topic via a worker configuration ("offset.storag

Re: [DISCUSS] KIP-910: Update Source offsets for Source Connectors without producing records

2023-07-04 Thread Yash Mayya
gt; >>> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-910%3A+Update+Source+offsets+for+Source+Connectors+without+producing+records > >>> with a newer approach which shelves the need for an explicit topic. > >>> > >>> Please review agai

[jira] [Created] (KAFKA-15145) AbstractWorkerSourceTask re-processes records filtered out by SMTs on retriable exceptions

2023-07-04 Thread Yash Mayya (Jira)
Yash Mayya created KAFKA-15145: -- Summary: AbstractWorkerSourceTask re-processes records filtered out by SMTs on retriable exceptions Key: KAFKA-15145 URL: https://issues.apache.org/jira/browse/KAFKA-15145

[VOTE] KIP-793: Allow sink connectors to be used with topic-mutating SMTs

2023-07-03 Thread Yash Mayya
Hi all, I'd like to start a vote on KIP-793 which enables sink connector implementations to be used with SMTs that mutate the topic / partition / offset information of a record. KIP - https://cwiki.apache.org/confluence/display/KAFKA/KIP-793%3A+Allow+sink+connectors+to+be+used+with+topic-mutating

Re: [DISCUSS] KIP-793: Sink Connectors: Support topic-mutating SMTs for async connectors (preCommit users)

2023-07-03 Thread Yash Mayya
ould prefer that API instead of using SinkTask::preCommit. > > [1] - > > https://github.com/apache/kafka/blob/12be344fdd3b20f338ccab87933b89049ce202a4/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/SubmittedRecords.java > > Cheers, > > Chris > > On Wed,

[jira] [Created] (KAFKA-15121) FileStreamSourceConnector and FileStreamSinkConnector should implement KIP-875 APIs

2023-06-26 Thread Yash Mayya (Jira)
Yash Mayya created KAFKA-15121: -- Summary: FileStreamSourceConnector and FileStreamSinkConnector should implement KIP-875 APIs Key: KAFKA-15121 URL: https://issues.apache.org/jira/browse/KAFKA-15121

[jira] [Created] (KAFKA-15113) Gracefully handle cases where a sink connector's admin and consumer client config overrides target different Kafka clusters

2023-06-22 Thread Yash Mayya (Jira)
Yash Mayya created KAFKA-15113: -- Summary: Gracefully handle cases where a sink connector's admin and consumer client config overrides target different Kafka clusters Key: KAFKA-15113 URL: https://issues.apach

Re: [DISCUSS] KIP-793: Sink Connectors: Support topic-mutating SMTs for async connectors (preCommit users)

2023-06-21 Thread Yash Mayya
o deal > with post-transform topic partitions instead. I'm on the fence with this > one, but if it's a choice between passing this KIP without modifying > SinkTask::open/close, or letting the KIP go dormant, I'd happily choose the > former. > > Thanks Yash and Greg for

Re: [ANNOUNCE] New committer: Divij Vaidya

2023-06-13 Thread Yash Mayya
Congratulations Divij! On Tue, Jun 13, 2023 at 9:20 PM Bruno Cadonna wrote: > Hi all, > > The PMC of Apache Kafka is pleased to announce a new Kafka committer > Divij Vaidya. > > Divij's major contributions are: > > GDPR compliance enforcement of kafka-site - > https://issues.apache.org/jira/bro

Re: [VOTE] 3.4.1 RC3

2023-06-06 Thread Yash Mayya
Hi Prem, You can follow the instructions here if you wish to unsubscribe from one or more mailing lists - https://kafka.apache.org/contact On Tue, Jun 6, 2023 at 2:44 PM Prem Sagar wrote: > Please remove my mail ID from the database. > > > Warm Regards > K Prem Sagar > Sr.Manager - Procurements

[jira] [Resolved] (KAFKA-14956) Flaky test org.apache.kafka.connect.integration.OffsetsApiIntegrationTest#testGetSinkConnectorOffsetsDifferentKafkaClusterTargeted

2023-05-29 Thread Yash Mayya (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-14956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yash Mayya resolved KAFKA-14956. Resolution: Fixed > Flaky test > org.apache.kafka.connect.integration.OffsetsApiIntegrati

Re: [DISCUSS] Adding non-committers as Github collaborators

2023-05-24 Thread Yash Mayya
gt; > > > > > > > Just a quick update: after merging the changes to asf.yaml, I > received > > a > > > > notification that the list is limited to only 10 people, not 20 as > the > > > > documentation states. > > > > > > > >

Re: [DISCUSS] Adding non-committers as Github collaborators

2023-05-24 Thread Yash Mayya
to asf.yaml, I received a > > notification that the list is limited to only 10 people, not 20 as the > > documentation states. > > > > Here is the list of folks who will now be able to triage PRs and trigger > > builds: Victoria Xia, Greg Harris, Divij Vaidya, Lu

Re: [DISCUSS] Adding non-committers as Github collaborators

2023-05-12 Thread Yash Mayya
to git shortlog). Congratulations to our new collaborators! > > Victoria Xia, Greg Harris, Divij Vaidya, Lucas Brutschy, Yash Mayya, > Philip Nee, vamossagar12,, Christo Lolov, Federico Valeri, andymg3, > RivenSun, Kirk True, Matthew de Detrich, Akhilesh C, Alyssa Huang, Artem > Liv

[jira] [Created] (KAFKA-14974) Restore backward compatibility in KafkaBasedLog

2023-05-08 Thread Yash Mayya (Jira)
Yash Mayya created KAFKA-14974: -- Summary: Restore backward compatibility in KafkaBasedLog Key: KAFKA-14974 URL: https://issues.apache.org/jira/browse/KAFKA-14974 Project: Kafka Issue Type: Task

[jira] [Created] (KAFKA-14933) Document Kafka Connect's log level REST APIs added in KIP-495

2023-04-25 Thread Yash Mayya (Jira)
Yash Mayya created KAFKA-14933: -- Summary: Document Kafka Connect's log level REST APIs added in KIP-495 Key: KAFKA-14933 URL: https://issues.apache.org/jira/browse/KAFKA-14933 Project:

Re: [ANNOUNCE] New PMC chair: Mickael Maison

2023-04-24 Thread Yash Mayya
Congratulations Mickael! On Fri, Apr 21, 2023 at 8:39 PM Jun Rao wrote: > Hi, everyone, > > After more than 10 years, I am stepping down as the PMC chair of Apache > Kafka. We now have a new chair Mickael Maison, who has been a PMC member > since 2020. I plan to continue to contribute to Apache

Re: [DISCUSS] KIP-910: Update Source offsets for Source Connectors without producing records

2023-04-24 Thread Yash Mayya
Hi Sagar, Thanks for the KIP! I have a few questions and comments: 1) I agree with Chris' point about the separation of a connector heartbeat mechanism and allowing source connectors to generate offsets without producing data. What is the purpose of the heartbeat topic here and are there any conc

Re: [DISCUSS] KIP-793: Sink Connectors: Support topic-mutating SMTs for async connectors (preCommit users)

2023-04-23 Thread Yash Mayya
> without doing a lot of book-keeping. > > I suppose there is a possible implementation of metadata book-keeping which > provides a reasonable system of virtual coordinates, it just ended up > equivalent to hydrating intermediate topics to compute a consistent record > orderin

[jira] [Created] (KAFKA-14910) Consider cancelling ongoing alter connector offsets requests when the connector is resumed

2023-04-14 Thread Yash Mayya (Jira)
Yash Mayya created KAFKA-14910: -- Summary: Consider cancelling ongoing alter connector offsets requests when the connector is resumed Key: KAFKA-14910 URL: https://issues.apache.org/jira/browse/KAFKA-14910

Re: [VOTE] KIP-875: First-class offsets support in Kafka Connect

2023-04-12 Thread Yash Mayya
tentially just > fire-and-forget the request and, if new tasks are started for the connector > in the meantime, trust that the request will get automatically fenced out > without doing any more work). > > Cheers, > > Chris > > On Wed, Apr 12, 2023 at 1:01 PM Yash Mayya wrot

Re: [VOTE] KIP-875: First-class offsets support in Kafka Connect

2023-04-12 Thread Yash Mayya
DISCUSS >> thread. >> > >> >> > >> +1 (binding). >> > >> >> > >> Best, >> > >> >> > >> On Wed, Mar 1, 2023 at 12:16 PM Tom Bentley >> > wrote: >> > >> >> > >> &g

[jira] [Created] (KAFKA-14876) Public documentation for new Kafka Connect offset management REST APIs

2023-04-02 Thread Yash Mayya (Jira)
Yash Mayya created KAFKA-14876: -- Summary: Public documentation for new Kafka Connect offset management REST APIs Key: KAFKA-14876 URL: https://issues.apache.org/jira/browse/KAFKA-14876 Project: Kafka

[jira] [Created] (KAFKA-14844) Kafka Connect's OffsetBackingStore interface should handle (de)serialization and connector namespacing

2023-03-24 Thread Yash Mayya (Jira)
Yash Mayya created KAFKA-14844: -- Summary: Kafka Connect's OffsetBackingStore interface should handle (de)serialization and connector namespacing Key: KAFKA-14844 URL: https://issues.apache.org/jira/browse/

Re: [DISCUSS] KIP-882: Make Kafka Connect REST API request timeouts configurable

2023-03-19 Thread Yash Mayya
> and other operations later). > But as it stands, I don't think I see the value-add of configurable > timeouts once there is another solution which targets the validate call > duration specifically, and I'm concerned that configurable timeouts would > be made irrelev

Re: [DISCUSS] KIP-793: Sink Connectors: Support topic-mutating SMTs for async connectors (preCommit users)

2023-03-12 Thread Yash Mayya
unctions and keep everything else backwards > compatible. After deriving all of the above, I think that's a reasonable > tradeoff to make. > > Thanks, > Greg > > On Tue, Feb 21, 2023 at 10:17 AM Chris Egerton > wrote: > > > Hi Yash, > > > > We'

Re: [ANNOUNCE] New Kafka PMC Member: Chris Egerton

2023-03-09 Thread Yash Mayya
Congratulations Chris! On Thu, Mar 9, 2023, 23:42 Jun Rao wrote: > Hi, Everyone, > > Chris Egerton has been a Kafka committer since July 2022. He has been very > instrumental to the community since becoming a committer. It's my pleasure > to announce that Chris is now a member of Kafka PMC. > >

Re: [ANNOUNCE] New Kafka PMC Member: David Arthur

2023-03-09 Thread Yash Mayya
Congrats David! On Thu, Mar 9, 2023, 23:42 Jun Rao wrote: > Hi, Everyone, > > David Arthur has been a Kafka committer since 2013. He has been very > instrumental to the community since becoming a committer. It's my pleasure > to announce that David is now a member of Kafka PMC. > > Congratulatio

Re: [DISCUSS] KIP-882: Make Kafka Connect REST API request timeouts configurable

2023-03-08 Thread Yash Mayya
re the only ones with a > synchronicity constraint. > Again, I'm not necessarily saying that you must implement such an > asynchronous validation scheme in this KIP, but we should consider if that > is a more extensible solution. If we decided to implement configurable > synchron

Re: [VOTE] KIP-882: Kafka Connect REST API timeout improvements

2023-03-02 Thread Yash Mayya
x27;, '10s' or '500ms'. > Is it too late to incorporate this change? > > Thanks, > Ashwin > > > On Wed, Mar 1, 2023 at 6:32 PM Yash Mayya wrote: > > > Hi all, > > > > I'd like to call for a vote on the (hopefully) straightforward K

Re: [DISCUSS] KIP-882: Make Kafka Connect REST API request timeouts configurable

2023-03-02 Thread Yash Mayya
nderstand if you want to keep this KIP as tightly focused as possible, > but i'm worried that it is addressing the symptom and not the problem. I > want to make sure that this change is impactful and isn't obsoleted by a > later improvement. > > Thanks, > Greg > >

Re: [DISCUSS] KIP-882: Make Kafka Connect REST API request timeouts configurable

2023-03-01 Thread Yash Mayya
//lists.apache.org/thread/mgx8lczx2f57pk7x3vh0nqk00s79grgp. Thanks, Yash On Sat, Nov 5, 2022 at 11:42 PM Sagar wrote: > Hey Yash, > > Thanks for the explanation. I think it should be fine to delegate the > validation directly to the leader. > > Thanks! > Sagar. > > On Sat, Nov 5, 2022 at 10:42

[VOTE] KIP-882: Kafka Connect REST API timeout improvements

2023-03-01 Thread Yash Mayya
Hi all, I'd like to call for a vote on the (hopefully) straightforward KIP-882 which adds support for configuring request timeouts on Kafka Connect REST APIs via query parameters along with a couple of related small improvements. KIP - https://cwiki.apache.org/confluence/display/KAFKA/KIP-882%3A+

Re: [DISCUSS] KIP-793: Sink Connectors: Support topic-mutating SMTs for async connectors (preCommit users)

2023-02-21 Thread Yash Mayya
f a task class has not overridden the > multi-arg variant. But I think this is going a bit too far and would prefer > to keep things simple(r) for now. > > Cheers, > > Chris > > > On Sun, Feb 19, 2023 at 2:34 AM Yash Mayya wrote: > > > Hi Chris, &g

Re: [DISCUSS] KIP-793: Sink Connectors: Support topic-mutating SMTs for async connectors (preCommit users)

2023-02-18 Thread Yash Mayya
proposal, and we can both take > a look and make sure everything looks good enough before opening a vote > thread. > > Finally, I think you make a convincing case for a time-based eviction > policy. I wasn't thinking about the fairly common SMT pattern of deriving a >

[jira] [Created] (KAFKA-14732) Use an exponential backoff retry mechanism while reconfiguring connector tasks

2023-02-18 Thread Yash Mayya (Jira)
Yash Mayya created KAFKA-14732: -- Summary: Use an exponential backoff retry mechanism while reconfiguring connector tasks Key: KAFKA-14732 URL: https://issues.apache.org/jira/browse/KAFKA-14732 Project

Re: [ANNOUNCE] New committer: Lucas Bradstreet

2023-02-16 Thread Yash Mayya
Congratulations Lucas! On Fri, Feb 17, 2023 at 3:25 AM Jun Rao wrote: > Hi, Everyone, > > The PMC of Apache Kafka is pleased to announce a new Kafka committer Lucas > Bradstreet. > > Lucas has been a long time Kafka contributor since Oct. 2018. He has been > extremely valuable for Kafka on both

Re: [DISCUSS] KIP-793: Sink Connectors: Support topic-mutating SMTs for async connectors (preCommit users)

2023-02-14 Thread Yash Mayya
ssary calls to close/open over a given sink > task's lifetime), but should not lead to guaranteed resource leaks or > failure to obey API contract in any cases. > > Cheers, > > Chris > > On Mon, Feb 13, 2023 at 10:54 AM Yash Mayya wrote: > > > Hi Chris, > >

Re: [DISCUSS] KIP-793: Sink Connectors: Support topic-mutating SMTs for async connectors (preCommit users)

2023-02-13 Thread Yash Mayya
ic partitions. I'm hoping that this logic can stay internal, and by not > painting ourselves into a corner with the KIP, we give ourselves leeway to > tweak it in the future if necessary without filing another KIP or > introducing a pluggable interface. > > Cheers, > > Chr

  1   2   >