Hi all, It seems like there are no objections to this KIP, so I've kicked off a vote thread: https://lists.apache.org/thread/dgq332o5j25rwddbvfydf7ttrclldw17
Cheers, Chris On Fri, Nov 24, 2023 at 10:39 PM Chris Egerton <fearthecel...@gmail.com> wrote: > Hi Yash, > > Thanks for taking a look! Yeah, it looks like we'll be forced to hold onto > the property until 5.0 if we can't introduce it at least one minor release > before 4.0. I don't think this is the worst thing. Although it'd be nice to > have the property completely removed when designing features like KIP-987, > if necessary, we can also declare any new features incompatible with > connectors that have opted out of enforcement of the tasks.max property > (and likely enforce this restraint programmatically via preflight > validation, failing connectors/tasks, log messages, etc.). > > Cheers, > > Chris > > On Wed, Nov 22, 2023 at 10:04 PM Yash Mayya <yash.ma...@gmail.com> wrote: > > > Hi Chris, > > > > Thanks for the well written and comprehensive KIP! Given that we're > already > > past the KIP freeze deadline for 3.7.0 ( > > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.7.0) > and > > there may not be a 3.8.0 release before the 4.0.0 release, would we then > be > > forced to punt the removal of "tasks.max.enforce" to a future 5.0.0 > > release? I don't have any other comments, and the proposed changes make > > sense to me. > > > > Thanks, > > Yash > > > > On Mon, Nov 20, 2023 at 10:50 PM Chris Egerton <fearthecel...@gmail.com> > > wrote: > > > > > Hi Hector, > > > > > > Thanks for taking a look! I think the key difference between the > proposed > > > behavior and the rejected alternative is that the set of tasks that > will > > be > > > running with the former is still a complete set of tasks, whereas the > set > > > of tasks in the latter is a subset of tasks. Also noteworthy but > slightly > > > less important: the problem will be more visible to users with the > former > > > (the connector will still be marked FAILED) than with the latter. > > > > > > Cheers, > > > > > > Chris > > > > > > On Tue, Nov 21, 2023, 00:53 Hector Geraldino (BLOOMBERG/ 919 3RD A) < > > > hgerald...@bloomberg.net> wrote: > > > > > > > Thanks for the KIP Chris, adding this check makes total sense. > > > > > > > > I do have one question. The second paragraph in the Public Interfaces > > > > section states: > > > > > > > > "If the connector generated excessive tasks after being reconfigured, > > > then > > > > any existing tasks for the connector will be allowed to continue > > running, > > > > unless that existing set of tasks also exceeds the tasks.max > property." > > > > > > > > Would not failing the connector land us in the second scenario of > > > > 'Rejected Alternatives'? > > > > > > > > From: dev@kafka.apache.org At: 11/11/23 00:27:44 UTC-5:00To: > > > > dev@kafka.apache.org > > > > Subject: [DISCUSS] KIP-1004: Enforce tasks.max property in Kafka > > Connect > > > > > > > > Hi all, > > > > > > > > I'd like to open up KIP-1004 for discussion: > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1004%3A+Enforce+tasks.max+ > > > > property+in+Kafka+Connect > > > > > > > > As a brief summary: this KIP proposes that the Kafka Connect runtime > > > start > > > > failing connectors that generate a greater number of tasks than the > > > > tasks.max property, with an optional emergency override that can be > > used > > > to > > > > continue running these (probably-buggy) connectors if absolutely > > > necessary. > > > > > > > > I'll be taking time off most of the next three weeks, so response > > latency > > > > may be a bit higher than usual, but I wanted to kick off the > discussion > > > in > > > > case we can land this in time for the upcoming 3.7.0 release. > > > > > > > > Cheers, > > > > > > > > Chris > > > > > > > > > > > > > > > > > >
