Hi Chris, Thanks for the well written and comprehensive KIP! Given that we're already past the KIP freeze deadline for 3.7.0 ( https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.7.0) and there may not be a 3.8.0 release before the 4.0.0 release, would we then be forced to punt the removal of "tasks.max.enforce" to a future 5.0.0 release? I don't have any other comments, and the proposed changes make sense to me.
Thanks, Yash On Mon, Nov 20, 2023 at 10:50 PM Chris Egerton <fearthecel...@gmail.com> wrote: > Hi Hector, > > Thanks for taking a look! I think the key difference between the proposed > behavior and the rejected alternative is that the set of tasks that will be > running with the former is still a complete set of tasks, whereas the set > of tasks in the latter is a subset of tasks. Also noteworthy but slightly > less important: the problem will be more visible to users with the former > (the connector will still be marked FAILED) than with the latter. > > Cheers, > > Chris > > On Tue, Nov 21, 2023, 00:53 Hector Geraldino (BLOOMBERG/ 919 3RD A) < > hgerald...@bloomberg.net> wrote: > > > Thanks for the KIP Chris, adding this check makes total sense. > > > > I do have one question. The second paragraph in the Public Interfaces > > section states: > > > > "If the connector generated excessive tasks after being reconfigured, > then > > any existing tasks for the connector will be allowed to continue running, > > unless that existing set of tasks also exceeds the tasks.max property." > > > > Would not failing the connector land us in the second scenario of > > 'Rejected Alternatives'? > > > > From: dev@kafka.apache.org At: 11/11/23 00:27:44 UTC-5:00To: > > dev@kafka.apache.org > > Subject: [DISCUSS] KIP-1004: Enforce tasks.max property in Kafka Connect > > > > Hi all, > > > > I'd like to open up KIP-1004 for discussion: > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1004%3A+Enforce+tasks.max+ > > property+in+Kafka+Connect > > > > As a brief summary: this KIP proposes that the Kafka Connect runtime > start > > failing connectors that generate a greater number of tasks than the > > tasks.max property, with an optional emergency override that can be used > to > > continue running these (probably-buggy) connectors if absolutely > necessary. > > > > I'll be taking time off most of the next three weeks, so response latency > > may be a bit higher than usual, but I wanted to kick off the discussion > in > > case we can land this in time for the upcoming 3.7.0 release. > > > > Cheers, > > > > Chris > > > > > > >
