Re: [VOTE] KIP-976: Cluster-wide dynamic log adjustment for Kafka Connect

2023-10-11 Thread Chris Egerton
as well > > De: dev@kafka.apache.org A: 10/09/23 05:16:06 UTC-4:00A: > dev@kafka.apache.org > Subject: Re: [VOTE] KIP-976: Cluster-wide dynamic log adjustment for Kafka > Connect > > Hi Chris, > > +1 (non binding) > > Thanks > Fede > > On Sun, Oct 8, 2023 at

Re: [VOTE] KIP-976: Cluster-wide dynamic log adjustment for Kafka Connect

2023-10-09 Thread Hector Geraldino (BLOOMBERG/ 919 3RD A)
Good stuff, +1 (non-binding) from me as well De: dev@kafka.apache.org A: 10/09/23 05:16:06 UTC-4:00A: dev@kafka.apache.org Subject: Re: [VOTE] KIP-976: Cluster-wide dynamic log adjustment for Kafka Connect Hi Chris, +1 (non binding) Thanks Fede On Sun, Oct 8, 2023 at 10:11 AM Yash Mayya

Re: Re: [VOTE] KIP-976: Cluster-wide dynamic log adjustment for Kafka Connect

2023-10-09 Thread hudeqi
Hi Chris, +1 (non-binding) Finally, there is no need to use external intrusion tools to change the log level of kafka connect online! Thanks for the KIP! best, hudeqi

Re: [VOTE] KIP-976: Cluster-wide dynamic log adjustment for Kafka Connect

2023-10-09 Thread Mickael Maison
2023 at 9:05 AM Chris Egerton > > > wrote: > > > > > > > > Hi all, > > > > > > > > I'd like to call for a vote on KIP-976, which augments the existing > > > dynamic > > > > logger adjustment REST API for

Re: [VOTE] KIP-976: Cluster-wide dynamic log adjustment for Kafka Connect

2023-10-09 Thread Federico Valeri
to apply changes > > cluster-wide > > > instead on a per-worker basis. > > > > > > The KIP: > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-976:+Cluster-wide+dynamic+log+adjustment+for+Kafka+Connect > > > > > > The discussion thread: > > > https://lists.apache.org/thread/w3x3f3jmyd1vfjxho06y8xgt6mhhzpl5 > > > > > > Cheers, > > > > > > Chris > >

Re: [VOTE] KIP-976: Cluster-wide dynamic log adjustment for Kafka Connect

2023-10-08 Thread Yash Mayya
gt; > logger adjustment REST API for Kafka Connect to apply changes > cluster-wide > > instead on a per-worker basis. > > > > The KIP: > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-976:+Cluster-wide+dynamic+log+adjustment+for+Kafka+Connect >

Re: [VOTE] KIP-976: Cluster-wide dynamic log adjustment for Kafka Connect

2023-10-06 Thread Greg Harris
: > > Hi all, > > I'd like to call for a vote on KIP-976, which augments the existing dynamic > logger adjustment REST API for Kafka Connect to apply changes cluster-wide > instead on a per-worker basis. > > The KIP: > https://cwiki.apache.org/confluence/display/KAFKA/K

[VOTE] KIP-976: Cluster-wide dynamic log adjustment for Kafka Connect

2023-10-06 Thread Chris Egerton
Hi all, I'd like to call for a vote on KIP-976, which augments the existing dynamic logger adjustment REST API for Kafka Connect to apply changes cluster-wide instead on a per-worker basis. The KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP-976:+Cluster-wide+dynamic+log+adjus

Re: KIP-976: Cluster-wide dynamic log adjustment for Kafka Connect

2023-09-10 Thread Sagar
Thanks Chris, The changes look good to me. 1) Regarding the suggestion to reduce the key sizes, the only intent was to make it easier to read. But then I missed the fact that the "org.apache.kafka.connect" isn't always going to be the prefix for these keys. We can live with whatever we have 2) H

Re: KIP-976: Cluster-wide dynamic log adjustment for Kafka Connect

2023-09-07 Thread Chris Egerton
Hi all, Thanks again for the reviews! Sagar: > The updated definition of last_modified looks good to me. As a continuation to point number 2, could we also mention that this could be used to get insights into the propagation of the cluster wide log level updates. It is implicit but probably bet

Re: KIP-976: Cluster-wide dynamic log adjustment for Kafka Connect

2023-09-06 Thread Yash Mayya
Hi Chris, Thanks for the clarification on the last modified timestamp tracking here and on the KIP, things look good to me now. On the persistence front, I hadn't considered the interplay between levels set by the log level REST APIs and those set by the log4j configuration files, and the user co

Re: KIP-976: Cluster-wide dynamic log adjustment for Kafka Connect

2023-09-06 Thread Ashwin
Hi Chris, > I agree with Yash here: we can cross that bridge when we come to it, unless there are problems that we'd encounter then that would be better addressed by adding request forwarding now. One potential argument I can think of is that the UX would be a little strange if the semantics for t

Re: KIP-976: Cluster-wide dynamic log adjustment for Kafka Connect

2023-09-05 Thread Sagar
Hey Chris, Thanks for making the updates. The updated definition of last_modified looks good to me. As a continuation to point number 2, could we also mention that this could be used to get insights into the propagation of the cluster wide log level updates. It is implicit but probably better to

Re: KIP-976: Cluster-wide dynamic log adjustment for Kafka Connect

2023-09-05 Thread Chris Egerton
Hi Sagar, Thanks for your thoughts! Responses inline: > 1) Considering that you have now clarified here that last_modified field would be stored in-memory, it is not mentioned in the KIP. Although that's something that's understandable, it wasn't apparent when reading the KIP. Probably, we could

Re: KIP-976: Cluster-wide dynamic log adjustment for Kafka Connect

2023-09-05 Thread Sagar
Hey Chris, Thanks for the KIP. Seems like a useful feature. I have some more questions/comments: 1) Considering that you have now clarified here that last_modified field would be stored in-memory, it is not mentioned in the KIP. Although that's something that's understandable, it wasn't apparent

Re: KIP-976: Cluster-wide dynamic log adjustment for Kafka Connect

2023-09-05 Thread Chris Egerton
Hi all, Thank you so much for the generous review comments! Happy to see interest in this feature. Inline responses follow. Ashwin: > Don't you foresee a future scope type which may require cluster metadata ? In that case, isn't it better to forward the requests to the leader in the initial imp

Re: KIP-976: Cluster-wide dynamic log adjustment for Kafka Connect

2023-09-03 Thread Federico Valeri
Hi Chris, thanks. This looks like a useful feature. Due to the idempotent nature of PUT, I guess that the last_modified timestamp won't change if the same request is repeated successively. Should we add a unit test for that? On Mon, Sep 4, 2023 at 6:17 AM Ashwin wrote: > > Hi Chris, > > Thanks f

Re: KIP-976: Cluster-wide dynamic log adjustment for Kafka Connect

2023-09-03 Thread Yash Mayya
> If no modifications to a logging namespace have > been made, won't the namespace itself be omitted > from the response? It looks like we currently only > return loggers that have non-null log levels in the *> **GET /admin/loggers* endpoint. This can be ignored - I didn't account for the fact tha

Re: KIP-976: Cluster-wide dynamic log adjustment for Kafka Connect

2023-09-03 Thread Yash Mayya
Hi Chris, Thanks for the KIP, this looks like a really useful addition to Kafka Connect's log level REST APIs! I have a few questions and comments: > If no modifications to the namespace have > been made since the worker was started, > they will be null If no modifications to a logging namespace

Re: KIP-976: Cluster-wide dynamic log adjustment for Kafka Connect

2023-09-03 Thread Ashwin
Hi Chris, Thanks for thinking about this useful feature ! I had a question regarding > Since cluster metadata is not required to handle these types of request, they will not be forwarded to the leader And later, we also mention about supporting more scope types in the future. Don't you foresee a

KIP-976: Cluster-wide dynamic log adjustment for Kafka Connect

2023-09-01 Thread Chris Egerton
Hi all, Can't imagine a worse time to publish a new KIP (it's late on a Friday and we're in the middle of the 3.6.0 release), but I wanted to put forth KIP-976 for discussion: https://cwiki.apache.org/confluence/display/KAFKA/KIP-976%3A+Cluster-wide+dynamic+log+adjustment+for+Kafka+Connect TL;DR: