Hello all, There is an improvement in KIP-877 with the following changes:
1. Removed KafkaPrincipalBuilder plugin: Its lifecycle is short, so adding metrics for it would be meaningless. 2. Added new extra tags for Login, and AuthenticateCallbackHandler plugins to avoid collisions. These plugins now define metrics under three different scenarios: a. Server component, ConnectionMode = Server → registers: "mechanism", "networkProcessor", "listener" b. Server component, ConnectionMode = Client → registers: "component", "node-id" c. Client component, ConnectionMode = Client → no extra tags 3. Added new extra tags for SslEngineFactory plugins to avoid collisions. These plugins now define metrics under three different scenarios: a. Server component, ConnectionMode = Server → registers: "networkProcessor", "listener" b. Server component, ConnectionMode = Client → registers: "component", "node-id" c. Client component, ConnectionMode = Client → no extra tags <https://cwiki.apache.org/confluence/x/lY3GDQ> Best Regards, Jiunn-Yang > Jun Rao <[email protected]> 於 2025年4月25日 凌晨2:47 寫道： > > Hi, Jiunn-Yang, > > Thanks for the explanation. The change sounds good to me then. > > Jun > > On Wed, Apr 23, 2025 at 4:57 AM 黃竣陽 <[email protected]> wrote: > >> Hello Jun, >> >> At this stage, KAFKA-18390 only involves modifications to internal data >> structures. Interface-level changes >> are currently out of scope. If any interface changes in the future, they >> will be tracked under a separate JIRA. >> >> Best Regards, >> Jiunn-Yang >> >>> Mickael Maison <[email protected]> 於 2025年4月23日 下午6:07 寫道： >>> >>> Hi Jun, >>> >>> KIP-877 is not in a release yet. So assuming nobody objects to the >>> change, we can do it before 4.1.0. >>> Then any further changes will require another KIP. >>> >>> Thanks, >>> Mickael >>> >>> >>> On Tue, Apr 22, 2025 at 8:05 PM Jun Rao <[email protected]> >> wrote: >>>> >>>> Hi, Jiunn-Yang, >>>> >>>> Thanks for the improvement. >>>> >>>> KAFKA-18390 changes a public interface. Do we have a KIP for that? >>>> >>>> Jun >>>> >>>> On Tue, Apr 22, 2025 at 3:39 AM 黃竣陽 <[email protected]> wrote: >>>> >>>>> Hello, >>>>> >>>>> I haven’t seen any feedback, so I’ll go ahead and implement the >>>>> improvement >>>>> regarding PluginMetrics#metricName. >>>>> >>>>> Best Regards, >>>>> Jiunn-Yang >>>>> >>>>>> 黃竣陽 <[email protected]> 於 2025年4月14日 晚上11:43 寫道： >>>>>> >>>>>> Hi, >>>>>> >>>>>> There is an improvement in KIP-877. >>>>>> >>>>>> There will be an update to the PluginMetrics#metricName method: the >> type >>>>> of the tags parameter will be changed >>>>>> from Map to LinkedHashMap. >>>>>> This change is necessary because the order of metric tags is important >>>>>> 1. If the tag order is inconsistent, identical metrics may be treated >> as >>>>> distinct ones by the metrics backend >>>>>> 2. KAFKA-18390 is updating metric naming to use LinkedHashMap. For >>>>> consistency, we should follow the same approach here. >>>>>> >>>>>> <https://cwiki.apache.org/confluence/x/lY3GDQ> >>>>>> >>>>>> Best Regards, >>>>>> Jiunn-Yang >>>>>> >>>>>>> Luke Chen <[email protected]> 於 2025年3月12日 上午10:21 寫道： >>>>>>> >>>>>>> HI Mickael, >>>>>>> >>>>>>> Thanks for the update. >>>>>>> The extra tags addition makes sense to me. >>>>>>> >>>>>>> Thanks. >>>>>>> Luke >>>>>>> >>>>>>> On Tue, Mar 11, 2025 at 8:31 PM Mickael Maison < >>>>> [email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> I've not seen any concerns or feedback. >>>>>>>> I'll wait a few more days, then we'll carry on the implementation of >>>>>>>> KIP-877 with the updated proposal (extra tags for some server side >>>>>>>> plugin). >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Mickael >>>>>>>> >>>>>>>> On Tue, Mar 4, 2025 at 6:57 PM Mickael Maison < >>>>> [email protected]> >>>>>>>> wrote: >>>>>>>>> >>>>>>>>> Hi, >>>>>>>>> >>>>>>>>> I found an issue while implementing KIP-877 for server plugins. >>>>>>>>> >>>>>>>>> Several server plugins can be instantiated multiple times. This can >>>>>>>> happen: >>>>>>>>> - when a server is running in combined mode (for example >> Authorizer is >>>>>>>>> instantiated by both brokers and controllers) >>>>>>>>> - when using multiple listeners (for example KafkaPrincipalBuilder >> is >>>>>>>>> instantiated per processor per listener) >>>>>>>>> In the voted proposal, I did not include enough tags for some >> plugins >>>>>>>>> to uniquely identify each instance and avoid metric name >> collisions. >>>>>>>>> >>>>>>>>> For that reason, I updated the KIP to specify extra tags (in >> addition >>>>>>>>> of "config" and "class") for the following plugin types: >>>>>>>>> - KafkaPrincipalBuilder >>>>>>>>> - Authorizer >>>>>>>>> - ClientQuotaCallback >>>>>>>>> - ConfigProvider >>>>>>>>> - AuthenticateCallbackHandler >>>>>>>>> - Login >>>>>>>>> - SslEngineFactory >>>>>>>>> >>>>>>>>> You can find the exact tags in the Proposed Changes section of the >>>>> KIP: >>>>>>>>> >>>>>>>> >>>>> >> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=231116181#KIP877:Mechanismforpluginsandconnectorstoregistermetrics-ProposedChanges >>>>>>>>> >>>>>>>>> Let me know if you have questions or concerns. >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Mickael >>>>>>>>> >>>>>>>>> >>>>>>>>> On Thu, Nov 14, 2024 at 3:07 PM Mickael Maison < >>>>> [email protected]> >>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> While implementing the KIP I noticed the Kafka Connect specifics >> had >>>>>>>>>> not been updated correctly based on the latest discussions. The >>>>>>>>>> consensus was that connector and tasks should use their context >> to >>>>>>>>>> retrieve the PluginMetrics instance if they want to register >> metrics. >>>>>>>>>> While it's a slightly different mechanism than for the other >> plugins >>>>>>>>>> (they need to implement Monitorable), this allows connector to >>>>> support >>>>>>>>>> this new feature and still work when deployed on an older Connect >>>>>>>>>> runtime. >>>>>>>>>> >>>>>>>>>> I have updated the KIP and I have 2 PRs ready for review: >>>>>>>>>> - Adding the APIs and support for producer and consumer plugins: >>>>>>>>>> https://github.com/apache/kafka/pull/17511 >>>>>>>>>> - Adding support for Connect plugins: >>>>>>>> https://github.com/apache/kafka/pull/17804 >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Mickael >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Thu, Aug 8, 2024 at 2:21 PM Mickael Maison < >>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>> Hi, >>>>>>>>>>> >>>>>>>>>>> I'm closing the vote. The vote has passed with >>>>>>>>>>> + 3 bindings votes from Chris, Tom and Luke, and +1 non-binding >> vote >>>>>>>>>>> from Hector. >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> Mickael >>>>>>>>>>> >>>>>>>>>>> On Thu, Aug 8, 2024 at 11:28 AM Luke Chen <[email protected]> >>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>> Hi Mickael, >>>>>>>>>>>> >>>>>>>>>>>> Thanks for the KIP. >>>>>>>>>>>> +1 (binding) from me. >>>>>>>>>>>> >>>>>>>>>>>> Thanks. >>>>>>>>>>>> Luke >>>>>>>>>>>> >>>>>>>>>>>> On Fri, Aug 2, 2024 at 4:57 AM Tom Bentley <[email protected] >>> >>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> +1 (binding). >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks Mickael! >>>>>>>>>>>>> >>>>>>>>>>>>> On Thu, 1 Aug 2024 at 05:12, Mickael Maison < >>>>>>>> [email protected]> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>> >>>>>>>>>>>>>> Bumping this thread to get some more votes and/or feedback. >>>>>>>>>>>>>> >>>>>>>>>>>>>> As I restarted the vote on June 10 after major changes, I'm >>>>>>>> only >>>>>>>>>>>>>> counting votes since then. >>>>>>>>>>>>>> So we have 1 binding (Chris) and 1 non-binding (Hector) votes. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>> Mickael >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Mon, Jul 8, 2024 at 8:40 PM Hector Geraldino (BLOOMBERG/ >>>>>>>> 919 3RD A) >>>>>>>>>>>>>> <[email protected]> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This will help eliminate some boilerplate code we have for >>>>>>>> our >>>>>>>>>>>>>> connectors. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> +1 (non-binding) >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> From: [email protected] At: 06/25/24 04:30:27 UTC-4:00To: >>>>>>>>>>>>>> [email protected] >>>>>>>>>>>>>>> Subject: Re: [VOTE] KIP-877: Mechanism for plugins and >>>>>>>> connectors to >>>>>>>>>>>>>> register metrics >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Bumping this thread. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Let me know if you have any feedback. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>> Mickael >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Mon, Jun 10, 2024 at 1:44 PM Chris Egerton < >>>>>>>> [email protected] >>>>>>>>>>>>>> >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> +1 (binding), thanks Mickael! >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Mon, Jun 10, 2024, 04:24 Mickael Maison < >>>>>>>> [email protected] >>>>>>>>>>>>>> >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Following the feedback in the DISCUSS thread, I made >>>>>>>> significant >>>>>>>>>>>>>>>>> changes to the proposal. So I'd like to restart a vote >>>>>>>> for KIP-877: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>> >>>>> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-877%3A+Mechanism+for+plugi >>>>>>>>>>>>>>> ns+and+connectors+to+register+metrics >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>> Mickael >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Thu, Jan 25, 2024 at 2:59 AM Tom Bentley < >>>>>>>> [email protected]> >>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Hi Mickael, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> You'll have seen that I left some comments on the >>>>>>>> discussion >>>>>>>>>>>>>> thread, but >>>>>>>>>>>>>>>>>> they're minor enough that I'm happy to vote +1 here. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Tom >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Thu, 11 Jan 2024 at 06:14, Mickael Maison < >>>>>>>>>>>>>> [email protected]> >>>>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Bumping this thread since I've not seen any feedback. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>>> Mickael >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> On Tue, Dec 19, 2023 at 10:03 AM Mickael Maison >>>>>>>>>>>>>>>>>>> <[email protected]> wrote: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Hi, >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> I'd like to start a vote on KIP-877: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>> >>>>> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-877%3A+Mechanism+for+plugi >>>>>>>>>>>>>>> ns+and+connectors+to+register+metrics >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Let me know if you have any feedback. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>>>> Mickael >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>> >>>>>> >>>>> >>>>> >> >>
