Magnus, I think these are arguments for observability in general, but not why kafka should sit between a client and a metics collector.
Ryanne On Wed, Jun 16, 2021, 10:27 AM Magnus Edenhill <mag...@edenhill.se> wrote: > Hi Ryanne, > > this proposal stems from a need to improve troubleshooting Kafka issues. > > As it currently stands, when an application team is experiencing Kafka > service degradation, > or the Kafka operator is seeing misbehaving clients, there are plenty of > steps that needs > to be taken before any client-side metrics can be observed at all, if at > all: > - Is the application even collecting client metrics? If not it needs to be > reconfigured or implemented, and restarted; > a restart may have business impact, and may also temporarily? remedy the > problem without giving any further insight > into what was wrong. > - Are the desired metrics collected? Where are they stored? For how long? > Is there enough correlating information > to map it to cluster-side metrics and events? Does the application > on-call know how to find the collected metrics? > - Export and send these metrics to whoever knows how to interpret them. In > what format? Are all relevant metadata fields > provided? > > The KIP aims to solve all these obstacles by giving the Kafka operator the > tools to collect this information. > > Regards, > Magnus > > > Den tis 15 juni 2021 kl 02:37 skrev Ryanne Dolan <ryannedo...@gmail.com>: > > > Magnus, I think such a substantial change requires more motivation than > is > > currently provided. As I read it, the motivation boils down to this: you > > want your clients to phone-home unless they opt-out. As stated in the > KIP, > > "there are plenty of existing solutions [...] to send metrics [...] to a > > collector", so the opt-out appears to be the only motivation. Am I > missing > > something? > > > > Ryanne > > > > On Wed, Jun 2, 2021 at 7:46 AM Magnus Edenhill <mag...@edenhill.se> > wrote: > > > > > Hey all, > > > > > > I'm proposing KIP-714 to add remote Client metrics and observability. > > > This functionality will allow centralized monitoring and > troubleshooting > > of > > > clients and their internals. > > > > > > Please see > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-714%3A+Client+metrics+and+observability > > > > > > Looking forward to your feedback! > > > > > > Regards, > > > Magnus > > > > > >
