Hi Ismael,
How come? They're public in the clients jar. I'm not doubting you, all I'm
really asking is "how should I have known this?"
Tom
On Fri, Dec 6, 2019 at 4:12 PM Ismael Juma <ism...@juma.me.uk> wrote:
> AbstractRequest and AbstractResponse are currently internal classes.
>
> Ismael
>
> On Fri, Dec 6, 2019 at 1:56 AM Tom Bentley <tbent...@redhat.com> wrote:
>
> > Hi,
> >
> > Couldn't this be done without exposing broker internals at the slightly
> > higher level of AbstractRequest and AbstractResponse? Those classes are
> > public. If the observer interface used Java default methods then adding a
> > new request type would not break existing implementations. I'm thinking
> > something like this:
> >
> > ```
> > public interface RequestObserver {
> > default void observeAny(RequestContext context, AbstractRequest
> > request) {}
> > default void observe(RequestContext context, MetadataRequest
> request) {
> > observeAny(context, request);
> > }
> > default void observe(RequestContext context, ProduceRequest request)
> {
> > observeAny(context, request);
> > }
> > default void observe(RequestContext context, FetchRequest request) {
> > observeAny(context, request);
> > }
> > ...
> > ```
> >
> > And similar for a `ResponseObserver`. Request classes would implement
> this
> > method
> >
> > ```
> > public abstract void observeForAudit(RequestContext context,
> > RequestObserver requestObserver);
> > ```
> >
> > where the implementation would look like this:
> >
> > ```
> > @Override
> > public void observe(RequestContext context, RequestObserver
> > requestObserver) {
> > requestObserver.observe(context, this);
> > }
> > ```
> >
> > I think this sufficiently abstracted to allow KafkaApis.handle() and
> > sendResponse() to call observe() generically.
> >
> > Kind regards,
> >
> > Tom
> >
> > On Wed, Dec 4, 2019 at 6:59 PM Lincong Li <andrewlinc...@gmail.com>
> wrote:
> >
> > > Hi Thomas,
> > >
> > > Thanks for your interest in KIP-388. As Ignacio and Radai have
> mentioned,
> > > this
> > > <
> > >
> >
> https://github.com/linkedin/kafka/commit/a378c8980af16e3c6d3f6550868ac0fd5a58682e
> > > >
> > > is our (LinkedIn's) implementation of KIP-388. The implementation and
> > > deployment of this broker-side observer has been working very well for
> us
> > > by far. On the other hand, I totally agree with the longer-term
> concerns
> > > raised by other committers. However we still decided to implement the
> KIP
> > > idea as a hot fix in order to solve our immediate problem and meet our
> > > business requirements.
> > >
> > > The "Rejected Alternatives for Kafka Audit" section at the end of
> KIP-388
> > > sheds some lights on the client-side auditor/interceptor/observer
> (sorry
> > > about the potential confusion caused by these words being used
> > > interchangeably).
> > >
> > > Best regards,
> > > Lincong Li
> > >
> > > On Wed, Dec 4, 2019 at 8:15 AM Thomas Aley <thomas.a...@ibm.com>
> wrote:
> > >
> > > > Thanks for the responses. I did worry about the challenge of
> exposing a
> > > > vast number of internal classes with general interceptor framework. A
> > > less
> > > > general solution more along the lines of the producer/consumer
> > > > interceptors on the client would satisfy the majority of use cases.
> If
> > we
> > > > are smart, we should be able to come up with a pattern that could be
> > > > extended further in future if the community sees the demand.
> > > >
> > > > Looking through the discussion thread for KIP-388, I see a lot of
> good
> > > > points to consider and I intend to dive further into this.
> > > >
> > > >
> > > > Tom Aley
> > > > thomas.a...@ibm.com
> > > >
> > > >
> > > >
> > > > From: Ismael Juma <ism...@juma.me.uk>
> > > > To: Kafka Users <us...@kafka.apache.org>
> > > > Cc: dev <dev@kafka.apache.org>
> > > > Date: 03/12/2019 16:12
> > > > Subject: [EXTERNAL] Re: Broker Interceptors
> > > >
> > > >
> > > >
> > > > The main challenge is doing this without exposing a bunch of internal
> > > > classes. I haven't seen a proposal that handles that aspect well so
> > far.
> > > >
> > > > Ismael
> > > >
> > > > On Tue, Dec 3, 2019 at 7:21 AM Sönke Liebau
> > > > <soenke.lie...@opencore.com.invalid> wrote:
> > > >
> > > > > Hi Thomas,
> > > > >
> > > > > I think that idea is worth looking at. As you say, if no
> interceptor
> > is
> > > > > configured then the performance overhead should be negligible.
> > > Basically
> > > > it
> > > > > is then up to the user to decide if he wants tomtake the
> performance
> > > > hit.
> > > > > We should make sure to think about monitoring capabilities like
> time
> > > > spent
> > > > > in the interceptor for records etc.
> > > > >
> > > > > The most obvious use case I think is server side schema validation,
> > > > which
> > > > > Confluent are also offering as part of their commercial product,
> but
> > > > other
> > > > > ideas come to mind as well.
> > > > >
> > > > > Best regards,
> > > > > Sönke
> > > > >
> > > > > Thomas Aley <thomas.a...@ibm.com> schrieb am Di., 3. Dez. 2019,
> > 10:45:
> > > > >
> > > > > > Hi M. Manna,
> > > > > >
> > > > > > Thank you for your feedback, any and all thoughts on this are
> > > > appreciated
> > > > > > from the community.
> > > > > >
> > > > > > I think it is important to distinguish that there are two parts
> to
> > > > this.
> > > > > > One would be a server side interceptor framework and the other
> > would
> > > > be
> > > > > > the interceptor implementations themselves.
> > > > > >
> > > > > > The idea would be that the Interceptor framework manifests as a
> > plug
> > > > > point
> > > > > > in the request/response paths that by itself has negligible
> > > > performance
> > > > > > impact as without an interceptor registered in the framework it
> is
> > > > > > essentially a no-op. This way the out-the-box behavior of the
> Kafka
> > > > > broker
> > > > > > remains essentially unchanged, it is only if the cluster
> > > administrator
> > > > > > registers an interceptor into the framework that the path of a
> > record
> > > > is
> > > > > > intercepted. This is much like the already accepted and
> implemented
> > > > > client
> > > > > > interceptors - the capability exists and it is an opt-in feature.
> > > > > >
> > > > > > As with the client interceptors and indeed interception in
> general,
> > > > the
> > > > > > interceptor implementations need to be thoughtfully crafted to
> > ensure
> > > > > > minimal performance impact. Yes the interceptor framework could
> tap
> > > > into
> > > > > > nearly everything but would only be tapping into the subset of
> APIs
> > > > that
> > > > > > the user wishes to intercept for their use case.
> > > > > >
> > > > > > Tom Aley
> > > > > > thomas.a...@ibm.com
> > > > > >
> > > > > >
> > > > > >
> > > > > > From: "M. Manna" <manme...@gmail.com>
> > > > > > To: Kafka Users <us...@kafka.apache.org>
> > > > > > Cc: dev@kafka.apache.org
> > > > > > Date: 02/12/2019 11:31
> > > > > > Subject: [EXTERNAL] Re: Broker Interceptors
> > > > > >
> > > > > >
> > > > > >
> > > > > > Hi Tom,
> > > > > >
> > > > > > On Mon, 2 Dec 2019 at 09:41, Thomas Aley <thomas.a...@ibm.com>
> > > wrote:
> > > > > >
> > > > > > > Hi Kafka community,
> > > > > > >
> > > > > > > I am hoping to get some feedback and thoughts about broker
> > > > > interceptors.
> > > > > > >
> > > > > > > KIP-42 Added Producer and Consumer interceptors which have
> > provided
> > > > > > Kafka
> > > > > > > users the ability to collect client side metrics and trace the
> > path
> > > > of
> > > > > > > individual messages end-to-end.
> > > > > > >
> > > > > > > This KIP also mentioned "Adding message interceptor on the
> broker
> > > > makes
> > > > > > a
> > > > > > > lot of sense, and will add more detail to monitoring. However,
> > the
> > > > > > > proposal is to do it later in a separate KIP".
> > > > > > >
> > > > > > > One of the motivations for leading with client interceptors was
> > to
> > > > gain
> > > > > > > experience and see how useable they are before tackling the
> > server
> > > > side
> > > > > > > implementation which would ultimately "allow us to have a more
> > > > > > > complete/detailed message monitoring".
> > > > > > >
> > > > > > > Broker interceptors could also provide more value than just
> more
> > > > > > complete
> > > > > > > and detailed monitoring such as server side schema validation,
> > so I
> > > > am
> > > > > > > curious to learn if anyone in the community has progressed this
> > > > work;
> > > > > > has
> > > > > > > ideas about other potential server side interceptor uses or has
> > > > > actually
> > > > > > > implemented something similar.
> > > > > > >
> > > > > >
> > > > > > I personally feel that the cost here is the impact on
> performance.
> > > If
> > > > I
> > > > > > am
> > > > > > right, this interceptor is going to tap into nearly everything.
> If
> > > you
> > > > > > have
> > > > > > strong guarantee (min.in.sync.replicas = N-1) then this may incur
> > > some
> > > > > > delay (and let's not forget inter broker comms protection by TLS
> > > > config).
> > > > > > This may not be desirable for some systems. That said, it would
> be
> > > > good
> > > > > to
> > > > > > know what others think about this.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > >
> > > > > > > Regards,
> > > > > > >
> > > > > > > Tom Aley
> > > > > > > thomas.a...@ibm.com
> > > > > > > Unless stated otherwise above:
> > > > > > > IBM United Kingdom Limited - Registered in England and Wales
> with
> > > > > number
> > > > > > > 741598.
> > > > > > > Registered office: PO Box 41, North Harbour, Portsmouth,
> > Hampshire
> > > > PO6
> > > > > > 3AU
> > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > Unless stated otherwise above:
> > > > > > IBM United Kingdom Limited - Registered in England and Wales with
> > > > number
> > > > > > 741598.
> > > > > > Registered office: PO Box 41, North Harbour, Portsmouth,
> Hampshire
> > > PO6
> > > > > 3AU
> > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > Unless stated otherwise above:
> > > > IBM United Kingdom Limited - Registered in England and Wales with
> > number
> > > > 741598.
> > > > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire
> PO6
> > > 3AU
> > > >
> > > >
> > >
> >
>
