Hi David,
That's a very interesting KIP and I wanted to share my two cents. I believe
there's a lot of value and use cases for the ability to intercept, mutate
and filter Kafka's messages, however I'm not sure if trying to achieve that
via in-band interceptors is the best approach for this.
- My mental model around one of Kafka's core values is the brokers'
focus on a single functionality (more or less): highly available and fault
tolerant commit log. I see this in many design decisions such as
off-loading responsibilities to the clients (partitioner, assignor,
consumer groups coordination etc).
- And the impact of this KIP on the Kafka server would be adding another
moving part to their "state of the world" that they try to maintain. What
if an interceptor goes bad? What if there're version-mismatch? etc, a lot
of responsibilities that can be managed very efficiently out-of-band IMHO.
- The comparison to NginX and Kubernetes is IMHO comparing apples to
oranges
- NginX
- Doesn't maintain persisted data.
- It's designed as a middleware, it's an interceptor by nature.
- Kubernetes
- CRDs extend the API surface, they don't "augment" existing APIs.
I think admission webhooks
<https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/>
is
Kubernetes' solution for providing interceptors.
- The admission webhooks are out-of-band, and in fact they're a
great example of "opening up your cluster for extensibility"
going wrong.
Installing a misbehaving webhook can brick the whole cluster.
As I mentioned, I see a value for users being able to intercept and
transform Kafka's messages. But I'm worried that having this as a core
Kafka feature might not be the best approach for achieving that.
Thanks,
--
Ahmed Abdalla
T: @devguyio <https://twitter.com/devguyio>
On Thu, Feb 9, 2023 at 8:28 PM David Mariassy <david.maria...@gmail.com>
wrote:
> Hi everyone,
>
> I'd like to get a discussion going for KIP-905
> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-905%3A+Broker+interceptors
> >,
> which proposes the addition of broker interceptors to the stack.
>
> The KIP contains the motivation, and lists the new public interfaces that
> this change would entail. Since my company had its quarterly hack days this
> week, I also took the liberty to throw together a first prototype of the
> proposed new feature here: https://github.com/apache/kafka/pull/13224.
>
> Looking forward to the group's feedback!
>
> Thanks,
> David
>
--
*Ahmed Abdalla*
