Re: Broker plugins

2014-03-13 Thread Johan Lundahl
Thanks for the response, indeed the encryption/concealment should ideally be done on the producer side but it's just not feasible in some of our applications so a middle layer would be needed. So far our thoughts have been around using Flume interceptors but that means introducing another "moving

Re: Broker plugins

2014-03-13 Thread Neha Narkhede
In general, the preference has been to avoid having user code run on the brokers since that just opens a can of worms where the broker logic get's complicated trying to deal with errors that the user code can throw. The suggestion is to push any user specific processing to the client side. In this

Broker plugins

2014-03-13 Thread Johan Lundahl
Hi, I have a use case for which it would be useful with pluggable processing functions in the broker. We have some data containing sensitive information which is legally ok to transmit over the internal network to the Kafka brokers and keep in volatile memory but not to flush to disk unconcealed/