Re: Using Kafka's ProducerInterceptor with Samza

2019-10-08 Thread Cameron Lee
That's another interesting data point that it works fine when using ThreadJobFactory. I'm still not sure why it's not working in YARN though. If you want, I have a a couple of suggestions you can try: 1) Implement a no-op interceptor and only add the dependencies needed for that. This could possibl

Re: Using Kafka's ProducerInterceptor with Samza

2019-10-07 Thread Malcolm McFarland
Thanks for the advice. I'm pretty sure that I referenced the Kafka 0.11 docs when implementing this (we're still on Samza 0.14.1), but I'll double check. The really strange thing is that the processors have no problems starting when I use a ThreadJobFactory, it's only radio silence on a YARN deploy

Re: Using Kafka's ProducerInterceptor with Samza

2019-10-07 Thread Cameron Lee
Hey Malcolm, Sorry for not responding to this earlier. Unfortunately, we don't have an example in which ProducerInterceptor is used in a Samza application. It's odd that the whole task fails when you try to use your interceptor. The only thing I could think of is that the Kafka producer fails to p

Re: Using Kafka's ProducerInterceptor with Samza

2019-10-07 Thread Malcolm McFarland
Hey y'all, just wanted to follow up on this: does anybody have a working example of using Kafka's ProducerInterceptor interface under YARN that I could take a look at, or that they could provide some guidance with? More generally, we're trying to keep an eye on Samza SSP instances that occasionall

Using Kafka's ProducerInterceptor with Samza

2019-09-10 Thread Malcolm McFarland
Hey folks, I'm trying to intercept outgoing Kafka messages before they're sent to do some basic tracking (ie get the actual assigned partition number, which isn't available in the StreamTask or MessageCollector classes). I've written a class that conforms to Kafka's ProducerInterceptor interface [