Re: Using Kafka's ProducerInterceptor with Samza

2019-10-08 Thread Cameron Lee
That's another interesting data point that it works fine when using ThreadJobFactory. I'm still not sure why it's not working in YARN though. If you want, I have a a couple of suggestions you can try: 1) Implement a no-op interceptor and only add the dependencies needed for that. This could possibl

Re: Using Kafka's ProducerInterceptor with Samza

2019-10-07 Thread Malcolm McFarland
Thanks for the advice. I'm pretty sure that I referenced the Kafka 0.11 docs when implementing this (we're still on Samza 0.14.1), but I'll double check. The really strange thing is that the processors have no problems starting when I use a ThreadJobFactory, it's only radio silence on a YARN deploy

Re: Using Kafka's ProducerInterceptor with Samza

2019-10-07 Thread Cameron Lee
Hey Malcolm, Sorry for not responding to this earlier. Unfortunately, we don't have an example in which ProducerInterceptor is used in a Samza application. It's odd that the whole task fails when you try to use your interceptor. The only thing I could think of is that the Kafka producer fails to p

Re: Using Kafka's ProducerInterceptor with Samza

2019-10-07 Thread Malcolm McFarland
Hey y'all, just wanted to follow up on this: does anybody have a working example of using Kafka's ProducerInterceptor interface under YARN that I could take a look at, or that they could provide some guidance with? More generally, we're trying to keep an eye on Samza SSP instances that occasionall