Re: RecordTooLargeException recovery

2016-10-06 Thread David Yu
Xinyu, Thanks for the answers. Those suggestions are helpful as well. David On Thu, Oct 6, 2016 at 12:48 PM xinyu liu wrote: > Hi, David, > > For your questions: > > 1) In this case Samza recovered but the changelog message was lost. In > 0.10.1 KafkaSystemProducer has a race condition: there

Re: RecordTooLargeException recovery

2016-10-06 Thread xinyu liu
Hi, David, For your questions: 1) In this case Samza recovered but the changelog message was lost. In 0.10.1 KafkaSystemProducer has a race condition: there is small chance the later send success might override the previous failure. The bug is fixed in the upcoming 0.11.0 release (SAMZA-1019). Th

RecordTooLargeException recovery

2016-10-06 Thread David Yu
Hi, Our Samza job (0.10.1) throws RecordTooLargeExceptions when flushing the KV store change to the changelog topic, as well as sending outputs to Kafka. We have two questions to this problem: 1. It seems that after the affected containers failed multiple times, the job was able to recover and mo