Hi, to unsubscribe you need to send a mail to user-unsubscr...@flink.apache.org<mailto:user-unsubscr...@flink.apache.org>.
Best, Zhanghao Chen ________________________________ From: Samir Vasani <samirvas...@gmail.com> Sent: Saturday, April 16, 2022 1:08:20 PM To: Jai Patel <jai.pa...@cloudkitchens.com> Cc: Nicolaus Weidner <nicolaus.weid...@ververica.com>; user <user@flink.apache.org>; Weixiang Sun <weixiang....@cloudkitchens.com> Subject: Re: java.lang.Exception: Job leader for job id 0efd8681eda64b072b72baef58722bc0 lost leadership. how to unsubscribe? On Sat, Apr 16, 2022 at 3:21 AM Jai Patel <jai.pa...@cloudkitchens.com<mailto:jai.pa...@cloudkitchens.com>> wrote: Hi Nico, Wanted to close the loop here. We did end up find a number of problems in our code: 1. Our operator was slow. It was iterating over several large Protobufs in a MapState then filtering it down to 1. We were able to identify that one up-front and significantly improve the runtime of the operator. 2. We did increase the total memory and raised the managed memory fraction from 40% to 70%. Both solutions were needed in order to bring us the stability we were looking for. Since then, everything has been running great. Thanks. Jai On Thu, Feb 24, 2022 at 2:42 AM Nicolaus Weidner <nicolaus.weid...@ververica.com<mailto:nicolaus.weid...@ververica.com>> wrote: Hi Jai, Do writes to ValueStates/MapStates have a direct on churn of the Flink State or is the data buffered in between? Writes to keyed state go directly to RocksDB. So there shouldn't be any memory issues with buffers overflowing or similar. In general, more memory should increase performance (larger cache sizes before having to write to disk), but less memory shouldn't cause crashes. Since the errors you encountered are not that specific, can you provide full logs surrounding such incidents? There is not much to go on without further info. Best, Nico
