Re: How to debug checkpoint/savepoint stuck in Flink 1.12.2

2021-03-12 Thread Arvid Heise
Yes, please send me the full stack trace. You could also send it to me personally if you don't want to share it on the ML. I'm especially interested in the legacy source thread that holds the lock 0x00058e8c5070 if you only want to share an excerpt. On Fri, Mar 12, 2021 at 2:29 AM ChangZhuo C

Re: How to debug checkpoint/savepoint stuck in Flink 1.12.2

2021-03-11 Thread 陳昌倬
On Thu, Mar 11, 2021 at 02:14:32PM +0100, Arvid Heise wrote: > Hi ChangZhuo, > > Did you upgrade to Flink 1.12.2 and change the settings at the time? If so, > could you maybe reset the settings to the old values on Flink 1.12.2 and > check if the job still gets stuck? Especially, turning off unali

Re: How to debug checkpoint/savepoint stuck in Flink 1.12.2

2021-03-11 Thread Arvid Heise
Hi ChangZhuo, Did you upgrade to Flink 1.12.2 and change the settings at the time? If so, could you maybe reset the settings to the old values on Flink 1.12.2 and check if the job still gets stuck? Especially, turning off unaligned checkpoints (UC) should clarify if it's a general issue in Flink 1

How to debug checkpoint/savepoint stuck in Flink 1.12.2

2021-03-10 Thread 陳昌倬
Hi, We have updated our Flink applications to 1.12.2, alone with the following modifications to improve its performance: - Use unaligned checkpoint - Change the following fs config - state.backend.fs.memory-threshold: 1048576 - state.backend.fs.write-buffer-size: 4194304 However, now our Fli