Re: Flink 1.14 stuck on INITIALIZING state after job restarting

2021-11-11 Thread Xiangyu Su
Thanks Jake! I use one pod/taskmanager on one instance, on the same taskmanager/instance some slots are able to switch from INITIALIZING to RUNNING like normal, but some slots on the same instance are not. so due to that the connection of the new task manager should be ok. available slots are cor

Re: Flink 1.14 stuck on INITIALIZING state after job restarting

2021-11-11 Thread Jake
Checkout new task manager instances connection status, if connect to job manager is normal, check available slot, check create connection at function initializes. > On Nov 11, 2021, at 18:01, Xiangyu Su wrote: > > Thank you Jake! > Enable debug level logging have to ask system engineer

Re: Flink 1.14 stuck on INITIALIZING state after job restarting

2021-11-11 Thread Xiangyu Su
Thank you Jake! Enable debug level logging have to ask system engineer first ;) do you know how to resolve this issue? Best On Thu, 11 Nov 2021 at 10:57, Jake wrote: > > Set log root level is DEBUG and check Job manager logs, you will get it. > > > > On Nov 11, 2021, at 17:02, Xiangyu Su

Re: Flink 1.14 stuck on INITIALIZING state after job restarting

2021-11-11 Thread Jake
Set log root level is DEBUG and check Job manager logs, you will get it. > On Nov 11, 2021, at 17:02, Xiangyu Su wrote: > > Hello Everyone, > > We are facing an issue on Flink 1.14.0. > Every time if the job gets restarted, some tasks/slots get stuck in > INITIALIZING state, and will neve

Flink 1.14 stuck on INITIALIZING state after job restarting

2021-11-11 Thread Xiangyu Su
Hello Everyone, We are facing an issue on Flink 1.14.0. Every time if the job gets restarted, some tasks/slots get stuck in INITIALIZING state, and will never switch to RUNNING. Any idea/suggestion about a solution to this issue? btw, our flink cluster runs on EKS, and using AWS spot instance fo