Re: Flink Zookeeper HA: FileNotFoundException blob - Jobmanager not starting up

2019-07-23 Thread Till Rohrmann
Hi Richard, it looks as if the zNode of a completed job has not been properly removed. Without the logs of the respective JobMaster, it is hard to debug any further. However, I suspect that this is an instance of FLINK-11665. I am currently working on a fix for it. [1] https://issues.apache.org/j

Re: Flink Zookeeper HA: FileNotFoundException blob - Jobmanager not starting up

2019-07-23 Thread Fabian Hueske
Good to know that you were able to fix the issue! I definitely agree that it would be good to know why this situation occurred. Am Di., 23. Juli 2019 um 14:38 Uhr schrieb Richard Deurwaarder < rich...@xeli.eu>: > Hi Fabian, > > I followed the advice of another flink user who mailed me directly,

Re: Flink Zookeeper HA: FileNotFoundException blob - Jobmanager not starting up

2019-07-23 Thread Richard Deurwaarder
Hi Fabian, I followed the advice of another flink user who mailed me directly, he has the same problem and told me to use something like: rmr zgrep /flink/hunch/jobgraphs/1dccee15d84e1d2cededf89758ac2482 which allowed us to start the job again. It might be nice to investigate what went wrong as i

Re: Flink Zookeeper HA: FileNotFoundException blob - Jobmanager not starting up

2019-07-23 Thread Fabian Hueske
Hi Richard, I hope you could resolve the problem in the meantime. Nonetheless, maybe Till (in CC) has an idea what could have gone wrong. Best, Fabian Am Mi., 17. Juli 2019 um 19:50 Uhr schrieb Richard Deurwaarder < rich...@xeli.eu>: > Hello, > > I've got a problem with our flink cluster where

Flink Zookeeper HA: FileNotFoundException blob - Jobmanager not starting up

2019-07-17 Thread Richard Deurwaarder
Hello, I've got a problem with our flink cluster where the jobmanager is not starting up anymore, because it tries to download non existant (blob) file from the zookeeper storage dir. We're running flink 1.8.0 on a kubernetes cluster and use the google storage connector [1] to store checkpoints,