Hi, Siew Wai Yow When the job is running, the states are stored in the local RocksDB, Flink will copy all the needed states to checkpointPath when doing a Checkpoint. If there have any failures, the job will be restored from the last previously *Successfully* checkpoint, and assign the restored states to all the current TM (These TMs do not need to be the same as before) .
Siew Wai Yow <wai_...@hotmail.com> 于2019年1月12日周六 上午11:24写道： > Thanks. But this is something I know. I would like to know will the other > TM take over the crashed TM's state to ensure data completion(say the state > BYKEY, different key state will be stored in different TM) OR the crashed > TM need to be recovered to continue? > > For example, 5 records, > rec1:KEYA > rec2:KEYB > rec3:KEYA > rec4:KEYC > rec5:KEYB > > TM1 stored state for rec1:KEYA, rec3:KEYA > TM2 stored state for rec2:KEYB, rec5:KEYB <--crashed, what happen to those > state stored inTM2 > TM3 stored state for rec4:KEYC > > In case TM2 crashed, rec2 and rec5 will be assigned to other TM? Or those > record only recover when TM2 being recover? > > Thanks. > > > ------------------------------ > *From:* Jamie Grier <jgr...@lyft.com> > *Sent:* Saturday, January 12, 2019 2:26 AM > *To:* Siew Wai Yow > *Cc:* user@flink.apache.org > *Subject:* Re: What happen to state in Flink Task Manager when crash? > > Flink is designed such that local state is backed up to a highly available > system such as HDFS or S3. When a TaskManager fails state is recovered > from there. > > I suggest reading this: > https://ci.apache.org/projects/flink/flink-docs-release-1.7/ops/state/state_backends.html > > > On Fri, Jan 11, 2019 at 7:33 AM Siew Wai Yow <wai_...@hotmail.com> wrote: > > Hello, > > May i know what happen to state stored in Flink Task Manager when this > Task manager crash. Say the state storage is rocksdb, would those data > transfer to other running Task Manager so that complete state data is ready > for data processing? > > Regards, > Yow > > -- Best, Congxian
