+ dev Blob store is used for jars, serialized job, and task information and logs. You can find some information at https://cwiki.apache.org/confluence/display/FLINK/FLIP-19%3A+Improved+BLOB+storage+architecture
I guess in your setup, Flink was able to pick up local files. HA setup presumes that Flink can survive the loss of that JM host and its local files. I'm not sure about K8s native setup - probably VoidBlobStore is enough if there is a persistent volume. But in the general case, FileSystemBlobStore should be used to store files on some DFS. Regards, Roman On Sat, Aug 29, 2020 at 6:42 PM Alexey Trenikhun <yen...@msn.com> wrote: > Did test with streaming job and FileSystemHaService using VoidBlobStore > (no HA Blob), looks like job was able to recover from both JM restart and > TM restart. Any idea in what use cases HA Blob is needed? > > Thanks, > Alexey > ------------------------------ > *From:* Alexey Trenikhun <yen...@msn.com> > *Sent:* Friday, August 28, 2020 11:31 AM > *To:* Khachatryan Roman <khachatryan.ro...@gmail.com> > *Cc:* Flink User Mail List <user@flink.apache.org> > *Subject:* Re: FileSystemHaServices and BlobStore > > Motivation is to have k8s HA setup without extra component - Zookeeper, > see [1] > > Purpose of BlobStore is vague to me, what kind of BLOBs are stored? Looks > like if we start job from savepoint, then persistence of BlobStore is > not necessary, but is it needed if we recover from checkpoint? > > Thanks, > Alexey > > [1]. https://issues.apache.org/jira/browse/FLINK-17598 > > > ------------------------------ > *From:* Khachatryan Roman <khachatryan.ro...@gmail.com> > *Sent:* Friday, August 28, 2020 9:24 AM > *To:* Alexey Trenikhun <yen...@msn.com> > *Cc:* Flink User Mail List <user@flink.apache.org> > *Subject:* Re: FileSystemHaServices and BlobStore > > Hello Alexey, > > I think you need FileSystemBlobStore as you are implementing HA Services, > and BLOBs should be highly available too. > However, I'm a bit concerned about the direction in general: it > essentially means re-implementing ZK functionality on top of FS. > What are the motivation and the use case? > > Regards, > Roman > > > On Fri, Aug 28, 2020 at 5:15 PM Alexey Trenikhun <yen...@msn.com> wrote: > > Hello, > I'm thinking about implementing FileSystemHaServices - single leader, but > persistent RunningJobRegistry, CheckpointIDCounter, > CompletedCheckpointStore and JobGraphStore. I'm not sure do you need > FileSystemBlobStore or VoidBlobStore is enough. Can't figure out, should > BlobStore survive JobManager crash. I see that ZookeeperHaServices use > FileSystemBlobStore, > but not clear is to due to having multiple JobManagers (leader + follower) > or necessity to preserve BLOBs on restart. > > Thanks, > Alexey > >
