I think before we have such interface, maybe we can make task-0 to do the global finalize work.
Best, Jark > 在 2019年9月6日，13:39，shimin yang <ysmcl...@gmail.com> 写道： > > Hi Jingsong, > > Big fan of this idea. We faced the same problem and resolved by adding a > distributed lock. It would be nice to have this feature in JobMaster, which > can replace the lock. > > Best, > Shimin > > JingsongLee <lzljs3620...@aliyun.com.invalid> 于2019年9月6日周五 下午12:20写道： > >> Hi devs: >> >> I try to implement streaming file sink for table[1] like StreamingFileSink. >> If the underlying is a HiveFormat, or a format that updates visibility >> through a metaStore, I have to update the metaStore in the >> notifyCheckpointComplete, but this operation occurs on the task side, >> which will lead to distributed access to the metaStore, which will >> lead to bottleneck. >> >> So I'm curious if we can support notifyOnMaster for >> notifyCheckpointComplete like FinalizeOnMaster. >> >> What do you think? >> >> [1] >> https://docs.google.com/document/d/15R3vZ1R_pAHcvJkRx_CWleXgl08WL3k_ZpnWSdzP7GY/edit?usp=sharing >> >> Best, >> Jingsong Lee
