Hi Fu, Thank you for the remind. I think it would work in my case as long as it's an atomic operation.
Dian Fu <dian0511...@gmail.com> 于2019年9月6日周五 下午2:22写道： > Hi Jingsong, > > Thanks for bring up this discussion. You can try to look at the > GlobalAggregateManager to see if it can meet your requirements. It can be > got via StreamingRuntimeContext#getGlobalAggregateManager(). > > Regards, > Dian > > > 在 2019年9月6日，下午1:39，shimin yang <ysmcl...@gmail.com> 写道： > > > > Hi Jingsong, > > > > Big fan of this idea. We faced the same problem and resolved by adding a > > distributed lock. It would be nice to have this feature in JobMaster, > which > > can replace the lock. > > > > Best, > > Shimin > > > > JingsongLee <lzljs3620...@aliyun.com.invalid> 于2019年9月6日周五 下午12:20写道： > > > >> Hi devs: > >> > >> I try to implement streaming file sink for table[1] like > StreamingFileSink. > >> If the underlying is a HiveFormat, or a format that updates visibility > >> through a metaStore, I have to update the metaStore in the > >> notifyCheckpointComplete, but this operation occurs on the task side, > >> which will lead to distributed access to the metaStore, which will > >> lead to bottleneck. > >> > >> So I'm curious if we can support notifyOnMaster for > >> notifyCheckpointComplete like FinalizeOnMaster. > >> > >> What do you think? > >> > >> [1] > >> > https://docs.google.com/document/d/15R3vZ1R_pAHcvJkRx_CWleXgl08WL3k_ZpnWSdzP7GY/edit?usp=sharing > >> > >> Best, > >> Jingsong Lee > >
