Hi devs: I try to implement streaming file sink for table[1] like StreamingFileSink. If the underlying is a HiveFormat, or a format that updates visibility through a metaStore, I have to update the metaStore in the notifyCheckpointComplete, but this operation occurs on the task side, which will lead to distributed access to the metaStore, which will lead to bottleneck.
So I'm curious if we can support notifyOnMaster for notifyCheckpointComplete like FinalizeOnMaster. What do you think? [1] https://docs.google.com/document/d/15R3vZ1R_pAHcvJkRx_CWleXgl08WL3k_ZpnWSdzP7GY/edit?usp=sharing Best, Jingsong Lee
