Hi All ! I’m trying to create a meta-info file, that contains link to file, created by Flink BucketingSink. At first I was trying to implement my own org.apache.flink.streaming.connectors.fs.Writer, that creates a meta-file on close method call. But I understood, that it’s not completely right, because when writer is closed, file, into which data were written, is in in-progress state and in final state it will change it’s name. So create any meta-info on writer closing, that links to the in-progress file, will lead my system to inconsistent state.
I looked through the sources of BucketingSink, and have not found an elegant way to perform any kind of subscription on moving file with data into final state. Maybe someone already had the same issue and found elegant way how it could be solved ? Also maybe someone know how this issue could be solved using other Flink tools/ components, because I'm not so long using Flink and maybe don’t know some of it's features. Thx.
