I have a process function defined with these interfaces
public class MyProcessFunction<IN> extends ProcessFunction<IN, OUT>
implements CheckpointedFunction, ProcessingTimeCallback {...}
In snapshotState() method, I want to close files and emit the metadata
about the closed files to downstream operator. it doesn't seem possible
with *snapshotState(FunctionSnapshotContext context*) interface.
I can keep metadata in snapshot and restore them during recovery. but if
there is no input record coming for a long time, * processElement(T value,
Context ctx, Collector<DataFile> out)* won't be called. Then I can't
forward the restored data to downstream operator with guaranteed latency.
I can add a timer. but it doesn't seem that *onProcessingTime(long
timestamp)* allows me to forward output to downstream operator either.
Thanks,
Steven