Hi Josh, You are right, FLINK-2646 is related to the problem of non-finialized files. If we could distinguish the cases why close() is called, we could do a proper clean-up if the job terminated because all data was processed.
Right now, the source and sink interfaces of the DataStream API are not really designed for finite / bounded data. In order to improve the support for bounded and unbounded data, we have some plans to design unified interfaces that can handle both cases well. This effort should also solve cases like the one that you described. Best, Fabian 2018-04-04 21:51 GMT+02:00 joshlemer <[email protected]>: > Actually sorry, I have found that this is most likely a manifestation of > https://issues.apache.org/jira/browse/FLINK-2646 as discussed elsewhere on > the mailing list. That is, in the second example "fromCollection" the > entire > stream ends before a checkpoint is made. Let's hope this is fixed some day > :-) > > > > -- > Sent from: http://apache-flink-user-mailing-list-archive.2336050. > n4.nabble.com/ >
