Hi, The BucketingSink truncates the file if the Hadoop FileSystem supports this operation (Hadoop 2.7 and above) [1]. What version of Hadoop are you using?
Best, Gary [1] https://github.com/apache/flink/blob/bcd028d75b0e5c5c691e24640a2196b2fdaf85e0/flink-connectors/flink-connector-filesystem/src/main/java/org/apache/flink/streaming/connectors/fs/bucketing/BucketingSink.java#L301 On Mon, May 14, 2018 at 1:37 PM, 张馨予 <342689...@qq.com> wrote: > Hi > > > I'm trying to copy data from kafka to HDFS . The data in HDFS is used to > do other computations by others in map/reduce. > If some tasks failed, the ".valid-length" file is created for the low > version hadoop. The problem is other people must know how to deal with the > ".valid-length" file, otherwise, the data may be not exactly-once. > Hence, why not rewrite a new file when restoring instead of writing a > ".valid-length" file. In this way, others who use the data in HDFS don't > need to know how to deal with the ".valid-length" file. > > > Thanks! > > > Zhang Xinyu
