Fwd: FsStateBackend，hdfs rpc api too much，FileCreated and FileDeleted is for what？

[陈Darling]

Yes，that’s the point , FilesCreated = CreateFileOps + FsDirMkdirOp  All I can say is  --thanks Darling  Andrew D.Lin 下面是被转发的邮件： 发件人: Yun Tang <myas...@live.com> 主题: 回复： FsStateBackend，hdfs rpc api too much，FileCreated and FileDeleted is for what？ 日期: 2019年7月23日 GMT+8 下午4:05:42 收件人: 陈Darling <chendonglin...@gmail.com>, "qcx978132...@gmail.com" <qcx978132...@gmail.com> 抄送: "user@flink.apache.org" <user@flink.apache.org> Hi Andrew FilesCreated = CreateFileOps + FsDirMkdirOp Please refer to [1] and [2] to know the meaning of this metrics. [1] https://github.com/apache/hadoop/blob/377f95bbe8d2d171b5d7b0bfa7559e67ca4aae46/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirMkdirOp.java#L178 [2] https://github.com/apache/hadoop/blob/377f95bbe8d2d171b5d7b0bfa7559e67ca4aae46/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java#L799 Best Yun Tang From: 陈Darling <chendonglin...@gmail.com> Sent: Tuesday, July 23, 2019 11:32 To: qcx978132...@gmail.com <qcx978132...@gmail.com> Cc: user@flink.apache.org <user@flink.apache.org>; myas...@live.com <myas...@live.com> Subject: Fwd: FsStateBackend，hdfs rpc api too much，FileCreated and FileDeleted is for what？   Hi Yun Tang Your suggestion is very very important to us.  According to your suggestion, We have suggested that users increase the interval time (1 to 5 minutes) and set state.backend.fs.memory-threshold=10k.   But we only have one hdfs cluster, we try to reduce Hdfs api call, I don't know if there is any possibility of re-optimization， Thank you very much for your patience and help.  Darling  Andrew D.Lin 下面是被转发的邮件： 发件人: Congxian Qiu <qcx978132...@gmail.com> 主题: 回复： FsStateBackend，hdfs rpc api too much，FileCreated and FileDeleted is for what？ 日期: 2019年7月23日 GMT+8 上午9:48:05 收件人: 陈Darling <chendonglin...@gmail.com> 抄送: user <user@flink.apache.org> Hi Andrew These API calls are for checkpoint file created/deleted, and there is an ongoing issue[1] which want to reduce the number. [1] https://issues.apache.org/jira/browse/FLINK-11696 Best, Congxian 陈Darling <chendonglin...@gmail.com> 于2019年7月22日周一 下午11:22写道： Hi We use  ‘FsStateBackend' as  our state beckend ! The following figure shows the frequency of the hdfs API call. I don’t understand FilesCreated and FileDeleted is for what?   All of these are necessary?   Is it possible to reduce some unnecessary？

Darling  Andrew D.Lin