Yes,that’s the point , FilesCreated = CreateFileOps + FsDirMkdirOp

 All I can say is  --thanks

Darling 
Andrew D.Lin



下面是被转发的邮件:

发件人: Yun Tang <myas...@live.com>
主题: 回复: FsStateBackend,hdfs rpc api too much,FileCreated and FileDeleted is for what?
日期: 2019年7月23日 GMT+8 下午4:05:42

Hi Andrew

FilesCreated = CreateFileOps + FsDirMkdirOp Please refer to [1] and [2] to know the meaning of this metrics.



Best
Yun Tang



From: 陈Darling <chendonglin...@gmail.com>
Sent: Tuesday, July 23, 2019 11:32
To: qcx978132...@gmail.com <qcx978132...@gmail.com>
Cc: user@flink.apache.org <user@flink.apache.org>; myas...@live.com <myas...@live.com>
Subject: Fwd: FsStateBackend,hdfs rpc api too much,FileCreated and FileDeleted is for what?
 
Hi Yun Tang

Your suggestion is very very important to us.
 According to your suggestion, We have suggested that users increase the interval time (1 to 5 minutes) and set state.backend.fs.memory-threshold=10k.
 
But we only have one hdfs cluster, we try to reduce Hdfs api call, I don't know if there is any possibility of re-optimization,

Thank you very much for your patience and help. 


Darling 
Andrew D.Lin



下面是被转发的邮件:

发件人: Congxian Qiu <qcx978132...@gmail.com>
主题: 回复: FsStateBackend,hdfs rpc api too much,FileCreated and FileDeleted is for what?
日期: 2019年7月23日 GMT+8 上午9:48:05
收件人: 陈Darling <chendonglin...@gmail.com>
抄送: user <user@flink.apache.org>

Hi Andrew

These API calls are for checkpoint file created/deleted, and there is an ongoing issue[1] which want to reduce the number.

陈Darling <chendonglin...@gmail.com> 于2019年7月22日周一 下午11:22写道:

Hi

We use  ‘FsStateBackend' as  our state beckend !


The following figure shows the frequency of the hdfs API call.

I don’t understand FilesCreated and FileDeleted is for what?   All of these are necessary? 

 Is it possible to reduce some unnecessary?











Darling 
Andrew D.Lin

Reply via email to