Hi Yun Tang

Your suggestion is very very important to us.
 According to your suggestion, We have suggested that users increase the interval time (1 to 5 minutes) and set state.backend.fs.memory-threshold=10k.
 
But we only have one hdfs cluster, we try to reduce Hdfs api call, I don't know if there is any possibility of re-optimization,

Thank you very much for your patience and help. 


Darling 
Andrew D.Lin



下面是被转发的邮件:

发件人: Congxian Qiu <[email protected]>
主题: 回复: FsStateBackend,hdfs rpc api too much,FileCreated and FileDeleted is for what?
日期: 2019年7月23日 GMT+8 上午9:48:05
收件人: 陈Darling <[email protected]>
抄送: user <[email protected]>

Hi Andrew

These API calls are for checkpoint file created/deleted, and there is an ongoing issue[1] which want to reduce the number.

陈Darling <[email protected]> 于2019年7月22日周一 下午11:22写道:

Hi

We use  ‘FsStateBackend' as  our state beckend !


The following figure shows the frequency of the hdfs API call.

I don’t understand FilesCreated and FileDeleted is for what?   All of these are necessary? 

 Is it possible to reduce some unnecessary?











Darling 
Andrew D.Lin

Reply via email to