Yes,that’s the point , FilesCreated = CreateFileOps + FsDirMkdirOp
All I can say is --thanks
Darling Andrew D.Lin
下面是被转发的邮件:
主题: 回复: FsStateBackend,hdfs rpc api too much,FileCreated and FileDeleted is for what?
日期: 2019年7月23日 GMT+8 下午4:05:42
Hi Andrew
FilesCreated = CreateFileOps + FsDirMkdirOp Please refer to [1] and [2] to know the meaning of this metrics.
Best Yun Tang
Hi Yun Tang
Your suggestion is very very important to us. According to your suggestion, We have suggested that users increase the interval time (1 to 5 minutes) and set state.backend.fs.memory-threshold=10k. But we only have one hdfs cluster, we try to reduce Hdfs api call, I don't know if there is any possibility of re-optimization,
Thank you very much for your patience and help.
Darling Andrew D.Lin
下面是被转发的邮件:
主题: 回复: FsStateBackend,hdfs rpc api too much,FileCreated and FileDeleted is for what?
日期: 2019年7月23日 GMT+8 上午9:48:05
Hi Andrew
These API calls are for checkpoint file created/deleted, and there is an ongoing issue[1] which want to reduce the number.
Hi
We use ‘FsStateBackend' as our state beckend !
The following figure shows the frequency of the hdfs API call.
I don’t understand FilesCreated and FileDeleted is for what? All of these are necessary?
Is it possible to reduce some unnecessary?
|