Hi Andrew

FilesCreated = CreateFileOps + FsDirMkdirOp Please refer to [1] and [2] to know 
the meaning of this metrics.


[1] 
https://github.com/apache/hadoop/blob/377f95bbe8d2d171b5d7b0bfa7559e67ca4aae46/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirMkdirOp.java#L178
[2] 
https://github.com/apache/hadoop/blob/377f95bbe8d2d171b5d7b0bfa7559e67ca4aae46/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeRpcServer.java#L799

Best
Yun Tang


________________________________
From: 陈Darling <[email protected]>
Sent: Tuesday, July 23, 2019 11:32
To: [email protected] <[email protected]>
Cc: [email protected] <[email protected]>; [email protected] 
<[email protected]>
Subject: Fwd: FsStateBackend,hdfs rpc api too much,FileCreated and FileDeleted 
is for what?

Hi Yun Tang

Your suggestion is very very important to us.
 According to your suggestion, We have suggested that users increase the 
interval time (1 to 5 minutes) and set state.backend.fs.memory-threshold=10k.

But we only have one hdfs cluster, we try to reduce Hdfs api call, I don't know 
if there is any possibility of re-optimization,

Thank you very much for your patience and help.


Darling
Andrew D.Lin



下面是被转发的邮件:

发件人: Congxian Qiu <[email protected]<mailto:[email protected]>>
主题: 回复: FsStateBackend,hdfs rpc api too much,FileCreated and FileDeleted is for 
what?
日期: 2019年7月23日 GMT+8 上午9:48:05
收件人: 陈Darling <[email protected]<mailto:[email protected]>>
抄送: user <[email protected]<mailto:[email protected]>>

Hi Andrew

These API calls are for checkpoint file created/deleted, and there is an 
ongoing issue[1] which want to reduce the number.
[1] https://issues.apache.org/jira/browse/FLINK-11696

Best,
Congxian


陈Darling <[email protected]<mailto:[email protected]>> 
于2019年7月22日周一 下午11:22写道:

Hi

We use  ‘FsStateBackend' as  our state beckend !


The following figure shows the frequency of the hdfs API call.

I don’t understand FilesCreated and FileDeleted is for what?   All of these are 
necessary?

 Is it possible to reduce some unnecessary?





[cid:[email protected]]






Darling
Andrew D.Lin

Reply via email to