yihua commented on code in PR #13007:
URL: https://github.com/apache/hudi/pull/13007#discussion_r2019386700
##########
hudi-io/src/main/java/org/apache/hudi/storage/StorageSchemes.java:
##########
@@ -86,6 +88,11 @@ public enum StorageSchemes {
// Hopsworks File System
HOPSFS("hopsfs", false, true);
+ // list files may bring pressure to storage with centralized meta service
like HDFS.
+ // when we want to get only part of files under a directory rather than all
files, use getStatus may be more friendly than listStatus.
+ // here is a trade-off between rpc times and throughput of storage meta
service
+ private static final Set<String> LIST_STATUS_FRIENDLY_SCHEMES = new
HashSet<>(Arrays.asList(FILE.scheme, S3.scheme, S3A.scheme, GCS.scheme));
Review Comment:
OK, I see your point. If this is not going to be widely used, I agree that
we can use the static variables.
##########
hudi-io/src/main/java/org/apache/hudi/storage/StorageSchemes.java:
##########
@@ -86,6 +88,11 @@ public enum StorageSchemes {
// Hopsworks File System
HOPSFS("hopsfs", false, true);
+ // list files may bring pressure to storage with centralized meta service
like HDFS.
+ // when we want to get only part of files under a directory rather than all
files, use getStatus may be more friendly than listStatus.
+ // here is a trade-off between rpc times and throughput of storage meta
service
+ private static final Set<String> LIST_STATUS_FRIENDLY_SCHEMES = new
HashSet<>(Arrays.asList(FILE.scheme, S3.scheme, S3A.scheme, GCS.scheme));
Review Comment:
OK, I see your point. If this is not going to be widely used, I agree that
we can use the static variable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
