morningman commented on code in PR #33937:
URL: https://github.com/apache/doris/pull/33937#discussion_r1579401538
##########
fe/fe-core/src/main/java/org/apache/doris/common/util/LocationPath.java:
##########
@@ -296,6 +296,9 @@ private FileSystemType getFileSystemType() {
fsType = FileSystemType.S3;
break;
case COSN:
+ // COSN use s3 client on FE side, because it need to complete
multi-part uploading files on FE side.
+ fsType = FileSystemType.S3;
Review Comment:
Will this affect origin behavior?
##########
fe/fe-core/src/main/java/org/apache/doris/fs/remote/ObjFileSystem.java:
##########
@@ -43,11 +44,20 @@ public ObjFileSystem(String name,
StorageBackend.StorageType type, ObjStorage<?>
this.objStorage = objStorage;
}
+ public ObjStorage<?> getObjStorage() {
+ return objStorage;
+ }
+
@Override
public Status exists(String remotePath) {
return objStorage.headObject(remotePath);
}
+ @Override
+ public Status directoryExists(String dir) {
+ return listFiles(dir, false, new ArrayList<>());
Review Comment:
Why not using `headObject`?
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/hive/HMSTransaction.java:
##########
@@ -1194,6 +1285,11 @@ public void prepareInsertExistPartition(PartitionAndMore
partitionAndMore) {
writePath,
targetPath,
partitionAndMore.getFileNames());
+ } else {
+ if
(!partitionAndMore.hivePartitionUpdate.s3_mpu_pending_uploads.isEmpty()) {
+ s3Commit(fileSystemExecutor, asyncFileSystemTaskFutures,
fileSystemTaskCancelled,
Review Comment:
This `fileSystemExecutor` is created by `Executors.newFixedThreadPool(16);`
So that each HMSTransaction will create 16 threads, so I suggest:
1. Use a global executor and shared by all txns.
2. Use method in `ThreadPoolManager.java` to create executor
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
