Using the broker to read hdfs and bos shields the implementation of different file systems, there are the following problems:
1. The process of reading and writing interacts with the broker too much through the thrift interface, which increases the probability of errors 2. The performance of reading via broker is worse than direct reading due to transit 3. The method of uploading files through the broker is to first upload to the temporary file and then rename it into the final file. In many cloud storages, renaming is not supported. Some of the methods of copy+delete are real copying rather than simple modification. Data is a waste of time, especially when there are more files to upload Therefore, we intend to support direct access through hdfs and s3 protocols, and support most cloud storage through s3 protocols, such as baidu bos, aliyun oss, etc.
