Can FE and BE access external storage without using broker? Ling Miao
寒江雪 <yangz...@gmail.com> 于2021年1月13日周三 下午7:37写道： > Using the broker to read hdfs and bos shields the implementation of > different file systems, there are the following problems: > > 1. The process of reading and writing interacts with the broker too much > through the thrift interface, which increases the probability of errors > 2. The performance of reading via broker is worse than direct reading > due to transit > 3. The method of uploading files through the broker is to first upload > to the temporary file and then rename it into the final file. In many > cloud > storages, renaming is not supported. Some of the methods of copy+delete > are > real copying rather than simple modification. Data is a waste of time, > especially when there are more files to upload > > Therefore, we intend to support direct access through hdfs and s3 > protocols, and support most cloud storage through s3 protocols, such as > baidu bos, aliyun oss, etc. >
