Thanks Jinglun for your work. Basically +1 from me to involve it into the Hadoop codebase. a. After a quick review of JIRA and PR, I think it is solid including document and code style. b. Contributors involved here are diverse who are from different projects and companies, and active enough. c. Community with Jinlun offline many times, and IMO he could be responsible to review and test about this module. Beside that, just suggest following the Hadoop guidelines[1] to develop the new features.
@Steve Loughran <ste...@cloudera.com> @Shilun Fan <slfan1...@foxmail.com> leave some comments including some concerns in JIRA, would you mind giving more suggestions for this discussion? Thanks. Best Regards, - He Xiaoqiao [1] https://hadoop.apache.org/bylaws.html On Sun, Jan 26, 2025 at 3:39 PM jinglun <jinglun...@qq.com.invalid> wrote: > Hello everyone, I'd like to discuss the integration of volcano engine tos > in hadoop. > > > Volcano Engine is a fast growing cloud vendor launched by ByteDance, and > TOS is the object storage service of Volcano Engine. A common way is to > store data into TOS and run Hadoop/Spark/Flink applications to access TOS. > But there is no original support for TOS in hadoop, thus it is not easy for > users to build their Big Data System based on TOS. > > My proposal is to integrate TOS with Hadoop to help users run their > applications on TOS. Users only need to do some simple configuration, then > their applications can read/write TOS without any code change. This work is > similar to AWS S3, AzureBlob, AliyunOSS, Tencnet COS and HuaweiCloud Object > Storage in Hadoop. > > > More details could be found at > https://issues.apache.org/jira/browse/HADOOP-19236. > > > 1. What is the progress of the work now? > The work is currently finished at branch HADOOP_19236. It is developed by > the EMR team of Volcano Engine and served many users from both cloud and > IDC for more than 2 years. > > > 2. How is the long-term maintenance and testing guaranteed? > The contributors are opensource friendly, including ZhengHu(PMC > of HBase and Iceberg), Jinglun(Committer of Hadoop), SunXin(Committer > of HBase), XianyinXin(Contributor of Spark), Rascal Wu(Contributor of > Flink), FangBo(Contributor of Hive) and Yuanzhihuan. We will all be > involved in the long-term maintenance of this work. As time goes by, > more people from the EMR team and the hadoop-tos users may join this work. > So I'm confident at the long-term maintenance and testing. > > > 3. Why should hadoop-tos interaged to hadoop codebase? Shall we use an > independent project? > Integration is for a better user experience. First, users don't need to go > to another repo to find the tos support. Second, users don't need to worry > about the versions mapping between hadoop and hadoop-tos. Finally, a > connector provided by hadoop community is more reliable and > trustworthy. > > > > > > > > > If you have any question, concern or any thing else that is unclear, > please let me know. Sincerely looking forward to your reply, thanks > very much.
