[
https://issues.apache.org/jira/browse/HADOOP-19236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17902091#comment-17902091
]
ASF GitHub Bot commented on HADOOP-19236:
-----------------------------------------
wojiaodoubao opened a new pull request, #7194:
URL: https://github.com/apache/hadoop/pull/7194
<!--
Thanks for sending a pull request!
1. If this is your first time, please read our contributor guidelines:
https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute
2. Make sure your PR title starts with JIRA issue id, e.g.,
'HADOOP-17799. Your PR title ...'.
-->
### Description of PR
Volcano Engine is a fast growing cloud vendor launched by ByteDance, and TOS
is the object storage service of Volcano Engine. A common way is to store data
into TOS and run Hadoop/Spark/Flink applications to access TOS. But there is no
original support for TOS in hadoop, thus it is not easy for users to build
their Big Data System based on TOS.
This work aims to integrate TOS with Hadoop to help users run their
applications on TOS. Users only need to do some simple configuration, then
their applications can read/write TOS without any code change. This work is
similar to AWS S3, AzureBlob, AliyunOSS, Tencnet COS and HuaweiCloud Object
Storage in Hadoop.
Please see the issue for more details.
https://issues.apache.org/jira/browse/HADOOP-19236
### How was this patch tested?
Unit tests need to connect to tos service. Setting the 6 environment
variables below to run unit tests.
```bash
export TOS_ACCESS_KEY_ID={YOUR_ACCESS_KEY}
export TOS_SECRET_ACCESS_KEY={YOUR_SECRET_ACCESS_KEY}
export TOS_ENDPOINT={TOS_SERVICE_ENDPOINT}
export FILE_STORAGE_ROOT=/tmp/local_dev/
export TOS_BUCKET={YOUR_BUCKET_NAME}
export TOS_UNIT_TEST_ENABLED=true
```
Then cd to hadoop project root directory, and run the test command below.
```bash
mvn -Dtest=org.apache.hadoop.fs.tosfs.** test -pl
org.apache.hadoop:hadoop-tos-core
```
### For code changes:
- [ ] Does the title or this PR starts with the corresponding JIRA issue id
(e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the
endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`,
`NOTICE-binary` files?
> Integration of Volcano Engine TOS in Hadoop.
> --------------------------------------------
>
> Key: HADOOP-19236
> URL: https://issues.apache.org/jira/browse/HADOOP-19236
> Project: Hadoop Common
> Issue Type: New Feature
> Components: fs, tools
> Affects Versions: 3.4.0
> Reporter: Jinglun
> Assignee: Zheng Hu
> Priority: Major
> Attachments: Integration of Volcano Engine TOS in Hadoop.pdf
>
>
> Volcano Engine is a fast growing cloud vendor launched by ByteDance, and TOS
> is the object storage service of Volcano Engine. A common way is to store
> data into TOS and run Hadoop/Spark/Flink applications to access TOS. But
> there is no original support for TOS in hadoop, thus it is not easy for users
> to build their Big Data System based on TOS.
>
> This work aims to integrate TOS with Hadoop to help users run their
> applications on TOS. Users only need to do some simple configuration, then
> their applications can read/write TOS without any code change. This work is
> similar to AWS S3, AzureBlob, AliyunOSS, Tencnet COS and HuaweiCloud Object
> Storage in Hadoop.
>
> Please see the attached document "Integration of Volcano Engine TOS in
> Hadoop" for more details.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
