zhouaoe opened a new pull request, #14305:
URL: https://github.com/apache/doris/pull/14305
# Proposed changes
Issue Number: close #13665
## Problem summary
Describe your changes.
1 Upgrade fs_broker module hadoop2.8.3->hadoop2.9.1
2 Broker support oss://
3 Version of jar file hadoop-huaweicloud used by broker is set to 2.8.3
## Checklist(Required)
1. Does it affect the original behavior:
- [ ] Yes
- [x] No
- [ ] I don't know
2. Has unit tests been added:
- [ ] Yes
- [x] No
- [ ] No Need
3. Has document been added or modified:
- [ ] Yes
- [x] No
- [ ] No Need
4. Does it need to update dependencies:
- [x] Yes
- [ ] No
5. Are there any changes that cannot be rolled back:
- [ ] Yes (If Yes, please explain WHY)
- [x] No
## Further comments
**1. Test case :doris broker升级hadoop2.9.1后访问obs**
验证hadoop2.9.1兼容华为obs的访问
LOAD LABEL demo.load_oss_label_hw (
DATA INFILE(
"obs://weinan-test1.obs.cn-east-3.myhuaweicloud.com/data2.csv"
) INTO TABLE example_tbl_hw COLUMNS TERMINATED BY ","
) WITH BROKER "broker_za" (
"fs.obs.access.key" = "xxxxx",
"fs.obs.secret.key" = "xxxxxxx",
"fs.obs.endpoint" = "https://obs.cn-east-3.myhuaweicloud.com";
)
测试结果符合预期，兼容obs的访问
执行结果：
后台记录：
结果检查：
**2. Test cast :Doris broker支持OSS协议测试**
broker中hadoop升级2.9.1后支持oss://的导入导出
1 Load with Broker
用例：先导入6条数据，再通过OSS导入7条数据（在原始数据基础上增加了一条）
原始数据：
10000,2017-10-01,北京,20,0,2017-10-01 06:00:00,20,10,10
10000,2017-10-01,北京,20,0,2017-10-01 07:00:00,15,2,2
10001,2017-10-01,北京,30,1,2017-10-01 17:05:45,2,22,22
10002,2017-10-02,上海,20,1,2017-10-02 12:59:12,200,5,5
10003,2017-10-02,广州,32,0,2017-10-02 11:20:00,30,11,11
10004,2017-10-01,深圳,35,0,2017-10-01 10:00:15,100,3,3
10004,2017-10-03,深圳,35,0,2017-10-03 10:20:22,11,6,6
OSS上的数据data.csv
10000,2017-10-01,北京,20,0,2017-10-01 06:00:00,20,10,10
10000,2017-10-01,北京,20,0,2017-10-01 07:00:00,15,2,2
10001,2017-10-01,北京,30,1,2017-10-01 17:05:45,2,22,22
10002,2017-10-02,上海,20,1,2017-10-02 12:59:12,200,5,5
10003,2017-10-02,广州,32,0,2017-10-02 11:20:00,30,11,11
10004,2017-10-01,深圳,35,0,2017-10-01 10:00:15,100,3,3
10004,2017-10-03,深圳,35,0,2017-10-03 10:20:22,11,6,6
10005,2017-10-03,深圳,35,0,2017-10-03 10:20:22,11,6,6
导入语句
LOAD LABEL demo.load_oss_label_1
(
DATA INFILE("oss://otsosstest/doris/data.csv")
INTO TABLE example_tbl
COLUMNS TERMINATED BY ","
)
WITH BROKER "broker_za"
(
"fs.oss.endpoint" = "https://xxxxx";,
"fs.oss.accessKeyId" = "xxxxx",
"fs.oss.accessKeySecret"="xxxxx"
)
执行结果：符合预期
1.前端执行：
2.Load任务
3.数据检查
2 Export with Borker
用例：将刚才的7条数据导入到OSS上
EXPORT TABLE demo.example_tbl
TO "oss://otsosstest/doris/export_broker/01/"
PROPERTIES
(
"label" = "export_from_doris_18",
"column_separator"=",",
"timeout" = "3600"
)
WITH BROKER "broker_za"
(
"fs.oss.endpoint" = "https://oss-cn-hangzhou.aliyuncs.com";,
"fs.oss.accessKeyId" = "xxxxx",
"fs.oss.accessKeySecret"="xxxxx"
);
测试结果符合预期
执行结果
后台执行结果
oss上导出的文件：
oss上导出的文件内容：
3 Outfile export with Broker
select * from demo.example_tbl
into outfile "oss://streamoss-1/doris/newbroker/01"
FORMAT AS CSV
PROPERTIES
(
"broker.name" = "broker_za",
"broker.fs.oss.endpoint" = "https://oss-cn-hangzhou.aliyuncs.com";,
"broker.fs.oss.accessKeyId" = "xxxxxx",
"broker.fs.oss.accessKeySecret"="xxxxxxx",
"column_separator" = ",",
"line_delimiter" = "\n",
检查结果：符合预期
执行结果
oss上导出的文件：
oss上导出的文件内容
If this is a relatively large or complex change, kick off the discussion at
[d...@doris.apache.org](mailto:d...@doris.apache.org) by explaining why you
chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org
For additional commands, e-mail: commits-h...@doris.apache.org
