-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/50359/
-----------------------------------------------------------
(Updated Aug. 4, 2016, 4:29 p.m.)
Review request for hive.
Changes
-------
Addressed minor comments.
Removed the code that was duplicating the rename() to S3. Instead, it gets HDFS
scratch directories for the required temporary files.
Bugs: HIVE-14270
https://issues.apache.org/jira/browse/HIVE-14270
Repository: hive-git
Description
-------
This patch will create a temporary directory for Hive intermediate data on HDFS
when S3 tables are used.
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/common/BlobStorageUtils.java
PRE-CREATION
common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
9f5f619359701b948f57d599a5bdc2ecbdff280a
common/src/test/org/apache/hadoop/hive/common/TestBlobStorageUtils.java
PRE-CREATION
ql/src/java/org/apache/hadoop/hive/ql/Context.java
89893eba9fd2316b9a393f06edefa837bb815faf
ql/src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java
5bd78862e1064d7f64a5d764571015a8df1101e8
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java
a01a7bdbfec962b6617e98091cdb1325c5b0e84f
ql/src/test/org/apache/hadoop/hive/ql/exec/TestContext.java PRE-CREATION
Diff: https://reviews.apache.org/r/50359/diff/
Testing
-------
NO PATCH
** NON-PARTITIONED TABLE
- create table dummy (id int);
3.651s
- insert into table s3dummy values (1);
39.231s
- insert overwrite table s3dummy values (1);
42.569s
- insert overwrite directory 's3a://spena-bucket/dirs/s3dummy' select * from
dummy; 30.136s
EXTERNAL TABLE
- create table s3dummy_ext like s3dummy location
's3a://spena-bucket/user/hive/warehouse/s3dummy'; 9.297s
- insert into table s3dummy_ext values (1);
45.855s
WITH PATCH
** NON-PARTITIONED TABLE
- create table s3dummy (id int) location
's3a://spena-bucket/user/hive/warehouse/s3dummy'; 3.945s
- insert into table s3dummy values (1);
15.025s
- insert overwrite table s3dummy values (1);
25.149s
- insert overwrite directory 's3a://spena-bucket/dirs/s3dummy' select * from
dummy; 19.158s
- from dummy insert overwrite table s3dummy select *;
25.469s
- from dummy insert into table s3dummy select *;
14.501s
** EXTERNAL TABLE
- create table s3dummy_ext like s3dummy location
's3a://spena-bucket/user/hive/warehouse/s3dummy'; 4.827s
- insert into table s3dummy_ext values (1);
16.070s
** PARTITIONED TABLE
- create table s3dummypart (id int) partitioned by (part int)
location 's3a://spena-bucket/user/hive/warehouse/s3dummypart';
3.176s
- alter table s3dummypart add partition (part=1);
3.229s
- alter table s3dummypart add partition (part=2);
3.124s
- insert into table s3dummypart partition (part=1) values (1);
14.876s
- insert overwrite table s3dummypart partition (part=1) values (1);
27.594s
- insert overwrite directory 's3a://spena-bucket/dirs/s3dummypart' select *
from dummypart; 22.298s
- from dummypart insert overwrite table s3dummypart partition (part=1) select
id; 29.001s
- from dummypart insert into table s3dummypart partition (part=1) select id;
14.869s
** DYNAMIC PARTITIONS
- insert into table s3dummypart partition (part) select id, 1 from dummypart;
15.185s
- insert into table s3dummypart partition (part) select id, 1 from dummypart;
18.820s
Thanks,
Sergio Pena
