[
https://issues.apache.org/jira/browse/HIVE-17852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16551709#comment-16551709
]
Hive QA commented on HIVE-17852:
--------------------------------
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12932500/HIVE-17852.21.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/12764/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/12764/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-12764/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-07-21 14:07:33.321
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-12764/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-07-21 14:07:33.324
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at e569ef0 HIVE-4367: enhance TRUNCATE syntax to drop data of
external table (Jason Dere, reviewed by Teddy Choi)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at e569ef0 HIVE-4367: enhance TRUNCATE syntax to drop data of
external table (Jason Dere, reviewed by Teddy Choi)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-07-21 14:07:34.392
+ rm -rf ../yetus_PreCommit-HIVE-Build-12764
+ mkdir ../yetus_PreCommit-HIVE-Build-12764
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-12764
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-12764/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error:
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/client/builder/StorageDescriptorBuilder.java:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/model/MStorageDescriptor.java:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/utils/HiveStrictManagedUtils.java:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java:
does not exist in index
error: standalone-metastore/metastore-common/src/main/resources/package.jdo:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/derby/hive-schema-4.0.0.derby.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mssql/hive-schema-4.0.0.mssql.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/mysql/hive-schema-4.0.0.mysql.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/oracle/hive-schema-4.0.0.oracle.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/main/sql/postgres/hive-schema-4.0.0.postgres.sql:
does not exist in index
error:
standalone-metastore/metastore-common/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java:
does not exist in index
error:
standalone-metastore/metastore-common/src/test/java/org/apache/hadoop/hive/metastore/cache/TestCachedStore.java:
does not exist in index
error:
standalone-metastore/metastore-common/src/test/java/org/apache/hadoop/hive/metastore/client/TestAddPartitions.java:
does not exist in index
error:
standalone-metastore/metastore-common/src/test/java/org/apache/hadoop/hive/metastore/client/TestAddPartitionsFromPartSpec.java:
does not exist in index
error:
standalone-metastore/metastore-common/src/test/java/org/apache/hadoop/hive/metastore/client/TestTablesCreateDropAlterTruncate.java:
does not exist in index
error:
standalone-metastore/metastore-common/src/test/java/org/apache/hadoop/hive/metastore/tools/TestSchemaToolForMetastore.java:
does not exist in index
error:
hive-blobstore/src/test/queries/clientpositive/insert_empty_into_blobstore.q:
does not exist in index
error:
hive-blobstore/src/test/results/clientpositive/insert_empty_into_blobstore.q.out:
does not exist in index
error:
hive-unit/src/test/java/org/apache/hadoop/hive/ql/history/TestHiveHistory.java:
does not exist in index
error:
hive-unit/src/test/java/org/apache/hive/service/server/TestInformationSchemaWithPrivilege.java:
does not exist in index
error: patch failed: pom.xml:819
Falling back to three-way merge...
Applied patch to 'pom.xml' cleanly.
error: src/java/org/apache/hadoop/hive/ql/ErrorMsg.java: does not exist in index
error: src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java: does not exist in
index
error: src/java/org/apache/hadoop/hive/ql/exec/FileSinkOperator.java: does not
exist in index
error: src/java/org/apache/hadoop/hive/ql/exec/MoveTask.java: does not exist in
index
error: src/java/org/apache/hadoop/hive/ql/io/merge/MergeFileWork.java: does not
exist in index
error: src/java/org/apache/hadoop/hive/ql/metadata/Hive.java: does not exist in
index
error: src/java/org/apache/hadoop/hive/ql/metadata/Partition.java: does not
exist in index
error: src/java/org/apache/hadoop/hive/ql/metadata/Table.java: does not exist
in index
error:
src/java/org/apache/hadoop/hive/ql/metadata/formatting/MetaDataFormatUtils.java:
does not exist in index
error: src/java/org/apache/hadoop/hive/ql/optimizer/GenMapRedUtils.java: does
not exist in index
error: src/java/org/apache/hadoop/hive/ql/parse/BaseSemanticAnalyzer.java: does
not exist in index
error: src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java: does
not exist in index
error: src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g: does not exist in
index
error: src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java: does
not exist in index
error: src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java: does not
exist in index
error:
src/java/org/apache/hadoop/hive/ql/parse/UpdateDeleteSemanticAnalyzer.java:
does not exist in index
error: src/java/org/apache/hadoop/hive/ql/plan/AlterTableDesc.java: does not
exist in index
error: src/java/org/apache/hadoop/hive/ql/plan/CreateTableDesc.java: does not
exist in index
error: src/java/org/apache/hadoop/hive/ql/plan/ImportTableDesc.java: does not
exist in index
error: src/java/org/apache/hadoop/hive/ql/plan/ListBucketingCtx.java: does not
exist in index
error: src/java/org/apache/hadoop/hive/ql/plan/SkewedColumnPositionPair.java:
does not exist in index
error: src/java/org/apache/hadoop/hive/ql/txn/compactor/CompactorMR.java: does
not exist in index
error: src/test/org/apache/hadoop/hive/ql/exec/TestExecDriver.java: does not
exist in index
error: src/test/org/apache/hadoop/hive/ql/metadata/TestHive.java: does not
exist in index
error: src/test/queries/clientnegative/load_stored_as_dirs.q: does not exist in
index
error: src/test/queries/clientnegative/truncate_column_list_bucketing.q: does
not exist in index
error: src/test/queries/clientpositive/create_alter_list_bucketing_table1.q:
does not exist in index
error: src/test/queries/clientpositive/infer_bucket_sort_list_bucket.q: does
not exist in index
error: src/test/queries/clientpositive/lb_fs_stats.q: does not exist in index
error: src/test/queries/clientpositive/list_bucket_dml_1.q: does not exist in
index
error: src/test/queries/clientpositive/list_bucket_dml_10.q: does not exist in
index
error: src/test/queries/clientpositive/list_bucket_dml_11.q: does not exist in
index
error: src/test/queries/clientpositive/list_bucket_dml_12.q: does not exist in
index
error: src/test/queries/clientpositive/list_bucket_dml_13.q: does not exist in
index
error: src/test/queries/clientpositive/list_bucket_dml_14.q: does not exist in
index
error: src/test/queries/clientpositive/list_bucket_dml_2.q: does not exist in
index
error: src/test/queries/clientpositive/list_bucket_dml_3.q: does not exist in
index
error: src/test/queries/clientpositive/list_bucket_dml_4.q: does not exist in
index
error: src/test/queries/clientpositive/list_bucket_dml_5.q: does not exist in
index
error: src/test/queries/clientpositive/list_bucket_dml_6.q: does not exist in
index
error: src/test/queries/clientpositive/list_bucket_dml_7.q: does not exist in
index
error: src/test/queries/clientpositive/list_bucket_dml_8.q: does not exist in
index
error: src/test/queries/clientpositive/list_bucket_dml_9.q: does not exist in
index
error: src/test/queries/clientpositive/list_bucket_query_multiskew_1.q: does
not exist in index
error: src/test/queries/clientpositive/list_bucket_query_multiskew_2.q: does
not exist in index
error: src/test/queries/clientpositive/list_bucket_query_multiskew_3.q: does
not exist in index
error: src/test/queries/clientpositive/mm_all.q: does not exist in index
error: src/test/queries/clientpositive/resourceplan.q: does not exist in index
error: src/test/queries/clientpositive/show_create_table_db_table.q: does not
exist in index
error: src/test/queries/clientpositive/stats_list_bucket.q: does not exist in
index
error: src/test/queries/clientpositive/strict_managed_tables_sysdb.q: does not
exist in index
error: src/test/queries/clientpositive/truncate_column_list_bucket.q: does not
exist in index
error: src/test/results/clientnegative/load_stored_as_dirs.q.out: does not
exist in index
error: src/test/results/clientnegative/truncate_column_list_bucketing.q.out:
does not exist in index
error:
src/test/results/clientpositive/create_alter_list_bucketing_table1.q.out: does
not exist in index
error: src/test/results/clientpositive/infer_bucket_sort_list_bucket.q.out:
does not exist in index
error: src/test/results/clientpositive/lb_fs_stats.q.out: does not exist in
index
error: src/test/results/clientpositive/list_bucket_dml_1.q.out: does not exist
in index
error: src/test/results/clientpositive/list_bucket_dml_11.q.out: does not exist
in index
error: src/test/results/clientpositive/list_bucket_dml_12.q.out: does not exist
in index
error: src/test/results/clientpositive/list_bucket_dml_13.q.out: does not exist
in index
error: src/test/results/clientpositive/list_bucket_dml_14.q.out: does not exist
in index
error: src/test/results/clientpositive/list_bucket_dml_2.q.out: does not exist
in index
error: src/test/results/clientpositive/list_bucket_dml_3.q.out: does not exist
in index
error: src/test/results/clientpositive/list_bucket_dml_4.q.out: does not exist
in index
error: src/test/results/clientpositive/list_bucket_dml_5.q.out: does not exist
in index
error: src/test/results/clientpositive/list_bucket_dml_6.q.out: does not exist
in index
error: src/test/results/clientpositive/list_bucket_dml_7.q.out: does not exist
in index
error: src/test/results/clientpositive/list_bucket_dml_8.q.out: does not exist
in index
error: src/test/results/clientpositive/list_bucket_dml_9.q.out: does not exist
in index
error: src/test/results/clientpositive/list_bucket_query_multiskew_1.q.out:
does not exist in index
error: src/test/results/clientpositive/list_bucket_query_multiskew_2.q.out:
does not exist in index
error: src/test/results/clientpositive/list_bucket_query_multiskew_3.q.out:
does not exist in index
error: src/test/results/clientpositive/llap/list_bucket_dml_10.q.out: does not
exist in index
error: src/test/results/clientpositive/llap/mm_all.q.out: does not exist in
index
error: src/test/results/clientpositive/llap/resourceplan.q.out: does not exist
in index
error: src/test/results/clientpositive/llap/strict_managed_tables_sysdb.q.out:
does not exist in index
error: src/test/results/clientpositive/mm_all.q.out: does not exist in index
error: src/test/results/clientpositive/show_create_table_db_table.q.out: does
not exist in index
error: src/test/results/clientpositive/spark/list_bucket_dml_10.q.out: does not
exist in index
error: src/test/results/clientpositive/spark/list_bucket_dml_2.q.out: does not
exist in index
error: src/test/results/clientpositive/stats_list_bucket.q.out: does not exist
in index
error: src/test/results/clientpositive/truncate_column_list_bucket.q.out: does
not exist in index
error:
metastore-common/src/main/java/org/apache/hadoop/hive/metastore/MetaStoreDirectSql.java:
does not exist in index
error:
metastore-common/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java:
does not exist in index
error:
metastore-common/src/main/java/org/apache/hadoop/hive/metastore/client/builder/StorageDescriptorBuilder.java:
does not exist in index
error:
metastore-common/src/main/java/org/apache/hadoop/hive/metastore/model/MStorageDescriptor.java:
does not exist in index
error:
metastore-common/src/main/java/org/apache/hadoop/hive/metastore/utils/HiveStrictManagedUtils.java:
does not exist in index
error:
metastore-common/src/main/java/org/apache/hadoop/hive/metastore/utils/MetaStoreUtils.java:
does not exist in index
error: metastore-common/src/main/resources/package.jdo: does not exist in index
error: metastore-common/src/main/sql/derby/hive-schema-4.0.0.derby.sql: does
not exist in index
error: metastore-common/src/main/sql/mssql/hive-schema-4.0.0.mssql.sql: does
not exist in index
error: metastore-common/src/main/sql/mysql/hive-schema-4.0.0.mysql.sql: does
not exist in index
error: metastore-common/src/main/sql/oracle/hive-schema-4.0.0.oracle.sql: does
not exist in index
error: metastore-common/src/main/sql/postgres/hive-schema-4.0.0.postgres.sql:
does not exist in index
error:
metastore-common/src/test/java/org/apache/hadoop/hive/metastore/TestHiveMetaStore.java:
does not exist in index
error:
metastore-common/src/test/java/org/apache/hadoop/hive/metastore/cache/TestCachedStore.java:
does not exist in index
error:
metastore-common/src/test/java/org/apache/hadoop/hive/metastore/client/TestAddPartitions.java:
does not exist in index
error:
metastore-common/src/test/java/org/apache/hadoop/hive/metastore/client/TestAddPartitionsFromPartSpec.java:
does not exist in index
error:
metastore-common/src/test/java/org/apache/hadoop/hive/metastore/client/TestTablesCreateDropAlterTruncate.java:
does not exist in index
error:
metastore-common/src/test/java/org/apache/hadoop/hive/metastore/tools/TestSchemaToolForMetastore.java:
does not exist in index
error: patch failed: pom.xml:33
Falling back to three-way merge...
Applied patch to 'pom.xml' cleanly.
fatal: git diff header lacks filename information when removing 2 leading
pathname components (line 1459)
The patch does not appear to apply with p0, p1, or p2
+ result=1
+ '[' 1 -ne 0 ']'
+ rm -rf yetus_PreCommit-HIVE-Build-12764
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12932500 - PreCommit-HIVE-Build
> remove support for list bucketing "stored as directories" in 3.0
> ----------------------------------------------------------------
>
> Key: HIVE-17852
> URL: https://issues.apache.org/jira/browse/HIVE-17852
> Project: Hive
> Issue Type: Bug
> Reporter: Sergey Shelukhin
> Assignee: Laszlo Bodor
> Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-17852.01.patch, HIVE-17852.02.patch,
> HIVE-17852.03.patch, HIVE-17852.04.patch, HIVE-17852.05.patch,
> HIVE-17852.06.patch, HIVE-17852.07.patch, HIVE-17852.08.patch,
> HIVE-17852.09.patch, HIVE-17852.10.patch, HIVE-17852.11.patch,
> HIVE-17852.12.patch, HIVE-17852.13.patch, HIVE-17852.14.patch,
> HIVE-17852.15.patch, HIVE-17852.16.patch, HIVE-17852.17.patch,
> HIVE-17852.18.patch, HIVE-17852.19.patch, HIVE-17852.20.patch,
> HIVE-17852.21.patch
>
>
> From the email thread:
> 1) LB, when stored as directories, adds a lot of low-level complexity to Hive
> tables that has to be accounted for in many places in the code where the
> files are written or modified - from FSOP to ACID/replication/export.
> 2) While working on some FSOP code I noticed that some of that logic is
> broken - e.g. the duplicate file removal from tasks, a pretty fundamental
> correctness feature in Hive, may be broken. LB also doesn’t appear to be
> compatible with e.g. regular bucketing.
> 3) The feature hasn’t seen development activity in a while; it also doesn’t
> appear to be used a lot.
> Keeping with the theme of cleaning up “legacy” code for 3.0, I was proposing
> we remove it.
> (2) also suggested that, if needed, it might be easier to implement similar
> functionality by adding some flexibility to partitions (which LB directories
> look like anyway); that would also keep the logic on a higher level of
> abstraction (split generation, partition pruning) as opposed to many
> low-level places like FSOP, etc.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
