[ https://issues.apache.org/jira/browse/HIVE-13989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16090716#comment-16090716 ]
Hive QA commented on HIVE-13989: -------------------------------- Here are the results of testing the latest attachment: https://issues.apache.org/jira/secure/attachment/12877663/HIVE-13989-branch-2.2.patch {color:red}ERROR:{color} -1 due to no test(s) being added or modified. {color:red}ERROR:{color} -1 due to 129 failed/errored test(s), 10591 tests executed *Failed tests:* {noformat} TestDerbyConnector - did not produce a TEST-*.xml file (likely timed out) (batchId=245) TestJdbcDriver2 - did not produce a TEST-*.xml file (likely timed out) (batchId=226) TestMiniLlapLocalCliDriver - did not produce a TEST-*.xml file (likely timed out) (batchId=168) [acid_globallimit.q,alter_merge_2_orc.q] org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_globallimit] (batchId=27) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2] (batchId=44) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[avro_nullable_union] (batchId=51) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_union_view] (batchId=19) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[druid_basic2] (batchId=10) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[druid_intervals] (batchId=21) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[druid_timeseries] (batchId=54) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[druid_topn] (batchId=3) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[dynpart_sort_optimization_acid] (batchId=41) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[explain_logical] (batchId=59) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join32] (batchId=17) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[list_bucket_dml_12] (batchId=46) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[list_bucket_dml_13] (batchId=23) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[order_null] (batchId=19) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[position_alias_test_1] (batchId=39) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_union_view] (batchId=27) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[show_create_table_db_table] (batchId=60) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats_list_bucket] (batchId=62) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[subquery_notin] (batchId=64) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[subquery_notin_having] (batchId=45) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[subquery_views] (batchId=13) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[table_access_keys_stats] (batchId=65) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[union_fast_stats] (batchId=47) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[union_view] (batchId=14) org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[view_cbo] (batchId=62) org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_join_with_different_encryption_keys] (batchId=173) org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl] (batchId=170) org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver[hbase_viewjoins] (batchId=88) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[dynamic_partition_pruning] (batchId=161) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[join_acid_non_acid] (batchId=166) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_llap_counters1] (batchId=160) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_llap_counters] (batchId=167) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[schema_evol_orc_acid_mapwork_part] (batchId=162) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[schema_evol_orc_acid_mapwork_table] (batchId=158) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[schema_evol_orc_acidvec_mapwork_part] (batchId=166) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[schema_evol_orc_acidvec_mapwork_table] (batchId=163) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[schema_evol_orc_nonvec_fetchwork_part] (batchId=163) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[schema_evol_orc_nonvec_fetchwork_table] (batchId=162) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[schema_evol_orc_nonvec_mapwork_part] (batchId=159) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[schema_evol_orc_nonvec_mapwork_part_all_primitive] (batchId=165) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[schema_evol_orc_nonvec_mapwork_table] (batchId=165) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[schema_evol_orc_vec_mapwork_part] (batchId=163) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[schema_evol_orc_vec_mapwork_part_all_primitive] (batchId=162) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[schema_evol_orc_vec_mapwork_table] (batchId=158) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[schema_evol_text_nonvec_mapwork_part] (batchId=166) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[schema_evol_text_nonvec_mapwork_part_all_primitive] (batchId=159) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[schema_evol_text_nonvec_mapwork_table] (batchId=161) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[schema_evol_text_vec_mapwork_part] (batchId=165) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[schema_evol_text_vec_mapwork_part_all_primitive] (batchId=158) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[schema_evol_text_vec_mapwork_table] (batchId=160) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[schema_evol_text_vecrow_mapwork_part] (batchId=158) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[schema_evol_text_vecrow_mapwork_part_all_primitive] (batchId=159) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[schema_evol_text_vecrow_mapwork_table] (batchId=166) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[subquery_multi] (batchId=159) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[subquery_scalar] (batchId=162) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[tez_dynpart_hashjoin_3] (batchId=162) org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[vector_complex_join] (batchId=163) org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[constprog_partitioner] (batchId=176) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5] (batchId=94) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainuser_1] (batchId=102) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[limit_pushdown] (batchId=112) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[metadataonly1] (batchId=111) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[schema_evol_orc_acid_mapwork_part] (batchId=103) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[schema_evol_orc_acid_mapwork_table] (batchId=92) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[schema_evol_orc_acidvec_mapwork_part] (batchId=116) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[schema_evol_orc_acidvec_mapwork_table] (batchId=105) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[schema_evol_orc_nonvec_fetchwork_part] (batchId=106) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[schema_evol_orc_nonvec_fetchwork_table] (batchId=102) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[schema_evol_orc_nonvec_mapwork_part] (batchId=93) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[schema_evol_orc_nonvec_mapwork_part_all_primitive] (batchId=113) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[schema_evol_orc_nonvec_mapwork_table] (batchId=112) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[schema_evol_orc_vec_mapwork_part] (batchId=105) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[schema_evol_orc_vec_mapwork_part_all_primitive] (batchId=104) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[schema_evol_orc_vec_mapwork_table] (batchId=92) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[schema_evol_text_nonvec_mapwork_part] (batchId=115) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[schema_evol_text_nonvec_mapwork_part_all_primitive] (batchId=96) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[schema_evol_text_nonvec_mapwork_table] (batchId=101) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[schema_evol_text_vec_mapwork_part] (batchId=111) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[schema_evol_text_vec_mapwork_part_all_primitive] (batchId=92) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[schema_evol_text_vec_mapwork_table] (batchId=98) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[schema_evol_text_vecrow_mapwork_part] (batchId=92) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[schema_evol_text_vecrow_mapwork_part_all_primitive] (batchId=95) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[schema_evol_text_vecrow_mapwork_table] (batchId=114) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[unionDistinct_1] (batchId=106) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[vector_join30] (batchId=101) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[vector_outer_join0] (batchId=109) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[vector_outer_join1] (batchId=105) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[vector_outer_join2] (batchId=100) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[vector_outer_join3] (batchId=101) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[vector_outer_join4] (batchId=116) org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[vector_outer_join5] (batchId=116) org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[avro_non_nullable_union] (batchId=85) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query23] (batchId=236) org.apache.hadoop.hive.cli.TestPerfCliDriver.testCliDriver[query83] (batchId=236) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[cbo_subq_not_in] (batchId=140) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[count] (batchId=130) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[date_udf] (batchId=133) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join0] (batchId=144) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[join_view] (batchId=154) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[leftsemijoin] (batchId=138) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[metadata_only_queries] (batchId=133) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[outer_join_ppr] (batchId=128) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ptf] (batchId=126) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_exists] (batchId=136) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_in] (batchId=146) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_multiinsert] (batchId=155) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[table_access_keys_stats] (batchId=149) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[temp_table_gb1] (batchId=133) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[udf_percentile] (batchId=146) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[union25] (batchId=148) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[union_null] (batchId=154) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_between_in] (batchId=143) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_cast_constant] (batchId=123) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vector_mapjoin_reduce] (batchId=153) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_0] (batchId=154) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_3] (batchId=152) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_div0] (batchId=148) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_short_regress] (batchId=139) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorized_ptf] (batchId=146) org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[windowing] (batchId=141) org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 (batchId=219) org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgressParallel (batchId=223) org.apache.hive.beeline.TestBeelineArgParsing.testAddLocalJarWithoutAddDriverClazz[0] (batchId=182) org.apache.hive.beeline.TestBeelineArgParsing.testAddLocalJar[0] (batchId=182) org.apache.hive.beeline.TestBeelineArgParsing.testAddLocalJar[1] (batchId=182) org.apache.hive.jdbc.TestJdbcWithMiniHS2.testAddJarConstructorUnCaching (batchId=229) {noformat} Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/6068/testReport Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/6068/console Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-6068/ Messages: {noformat} Executing org.apache.hive.ptest.execution.TestCheckPhase Executing org.apache.hive.ptest.execution.PrepPhase Executing org.apache.hive.ptest.execution.ExecutionPhase Executing org.apache.hive.ptest.execution.ReportingPhase Tests exited with: TestsFailedException: 129 tests failed {noformat} This message is automatically generated. ATTACHMENT ID: 12877663 - PreCommit-HIVE-Build > Extended ACLs are not handled according to specification > -------------------------------------------------------- > > Key: HIVE-13989 > URL: https://issues.apache.org/jira/browse/HIVE-13989 > Project: Hive > Issue Type: Bug > Components: HCatalog > Affects Versions: 1.2.1, 2.0.0 > Reporter: Chris Drome > Assignee: Chris Drome > Attachments: HIVE-13989.1-branch-1.patch, HIVE-13989.1.patch, > HIVE-13989-branch-1.patch, HIVE-13989-branch-2.2.patch, > HIVE-13989-branch-2.2.patch > > > Hive takes two approaches to working with extended ACLs depending on whether > data is being produced via a Hive query or HCatalog APIs. A Hive query will > run an FsShell command to recursively set the extended ACLs for a directory > sub-tree. HCatalog APIs will attempt to build up the directory sub-tree > programmatically and runs some code to set the ACLs to match the parent > directory. > Some incorrect assumptions were made when implementing the extended ACLs > support. Refer to https://issues.apache.org/jira/browse/HDFS-4685 for the > design documents of extended ACLs in HDFS. These documents model the > implementation after the POSIX implementation on Linux, which can be found at > http://www.vanemery.com/Linux/ACL/POSIX_ACL_on_Linux.html. > The code for setting extended ACLs via HCatalog APIs is found in > HdfsUtils.java: > {code} > if (aclEnabled) { > aclStatus = sourceStatus.getAclStatus(); > if (aclStatus != null) { > LOG.trace(aclStatus.toString()); > aclEntries = aclStatus.getEntries(); > removeBaseAclEntries(aclEntries); > //the ACL api's also expect the tradition user/group/other permission > in the form of ACL > aclEntries.add(newAclEntry(AclEntryScope.ACCESS, AclEntryType.USER, > sourcePerm.getUserAction())); > aclEntries.add(newAclEntry(AclEntryScope.ACCESS, AclEntryType.GROUP, > sourcePerm.getGroupAction())); > aclEntries.add(newAclEntry(AclEntryScope.ACCESS, AclEntryType.OTHER, > sourcePerm.getOtherAction())); > } > } > {code} > We found that DEFAULT extended ACL rules were not being inherited properly by > the directory sub-tree, so the above code is incomplete because it > effectively drops the DEFAULT rules. The second problem is with the call to > {{sourcePerm.getGroupAction()}}, which is incorrect in the case of extended > ACLs. When extended ACLs are used the GROUP permission is replaced with the > extended ACL mask. So the above code will apply the wrong permissions to the > GROUP. Instead the correct GROUP permissions now need to be pulled from the > AclEntry as returned by {{getAclStatus().getEntries()}}. See the > implementation of the new method {{getDefaultAclEntries}} for details. > Similar issues exist with the HCatalog API. None of the API accounts for > setting extended ACLs on the directory sub-tree. The changes to the HCatalog > API allow the extended ACLs to be passed into the required methods similar to > how basic permissions are passed in. When building the directory sub-tree the > extended ACLs of the table directory are inherited by all sub-directories, > including the DEFAULT rules. > Replicating the problem: > Create a table to write data into (I will use acl_test as the destination and > words_text as the source) and set the ACLs as follows: > {noformat} > $ hdfs dfs -setfacl -m > default:user::rwx,default:group::r-x,default:mask::rwx,default:user:hdfs:rwx,group::r-x,user:hdfs:rwx > /user/cdrome/hive/acl_test > $ hdfs dfs -ls -d /user/cdrome/hive/acl_test > drwxrwx---+ - cdrome hdfs 0 2016-07-13 20:36 > /user/cdrome/hive/acl_test > $ hdfs dfs -getfacl -R /user/cdrome/hive/acl_test > # file: /user/cdrome/hive/acl_test > # owner: cdrome > # group: hdfs > user::rwx > user:hdfs:rwx > group::r-x > mask::rwx > other::--- > default:user::rwx > default:user:hdfs:rwx > default:group::r-x > default:mask::rwx > default:other::--- > {noformat} > Note that the basic GROUP permission is set to {{rwx}} after setting the > ACLs. The ACLs explicitly set the DEFAULT rules and a rule specifically for > the {{hdfs}} user. > Run the following query to populate the table: > {noformat} > insert into acl_test partition (dt='a', ds='b') select a, b from words_text > where dt = 'c'; > {noformat} > Note that words_text only has a single partition key. > Now examine the ACLs for the resulting directories: > {noformat} > $ hdfs dfs -getfacl -R /user/cdrome/hive/acl_test > # file: /user/cdrome/hive/acl_test > # owner: cdrome > # group: hdfs > user::rwx > user:hdfs:rwx > group::r-x > mask::rwx > other::--- > default:user::rwx > default:user:hdfs:rwx > default:group::r-x > default:mask::rwx > default:other::--- > # file: /user/cdrome/hive/acl_test/dt=a > # owner: cdrome > # group: hdfs > user::rwx > user:hdfs:rwx > group::rwx > mask::rwx > other::--- > default:user::rwx > default:user:hdfs:rwx > default:group::rwx > default:mask::rwx > default:other::--- > # file: /user/cdrome/hive/acl_test/dt=a/ds=b > # owner: cdrome > # group: hdfs > user::rwx > user:hdfs:rwx > group::rwx > mask::rwx > other::--- > default:user::rwx > default:user:hdfs:rwx > default:group::rwx > default:mask::rwx > default:other::--- > # file: /user/cdrome/hive/acl_test/dt=a/ds=b/000000_0.deflate > # owner: cdrome > # group: hdfs > user::rwx > user:hdfs:rwx > group::rwx > mask::rwx > other::--- > {noformat} > Note that the GROUP permission is now erroneously set to {{rwx}} because of > the code mentioned above; it is set to the same value as the ACL mask. > The code changes for the HCatalog APIs is synonymous to the > {{applyGroupAndPerms}} method which ensures that all new directories are > created with the same permissions as the table. This patch will ensure that > changes to intermediate directories will not be propagated, instead the table > ACLs will be applied to all new directories created. > I would also like to call out that the older versions of HDFS which support > ACLs had a number issues in addition to those mentioned here which appear to > have been addressed in later versions of Hadoop. This patch was originally > written to work with a version of Hadoop-2.6, we are now using Hadoop-2.7 > which appears to have fixed some of them. However, I think that this patch is > still required for correct behavior of ACLs with Hive/HCatalog. -- This message was sent by Atlassian JIRA (v6.4.14#64029)