[jira] [Updated] (HIVE-13815) Improve logic to infer false predicates
[
https://issues.apache.org/jira/browse/HIVE-13815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-13815:
---
Attachment: HIVE-13815.01.patch
> Improve logic to infer false predicates
> ---
>
> Key: HIVE-13815
> URL: https://issues.apache.org/jira/browse/HIVE-13815
> Project: Hive
> Issue Type: Sub-task
> Components: CBO
>Affects Versions: 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13815.01.patch, HIVE-13815.patch
>
>
> Follow-up/extension of the work done in HIVE-13068.
> Ex.
> ql/src/test/results/clientpositive/annotate_stats_filter.q.out
> {{predicate: ((year = 2001) and (state = 'OH') and (state = 'FL')) (type:
> boolean)}} -> {{false}}
> ql/src/test/results/clientpositive/cbo_rp_join1.q.out
> {{predicate: ((_col0 = _col1) and (_col1 = 40) and (_col0 = 40)) (type:
> boolean)}} -> {{predicate: ((_col1 = 40) and (_col0 = 40)) (type: boolean)}}
> ql/src/test/results/clientpositive/constprog_semijoin.q.out
> {{predicate: (((id = 100) = true) and (id <> 100)) (type: boolean)}} ->
> {{false}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Work started] (HIVE-13815) Improve logic to infer false predicates
[
https://issues.apache.org/jira/browse/HIVE-13815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-13815 started by Jesus Camacho Rodriguez.
--
> Improve logic to infer false predicates
> ---
>
> Key: HIVE-13815
> URL: https://issues.apache.org/jira/browse/HIVE-13815
> Project: Hive
> Issue Type: Sub-task
> Components: CBO
>Affects Versions: 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13815.01.patch, HIVE-13815.patch
>
>
> Follow-up/extension of the work done in HIVE-13068.
> Ex.
> ql/src/test/results/clientpositive/annotate_stats_filter.q.out
> {{predicate: ((year = 2001) and (state = 'OH') and (state = 'FL')) (type:
> boolean)}} -> {{false}}
> ql/src/test/results/clientpositive/cbo_rp_join1.q.out
> {{predicate: ((_col0 = _col1) and (_col1 = 40) and (_col0 = 40)) (type:
> boolean)}} -> {{predicate: ((_col1 = 40) and (_col0 = 40)) (type: boolean)}}
> ql/src/test/results/clientpositive/constprog_semijoin.q.out
> {{predicate: (((id = 100) = true) and (id <> 100)) (type: boolean)}} ->
> {{false}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13815) Improve logic to infer false predicates
[
https://issues.apache.org/jira/browse/HIVE-13815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-13815:
---
Status: Patch Available (was: In Progress)
> Improve logic to infer false predicates
> ---
>
> Key: HIVE-13815
> URL: https://issues.apache.org/jira/browse/HIVE-13815
> Project: Hive
> Issue Type: Sub-task
> Components: CBO
>Affects Versions: 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13815.01.patch, HIVE-13815.patch
>
>
> Follow-up/extension of the work done in HIVE-13068.
> Ex.
> ql/src/test/results/clientpositive/annotate_stats_filter.q.out
> {{predicate: ((year = 2001) and (state = 'OH') and (state = 'FL')) (type:
> boolean)}} -> {{false}}
> ql/src/test/results/clientpositive/cbo_rp_join1.q.out
> {{predicate: ((_col0 = _col1) and (_col1 = 40) and (_col0 = 40)) (type:
> boolean)}} -> {{predicate: ((_col1 = 40) and (_col0 = 40)) (type: boolean)}}
> ql/src/test/results/clientpositive/constprog_semijoin.q.out
> {{predicate: (((id = 100) = true) and (id <> 100)) (type: boolean)}} ->
> {{false}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13815) Improve logic to infer false predicates
[
https://issues.apache.org/jira/browse/HIVE-13815?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-13815:
---
Status: Open (was: Patch Available)
> Improve logic to infer false predicates
> ---
>
> Key: HIVE-13815
> URL: https://issues.apache.org/jira/browse/HIVE-13815
> Project: Hive
> Issue Type: Sub-task
> Components: CBO
>Affects Versions: 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13815.01.patch, HIVE-13815.patch
>
>
> Follow-up/extension of the work done in HIVE-13068.
> Ex.
> ql/src/test/results/clientpositive/annotate_stats_filter.q.out
> {{predicate: ((year = 2001) and (state = 'OH') and (state = 'FL')) (type:
> boolean)}} -> {{false}}
> ql/src/test/results/clientpositive/cbo_rp_join1.q.out
> {{predicate: ((_col0 = _col1) and (_col1 = 40) and (_col0 = 40)) (type:
> boolean)}} -> {{predicate: ((_col1 = 40) and (_col0 = 40)) (type: boolean)}}
> ql/src/test/results/clientpositive/constprog_semijoin.q.out
> {{predicate: (((id = 100) = true) and (id <> 100)) (type: boolean)}} ->
> {{false}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13815) Improve logic to infer false predicates
[
https://issues.apache.org/jira/browse/HIVE-13815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15393405#comment-15393405
]
Jesus Camacho Rodriguez commented on HIVE-13815:
Triggering QA again as there are quite a few q file changes.
> Improve logic to infer false predicates
> ---
>
> Key: HIVE-13815
> URL: https://issues.apache.org/jira/browse/HIVE-13815
> Project: Hive
> Issue Type: Sub-task
> Components: CBO
>Affects Versions: 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13815.01.patch, HIVE-13815.patch
>
>
> Follow-up/extension of the work done in HIVE-13068.
> Ex.
> ql/src/test/results/clientpositive/annotate_stats_filter.q.out
> {{predicate: ((year = 2001) and (state = 'OH') and (state = 'FL')) (type:
> boolean)}} -> {{false}}
> ql/src/test/results/clientpositive/cbo_rp_join1.q.out
> {{predicate: ((_col0 = _col1) and (_col1 = 40) and (_col0 = 40)) (type:
> boolean)}} -> {{predicate: ((_col1 = 40) and (_col0 = 40)) (type: boolean)}}
> ql/src/test/results/clientpositive/constprog_semijoin.q.out
> {{predicate: (((id = 100) = true) and (id <> 100)) (type: boolean)}} ->
> {{false}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14324) ORC PPD for floats is broken
[
https://issues.apache.org/jira/browse/HIVE-14324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15393419#comment-15393419
]
Hive QA commented on HIVE-14324:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12820002/HIVE-14324.2.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 18 failed/errored test(s), 10321 tests
executed
*Failed tests:*
{noformat}
TestColumn - did not produce a TEST-*.xml file
TestCookieSigner - did not produce a TEST-*.xml file
TestJdbcWithSQLAuthUDFBlacklist - did not produce a TEST-*.xml file
TestLdapAtnProviderWithMiniDS - did not produce a TEST-*.xml file
TestLdapAuthenticationProviderImpl - did not produce a TEST-*.xml file
TestMsgBusConnection - did not produce a TEST-*.xml file
TestPlainSaslHelper - did not produce a TEST-*.xml file
TestPluggableHiveSessionImpl - did not produce a TEST-*.xml file
TestSessionCleanup - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.llap.daemon.impl.TestLlapTokenChecker.testCheckPermissions
org.apache.hadoop.hive.llap.daemon.impl.TestLlapTokenChecker.testGetToken
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testDelayedLocalityNodeCommErrorImmediateAllocation
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.testConnections
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/644/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/644/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-644/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 18 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12820002 - PreCommit-HIVE-MASTER-Build
> ORC PPD for floats is broken
>
>
> Key: HIVE-14324
> URL: https://issues.apache.org/jira/browse/HIVE-14324
> Project: Hive
> Issue Type: Bug
> Components: ORC
>Affects Versions: 1.3.0, 2.0.0, 2.1.0, 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Critical
> Attachments: HIVE-14324.1.patch, HIVE-14324.2.patch
>
>
> ORC stores min/max stats, bloom filters by passing floats as doubles using
> java's widening conversion. So if we write a float value of 0.22 to ORC file,
> the min/max stats and bloom filter will use 0.219988079071 double value.
> But when we do PPD, SARG creates literals by converting float to string and
> then to double which compares 0.22 to 0.219988079071 and fails PPD
> evaluation.
> {code}
> hive> create table orc_float (f float) stored as orc;
> hive> insert into table orc_float values(0.22);
> hive> set hive.optimize.index.filter=true;
> hive> select * from orc_float where f=0.22;
> OK
> hive> set hive.optimize.index.filter=false;
> hive> select * from orc_float where f=0.22;
> OK
> 0.22
> {code}
> This is not a problem for doubles and decimals.
> This issue was introduced in HIVE-8460 but back then there was no strict type
> check when SARGs are created and also PPD evaluation does not convert to
> column type. But now predicate leaf creation in SARG enforces strict type
> check for boxed literals and predicate type and PPD evaluation converts stats
> and constants to column type (predicate).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-12077) MSCK Repair table should fix partitions in batches
[ https://issues.apache.org/jira/browse/HIVE-12077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam updated HIVE-12077: Status: Open (was: Patch Available) > MSCK Repair table should fix partitions in batches > --- > > Key: HIVE-12077 > URL: https://issues.apache.org/jira/browse/HIVE-12077 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Ryan P >Assignee: Chinna Rao Lalam > Attachments: HIVE-12077.1.patch, HIVE-12077.2.patch, > HIVE-12077.3.patch, HIVE-12077.4.patch > > > If a user attempts to run MSCK REPAIR TABLE on a directory with a large > number of untracked partitions HMS will OOME. I suspect this is because it > attempts to do one large bulk load in an effort to save time. Ultimately this > can lead to a collection so large in size that HMS eventually hits an Out of > Memory Exception. > Instead I suggest that Hive include a configurable batch size that HMS can > use to break up the load. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12077) MSCK Repair table should fix partitions in batches
[ https://issues.apache.org/jira/browse/HIVE-12077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam updated HIVE-12077: Attachment: HIVE-12077.5.patch > MSCK Repair table should fix partitions in batches > --- > > Key: HIVE-12077 > URL: https://issues.apache.org/jira/browse/HIVE-12077 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Ryan P >Assignee: Chinna Rao Lalam > Attachments: HIVE-12077.1.patch, HIVE-12077.2.patch, > HIVE-12077.3.patch, HIVE-12077.4.patch, HIVE-12077.5.patch > > > If a user attempts to run MSCK REPAIR TABLE on a directory with a large > number of untracked partitions HMS will OOME. I suspect this is because it > attempts to do one large bulk load in an effort to save time. Ultimately this > can lead to a collection so large in size that HMS eventually hits an Out of > Memory Exception. > Instead I suggest that Hive include a configurable batch size that HMS can > use to break up the load. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12077) MSCK Repair table should fix partitions in batches
[ https://issues.apache.org/jira/browse/HIVE-12077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinna Rao Lalam updated HIVE-12077: Status: Patch Available (was: Open) > MSCK Repair table should fix partitions in batches > --- > > Key: HIVE-12077 > URL: https://issues.apache.org/jira/browse/HIVE-12077 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Ryan P >Assignee: Chinna Rao Lalam > Attachments: HIVE-12077.1.patch, HIVE-12077.2.patch, > HIVE-12077.3.patch, HIVE-12077.4.patch, HIVE-12077.5.patch > > > If a user attempts to run MSCK REPAIR TABLE on a directory with a large > number of untracked partitions HMS will OOME. I suspect this is because it > attempts to do one large bulk load in an effort to save time. Ultimately this > can lead to a collection so large in size that HMS eventually hits an Out of > Memory Exception. > Instead I suggest that Hive include a configurable batch size that HMS can > use to break up the load. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14323) Reduce number of FS permissions and redundant FS operations
[
https://issues.apache.org/jira/browse/HIVE-14323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15393550#comment-15393550
]
Steve Loughran commented on HIVE-14323:
---
if the file isn't there, delete() returns false. This is tested in
{{hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/fs/contract/AbstractContractDeleteTest.java}};
nothing subclasses/disables the relevant tests. Looks like the javadoc is
incomplete.
Fancy submitting a hadoop patch there being slightly more informative, or say
"look at the filesystem specification for the specific semantics", + "Does not
raise an exception if the path is not present"
> Reduce number of FS permissions and redundant FS operations
> ---
>
> Key: HIVE-14323
> URL: https://issues.apache.org/jira/browse/HIVE-14323
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-14323.1.patch
>
>
> Some examples are given below.
> 1. When creating stage directory, FileUtils sets the directory permissions by
> running a set of chgrp and chmod commands. In systems like S3, this would not
> be relevant.
> 2. In some cases, fs.delete() is followed by fs.exists(). In this case, it
> might be redundant to check for exists() (lookup ops are expensive in systems
> like S3).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14251) Union All of different types resolves to incorrect data
[
https://issues.apache.org/jira/browse/HIVE-14251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15393609#comment-15393609
]
Hive QA commented on HIVE-14251:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12820005/HIVE-14251.3.patch
{color:green}SUCCESS:{color} +1 due to 14 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 14 failed/errored test(s), 10305 tests
executed
*Failed tests:*
{noformat}
TestHS2HttpServer - did not produce a TEST-*.xml file
TestHiveSQLException - did not produce a TEST-*.xml file
TestLdapAtnProviderWithMiniDS - did not produce a TEST-*.xml file
TestMiniTezCliDriver-insert_values_non_partitioned.q-update_after_multiple_inserts.q-tez_union_dynamic_partition.q-and-12-more
- did not produce a TEST-*.xml file
TestMsgBusConnection - did not produce a TEST-*.xml file
TestServerOptionsProcessor - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_partition_change_col
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.llap.daemon.impl.TestLlapTokenChecker.testCheckPermissions
org.apache.hadoop.hive.llap.daemon.impl.TestLlapTokenChecker.testGetToken
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.testConnections
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/645/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/645/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-645/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 14 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12820005 - PreCommit-HIVE-MASTER-Build
> Union All of different types resolves to incorrect data
> ---
>
> Key: HIVE-14251
> URL: https://issues.apache.org/jira/browse/HIVE-14251
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 2.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-14251.1.patch, HIVE-14251.2.patch,
> HIVE-14251.3.patch
>
>
> create table src(c1 date, c2 int, c3 double);
> insert into src values ('2016-01-01',5,1.25);
> select * from
> (select c1 from src union all
> select c2 from src union all
> select c3 from src) t;
> It will return NULL for the c1 values. Seems the common data type is resolved
> to the last c3 which is double.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14335) TaskDisplay's return value is not getting deserialized properly
[ https://issues.apache.org/jira/browse/HIVE-14335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15393710#comment-15393710 ] Rajat Khandelwal commented on HIVE-14335: - [~szehon] Can you please take a look. The change is just renaming of a variable. One test case has been modified. The test case modifications only succeed along with the code changes. > TaskDisplay's return value is not getting deserialized properly > --- > > Key: HIVE-14335 > URL: https://issues.apache.org/jira/browse/HIVE-14335 > Project: Hive > Issue Type: Bug >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14315) Implement StatsProvidingRecordReader for ParquetRecordReaderWrapper
[
https://issues.apache.org/jira/browse/HIVE-14315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15393744#comment-15393744
]
Hive QA commented on HIVE-14315:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12820021/HIVE-14315.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 11 failed/errored test(s), 10348 tests
executed
*Failed tests:*
{noformat}
TestCookieSigner - did not produce a TEST-*.xml file
TestHiveSQLException - did not produce a TEST-*.xml file
TestMsgBusConnection - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.llap.daemon.impl.TestLlapTokenChecker.testCheckPermissions
org.apache.hadoop.hive.llap.daemon.impl.TestLlapTokenChecker.testGetToken
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.testConnections
org.apache.hive.spark.client.TestSparkClient.testJobSubmission
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/646/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/646/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-646/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 11 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12820021 - PreCommit-HIVE-MASTER-Build
> Implement StatsProvidingRecordReader for ParquetRecordReaderWrapper
> ---
>
> Key: HIVE-14315
> URL: https://issues.apache.org/jira/browse/HIVE-14315
> Project: Hive
> Issue Type: New Feature
> Components: Statistics
>Affects Versions: 2.1.0
>Reporter: Chao Sun
>Assignee: Chao Sun
> Attachments: HIVE-14315.0.patch, HIVE-14315.1.patch
>
>
> Currently only ORC supports {{analyze table ... compute statistics noscan}}
> (via HIVE-6578) where stats such as # of rows, raw datasize, etc., can be
> obtained via the footer. The similar functionality should be implemented on
> Parquet since it also has the info in the metadata.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Work started] (HIVE-14335) TaskDisplay's return value is not getting deserialized properly
[ https://issues.apache.org/jira/browse/HIVE-14335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-14335 started by Rajat Khandelwal. --- > TaskDisplay's return value is not getting deserialized properly > --- > > Key: HIVE-14335 > URL: https://issues.apache.org/jira/browse/HIVE-14335 > Project: Hive > Issue Type: Bug >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14335) TaskDisplay's return value is not getting deserialized properly
[ https://issues.apache.org/jira/browse/HIVE-14335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajat Khandelwal updated HIVE-14335: Attachment: HIVE-14335.01.patch > TaskDisplay's return value is not getting deserialized properly > --- > > Key: HIVE-14335 > URL: https://issues.apache.org/jira/browse/HIVE-14335 > Project: Hive > Issue Type: Bug >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Attachments: HIVE-14335.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14335) TaskDisplay's return value is not getting deserialized properly
[ https://issues.apache.org/jira/browse/HIVE-14335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15393753#comment-15393753 ] Rajat Khandelwal commented on HIVE-14335: - Taking patch from reviewboard and attaching > TaskDisplay's return value is not getting deserialized properly > --- > > Key: HIVE-14335 > URL: https://issues.apache.org/jira/browse/HIVE-14335 > Project: Hive > Issue Type: Bug >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Attachments: HIVE-14335.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14335) TaskDisplay's return value is not getting deserialized properly
[ https://issues.apache.org/jira/browse/HIVE-14335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajat Khandelwal updated HIVE-14335: Status: Patch Available (was: In Progress) > TaskDisplay's return value is not getting deserialized properly > --- > > Key: HIVE-14335 > URL: https://issues.apache.org/jira/browse/HIVE-14335 > Project: Hive > Issue Type: Bug >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Attachments: HIVE-14335.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14336) Make usage of VectorUDFAdaptor configurable
[
https://issues.apache.org/jira/browse/HIVE-14336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-14336:
Attachment: HIVE-14336.01.patch
> Make usage of VectorUDFAdaptor configurable
> ---
>
> Key: HIVE-14336
> URL: https://issues.apache.org/jira/browse/HIVE-14336
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-14336.01.patch
>
>
> Add a Hive configuration variable:
> {code}
> hive.vectorized.adaptor.usage.mode = {none, chosen, all}
> {code}
> for configuring whether to attempt vectorization using the VectorUDFAdaptor.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14336) Make usage of VectorUDFAdaptor configurable
[
https://issues.apache.org/jira/browse/HIVE-14336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-14336:
Status: Patch Available (was: Open)
> Make usage of VectorUDFAdaptor configurable
> ---
>
> Key: HIVE-14336
> URL: https://issues.apache.org/jira/browse/HIVE-14336
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-14336.01.patch
>
>
> Add a Hive configuration variable:
> {code}
> hive.vectorized.adaptor.usage.mode = {none, chosen, all}
> {code}
> for configuring whether to attempt vectorization using the VectorUDFAdaptor.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14123) Add beeline configuration option to show database in the prompt
[ https://issues.apache.org/jira/browse/HIVE-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergio Peña updated HIVE-14123: --- Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Thanks [~pvary] I committed this to 2.2 > Add beeline configuration option to show database in the prompt > --- > > Key: HIVE-14123 > URL: https://issues.apache.org/jira/browse/HIVE-14123 > Project: Hive > Issue Type: Improvement > Components: Beeline, CLI >Affects Versions: 2.2.0 >Reporter: Peter Vary >Assignee: Peter Vary >Priority: Minor > Fix For: 2.2.0 > > Attachments: HIVE-14123.10.patch, HIVE-14123.2.patch, > HIVE-14123.3.patch, HIVE-14123.4.patch, HIVE-14123.5.patch, > HIVE-14123.6.patch, HIVE-14123.7.patch, HIVE-14123.8.patch, > HIVE-14123.9.patch, HIVE-14123.patch > > > There are several jira issues complaining that, the Beeline does not respect > hive.cli.print.current.db. > This is partially true, since in embedded mode, it uses the > hive.cli.print.current.db to change the prompt, since HIVE-10511. > In beeline mode, I think this function should use a beeline command line > option instead, like for the showHeader option emphasizing, that this is a > client side option. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14316) TestLlapTokenChecker.testCheckPermissions, testGetToken fail
[
https://issues.apache.org/jira/browse/HIVE-14316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15393924#comment-15393924
]
Hive QA commented on HIVE-14316:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12820022/HIVE-14316.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 10353 tests
executed
*Failed tests:*
{noformat}
TestMsgBusConnection - did not produce a TEST-*.xml file
TestSessionHooks - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testDelayedLocalityNodeCommErrorImmediateAllocation
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.testConnections
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/647/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/647/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-647/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12820022 - PreCommit-HIVE-MASTER-Build
> TestLlapTokenChecker.testCheckPermissions, testGetToken fail
>
>
> Key: HIVE-14316
> URL: https://issues.apache.org/jira/browse/HIVE-14316
> Project: Hive
> Issue Type: Test
>Reporter: Siddharth Seth
>Assignee: Sergey Shelukhin
> Attachments: HIVE-14316.patch
>
>
> cc [~sershe]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14251) Union All of different types resolves to incorrect data
[
https://issues.apache.org/jira/browse/HIVE-14251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-14251:
Attachment: HIVE-14251.4.patch
Patch-4: fix one unit test failure.
> Union All of different types resolves to incorrect data
> ---
>
> Key: HIVE-14251
> URL: https://issues.apache.org/jira/browse/HIVE-14251
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 2.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-14251.1.patch, HIVE-14251.2.patch,
> HIVE-14251.3.patch, HIVE-14251.4.patch
>
>
> create table src(c1 date, c2 int, c3 double);
> insert into src values ('2016-01-01',5,1.25);
> select * from
> (select c1 from src union all
> select c2 from src union all
> select c3 from src) t;
> It will return NULL for the c1 values. Seems the common data type is resolved
> to the last c3 which is double.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13815) Improve logic to infer false predicates
[
https://issues.apache.org/jira/browse/HIVE-13815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15393930#comment-15393930
]
Ashutosh Chauhan commented on HIVE-13815:
-
yeah.. quite a few golden file updates.. can you create a RB
> Improve logic to infer false predicates
> ---
>
> Key: HIVE-13815
> URL: https://issues.apache.org/jira/browse/HIVE-13815
> Project: Hive
> Issue Type: Sub-task
> Components: CBO
>Affects Versions: 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13815.01.patch, HIVE-13815.patch
>
>
> Follow-up/extension of the work done in HIVE-13068.
> Ex.
> ql/src/test/results/clientpositive/annotate_stats_filter.q.out
> {{predicate: ((year = 2001) and (state = 'OH') and (state = 'FL')) (type:
> boolean)}} -> {{false}}
> ql/src/test/results/clientpositive/cbo_rp_join1.q.out
> {{predicate: ((_col0 = _col1) and (_col1 = 40) and (_col0 = 40)) (type:
> boolean)}} -> {{predicate: ((_col1 = 40) and (_col0 = 40)) (type: boolean)}}
> ql/src/test/results/clientpositive/constprog_semijoin.q.out
> {{predicate: (((id = 100) = true) and (id <> 100)) (type: boolean)}} ->
> {{false}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13815) Improve logic to infer false predicates
[
https://issues.apache.org/jira/browse/HIVE-13815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15393966#comment-15393966
]
Jesus Camacho Rodriguez commented on HIVE-13815:
Sure, this is the link: https://reviews.apache.org/r/50447/
Thanks
> Improve logic to infer false predicates
> ---
>
> Key: HIVE-13815
> URL: https://issues.apache.org/jira/browse/HIVE-13815
> Project: Hive
> Issue Type: Sub-task
> Components: CBO
>Affects Versions: 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13815.01.patch, HIVE-13815.patch
>
>
> Follow-up/extension of the work done in HIVE-13068.
> Ex.
> ql/src/test/results/clientpositive/annotate_stats_filter.q.out
> {{predicate: ((year = 2001) and (state = 'OH') and (state = 'FL')) (type:
> boolean)}} -> {{false}}
> ql/src/test/results/clientpositive/cbo_rp_join1.q.out
> {{predicate: ((_col0 = _col1) and (_col1 = 40) and (_col0 = 40)) (type:
> boolean)}} -> {{predicate: ((_col1 = 40) and (_col0 = 40)) (type: boolean)}}
> ql/src/test/results/clientpositive/constprog_semijoin.q.out
> {{predicate: (((id = 100) = true) and (id <> 100)) (type: boolean)}} ->
> {{false}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13688) Variable substitutions not working for shell commands in beeline
[
https://issues.apache.org/jira/browse/HIVE-13688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15394016#comment-15394016
]
Sergio Peña commented on HIVE-13688:
What variables are used for substitution? Doesn't the user provide such
variables (as a sql script or command parameter) when running beeline? If so,
then they need to be aware that those variables may be used for substitution
when running their queries (even on beeline), don't they?
> Variable substitutions not working for shell commands in beeline
>
>
> Key: HIVE-13688
> URL: https://issues.apache.org/jira/browse/HIVE-13688
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 1.1.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
>
> Appears that the variable substitutions work fine for DML and DDL commands.
> > set hivevar:v1=databases;
> No rows affected (0.004 seconds)
> > show ${v1};
> ++--+
> | database_name |
> ++--+
> | default|
> ++--+
> 1 row selected (0.243 seconds)
> > set hivevar:v1=tmp;
> No rows affected (0.061 seconds)
> > create table ${v1} (col1 string);
> No rows affected (0.065 seconds)
> > describe ${v1};
> INFO : Compiling
> command(queryId=hive_20160504105151_cb6ab7f1-26ac-4733-80ed-13a5134ba2b5):
> describe tmp
> INFO : Executing
> command(queryId=hive_20160504105151_cb6ab7f1-26ac-4733-80ed-13a5134ba2b5):
> describe tmp
> +---++--+--+
> | col_name | data_type | comment |
> +---++--+--+
> | col1 | string | |
> +---++--+--+
> 1 row selected (0.121 seconds)
> But it does not appear to work for shell commands (introduced in HIVE-6791 )
> > !sh sudo -u hdfs hdfs dfs -mkdir /user/root/${v1}
> mkdir: `/user/root/${v1}': No such file or directory <== problem
> Command failed with exit code = 1
> > !sh echo ${v1};
> ${v1};
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14296) Session count is not decremented when HS2 clients do not shutdown cleanly.
[
https://issues.apache.org/jira/browse/HIVE-14296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15394025#comment-15394025
]
Naveen Gangam commented on HIVE-14296:
--
Thanks [~szehon] and [~mohitsabharwal]. Could one of you please commit when you
get a chance? Thanks in advance
> Session count is not decremented when HS2 clients do not shutdown cleanly.
> --
>
> Key: HIVE-14296
> URL: https://issues.apache.org/jira/browse/HIVE-14296
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 2.0.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
> Attachments: HIVE-14296.2.patch, HIVE-14296.patch
>
>
> When a JDBC client like beeline abruptly disconnects from HS2, the session
> gets closed on the serverside but the session count reported in the logs is
> incorrect. It never gets decremented.
> For example, I created 6 connections from the same instance of beeline to HS2.
> {code}
> 2016-07-20T15:05:17,987 INFO [HiveServer2-Handler-Pool: Thread-40]
> thrift.ThriftCLIService: Opened a session SessionHandle
> [28b225ee-204f-4b3e-b4fd-0039ef8e276e], current sessions: 1
> .
> 2016-07-20T15:05:24,239 INFO [HiveServer2-Handler-Pool: Thread-45]
> thrift.ThriftCLIService: Opened a session SessionHandle
> [1d267de8-ff9a-4e76-ac5c-e82c871588e7], current sessions: 2
> .
> 2016-07-20T15:05:25,710 INFO [HiveServer2-Handler-Pool: Thread-50]
> thrift.ThriftCLIService: Opened a session SessionHandle
> [04d53deb-8965-464b-aa3f-7042304cfb54], current sessions: 3
> .
> 2016-07-20T15:05:26,795 INFO [HiveServer2-Handler-Pool: Thread-55]
> thrift.ThriftCLIService: Opened a session SessionHandle
> [b4bb8b86-74e1-4e3c-babb-674d34ad1caf], current sessions: 4
> 2016-07-20T15:05:28,160 INFO [HiveServer2-Handler-Pool: Thread-60]
> thrift.ThriftCLIService: Opened a session SessionHandle
> [6d3c3ed9-fadb-4673-8c15-3315b7e2995d], current sessions: 5
> .
> 2016-07-20T15:05:29,136 INFO [HiveServer2-Handler-Pool: Thread-65]
> thrift.ThriftCLIService: Opened a session SessionHandle
> [88b630c0-f272-427d-8263-febfef8d], current sessions: 6
> {code}
> When I CNTRL-C the beeline process, in the HS2 logs I see
> {code}
> 2016-07-20T15:11:37,858 INFO [HiveServer2-Handler-Pool: Thread-55]
> thrift.ThriftCLIService: Session disconnected without closing properly.
> 2016-07-20T15:11:37,858 INFO [HiveServer2-Handler-Pool: Thread-40]
> thrift.ThriftCLIService: Session disconnected without closing properly.
> 2016-07-20T15:11:37,858 INFO [HiveServer2-Handler-Pool: Thread-65]
> thrift.ThriftCLIService: Session disconnected without closing properly.
> 2016-07-20T15:11:37,858 INFO [HiveServer2-Handler-Pool: Thread-60]
> thrift.ThriftCLIService: Session disconnected without closing properly.
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-50]
> thrift.ThriftCLIService: Session disconnected without closing properly.
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-45]
> thrift.ThriftCLIService: Session disconnected without closing properly.
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-55]
> thrift.ThriftCLIService: Closing the session: SessionHandle
> [b4bb8b86-74e1-4e3c-babb-674d34ad1caf]
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-40]
> thrift.ThriftCLIService: Closing the session: SessionHandle
> [28b225ee-204f-4b3e-b4fd-0039ef8e276e]
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-65]
> thrift.ThriftCLIService: Closing the session: SessionHandle
> [88b630c0-f272-427d-8263-febfef8d]
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-60]
> thrift.ThriftCLIService: Closing the session: SessionHandle
> [6d3c3ed9-fadb-4673-8c15-3315b7e2995d]
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-45]
> thrift.ThriftCLIService: Closing the session: SessionHandle
> [1d267de8-ff9a-4e76-ac5c-e82c871588e7]
> 2016-07-20T15:11:37,859 INFO [HiveServer2-Handler-Pool: Thread-50]
> thrift.ThriftCLIService: Closing the session: SessionHandle
> [04d53deb-8965-464b-aa3f-7042304cfb54]
> {code}
> The next time I connect to HS2 via beeline, I see
> {code}
> 2016-07-20T15:14:33,679 INFO [HiveServer2-Handler-Pool: Thread-50]
> thrift.ThriftCLIService: Client protocol version: HIVE_CLI_SERVICE_PROTOCOL_V8
> 2016-07-20T15:14:33,710 INFO [HiveServer2-Handler-Pool: Thread-50]
> session.SessionState: Created HDFS directory:
> /tmp/hive/hive/d47759e8-df3a-4504-9f28-99ff5247352c
> 2016-07-20T15:14:33,725 INFO [HiveServer2-Handler-Pool: Thread-50]
> session.SessionState: Created local directory:
> /var/folders/_3/0w477k4j5bjd6h967rw4vflwgp/T/ngangam/d47759e8-df3a-4504-9f28-99ff5247352c
> 2016-07-20T15:14:33,73
[jira] [Commented] (HIVE-13815) Improve logic to infer false predicates
[
https://issues.apache.org/jira/browse/HIVE-13815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15394039#comment-15394039
]
Ashutosh Chauhan commented on HIVE-13815:
-
+1
> Improve logic to infer false predicates
> ---
>
> Key: HIVE-13815
> URL: https://issues.apache.org/jira/browse/HIVE-13815
> Project: Hive
> Issue Type: Sub-task
> Components: CBO
>Affects Versions: 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13815.01.patch, HIVE-13815.patch
>
>
> Follow-up/extension of the work done in HIVE-13068.
> Ex.
> ql/src/test/results/clientpositive/annotate_stats_filter.q.out
> {{predicate: ((year = 2001) and (state = 'OH') and (state = 'FL')) (type:
> boolean)}} -> {{false}}
> ql/src/test/results/clientpositive/cbo_rp_join1.q.out
> {{predicate: ((_col0 = _col1) and (_col1 = 40) and (_col0 = 40)) (type:
> boolean)}} -> {{predicate: ((_col1 = 40) and (_col0 = 40)) (type: boolean)}}
> ql/src/test/results/clientpositive/constprog_semijoin.q.out
> {{predicate: (((id = 100) = true) and (id <> 100)) (type: boolean)}} ->
> {{false}}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14259) FileUtils.isSubDir may return incorrect result
[ https://issues.apache.org/jira/browse/HIVE-14259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15394089#comment-15394089 ] Ashutosh Chauhan commented on HIVE-14259: - [~kgyrtkirk] You wanna update your patch as per [~ruili] suggestion? > FileUtils.isSubDir may return incorrect result > -- > > Key: HIVE-14259 > URL: https://issues.apache.org/jira/browse/HIVE-14259 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Minor > Attachments: HIVE-14259.1.patch > > > while I was working on HIVE-12244 i've looked around for utility > methods...i've found this method; but it considers path: `/dir12` inside > `/dir1` > which is not true -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14332) Reduce logging from VectorMapOperator
[ https://issues.apache.org/jira/browse/HIVE-14332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15394108#comment-15394108 ] Siddharth Seth commented on HIVE-14332: --- Would it be useful to retain this at the DEBUG level? (or is the information available in some other way). I'm +1 on either - i.e. the current patch, or moving it to debug. Your call on whether it is useful for debugging or not. > Reduce logging from VectorMapOperator > - > > Key: HIVE-14332 > URL: https://issues.apache.org/jira/browse/HIVE-14332 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-14332.01.patch > > > org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator: VectorMapOperator > path: > hdfs://cn108-10.l42scl.hortonworks.com:8020/apps/hive/warehouse/tpcds_bin_partitioned_orc_200.db/store_sales/ss_sold_date_sk=2451710, > read type VECTORIZED_INPUT_FILE_FORMAT, vector deserialize type NONE, > aliases store_sales > Lines like this repeat all over the log. This gets really big with a large > number of partitions. 6MB of logs per node for a 30 task query running for 20 > seconds on a 3 node cluster. > Instead of logging this line - can we have a consolidated log / logging only > if something abnormal happens ... or a shorter log message. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13930) upgrade Hive to latest Hadoop version
[
https://issues.apache.org/jira/browse/HIVE-13930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15394122#comment-15394122
]
Hive QA commented on HIVE-13930:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12820014/HIVE-13930.05.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 584 failed/errored test(s), 10365 tests
executed
*Failed tests:*
{noformat}
TestMsgBusConnection - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_move_tbl
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_add_part_multiple
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_alter_merge_orc
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_alter_merge_stats_orc
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_annotate_stats_join
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join0
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join10
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join11
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join12
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join13
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join14
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join15
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join16
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join17
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join18
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join18_multi_distinct
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join19
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join20
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join21
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join22
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join23
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join24
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join26
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join27
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join28
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join29
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join3
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join30
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join31
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join4
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join5
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join6
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join7
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join8
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join9
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join_filters
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join_nulls
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join_reordering_values
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join_stats
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join_stats2
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_join_without_localtask
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_smb_mapjoin_14
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_10
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_12
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_13
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_14
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_15
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_16
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_3
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sor
[jira] [Commented] (HIVE-12727) refactor Hive strict checks to be more granular, allow order by no limit and no partition filter by default for now
[ https://issues.apache.org/jira/browse/HIVE-12727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15394138#comment-15394138 ] Xuefu Zhang commented on HIVE-12727: It seems to be that the default value for hive.mapred.mode should be null then. Right now the default value is "nonstrict", which makes the three new configs useless. Thoughts, [~sershe]? > refactor Hive strict checks to be more granular, allow order by no limit and > no partition filter by default for now > --- > > Key: HIVE-12727 > URL: https://issues.apache.org/jira/browse/HIVE-12727 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Blocker > Labels: TODOC2.0 > Fix For: 2.0.0 > > Attachments: HIVE-12727.01.patch, HIVE-12727.02.patch, > HIVE-12727.03.patch, HIVE-12727.04.patch, HIVE-12727.05.patch, > HIVE-12727.06.patch, HIVE-12727.07.patch, HIVE-12727.patch > > > Making strict mode the default recently appears to have broken many normal > queries, such as some TPCDS benchmark queries, e.g. Q85: > Response message: org.apache.hive.service.cli.HiveSQLException: Error while > compiling statement: FAILED: SemanticException [Error 10041]: No partition > predicate found for Alias "web_sales" Table "web_returns" > We should remove this restriction from strict mode, or change the default > back to non-strict. Perhaps make a 3-value parameter, nonstrict, semistrict, > and strict, for backward compat for people who are relying on strict already. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13930) upgrade Hive to latest Hadoop version
[ https://issues.apache.org/jira/browse/HIVE-13930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15394162#comment-15394162 ] Sahil Takiar commented on HIVE-13930: - Looks like the SparkCliDriver is still failing, let me take a closer look. > upgrade Hive to latest Hadoop version > - > > Key: HIVE-13930 > URL: https://issues.apache.org/jira/browse/HIVE-13930 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13930.01.patch, HIVE-13930.02.patch, > HIVE-13930.03.patch, HIVE-13930.04.patch, HIVE-13930.05.patch, > HIVE-13930.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Issue Comment Deleted] (HIVE-13930) upgrade Hive to latest Hadoop version
[ https://issues.apache.org/jira/browse/HIVE-13930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar updated HIVE-13930: Comment: was deleted (was: Test failures look unrelated.) > upgrade Hive to latest Hadoop version > - > > Key: HIVE-13930 > URL: https://issues.apache.org/jira/browse/HIVE-13930 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13930.01.patch, HIVE-13930.02.patch, > HIVE-13930.03.patch, HIVE-13930.04.patch, HIVE-13930.05.patch, > HIVE-13930.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14170) Beeline IncrementalRows should buffer rows and incrementally re-calculate width if TableOutputFormat is used
[
https://issues.apache.org/jira/browse/HIVE-14170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15394161#comment-15394161
]
Sahil Takiar commented on HIVE-14170:
-
Test failures look unrelated.
> Beeline IncrementalRows should buffer rows and incrementally re-calculate
> width if TableOutputFormat is used
>
>
> Key: HIVE-14170
> URL: https://issues.apache.org/jira/browse/HIVE-14170
> Project: Hive
> Issue Type: Sub-task
> Components: Beeline
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Attachments: HIVE-14170.1.patch, HIVE-14170.2.patch,
> HIVE-14170.3.patch, HIVE-14170.4.patch
>
>
> If {{--incremental}} is specified in Beeline, rows are meant to be printed
> out immediately. However, if {{TableOutputFormat}} is used with this option
> the formatting can look really off.
> The reason is that {{IncrementalRows}} does not do a global calculation of
> the optimal width size for {{TableOutputFormat}} (it can't because it only
> sees one row at a time). The output of {{BufferedRows}} looks much better
> because it can do this global calculation.
> If {{--incremental}} is used, and {{TableOutputFormat}} is used, the width
> should be re-calculated every "x" rows ("x" can be configurable and by
> default it can be 1000).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13930) upgrade Hive to latest Hadoop version
[ https://issues.apache.org/jira/browse/HIVE-13930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15394160#comment-15394160 ] Sahil Takiar commented on HIVE-13930: - Test failures look unrelated. > upgrade Hive to latest Hadoop version > - > > Key: HIVE-13930 > URL: https://issues.apache.org/jira/browse/HIVE-13930 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13930.01.patch, HIVE-13930.02.patch, > HIVE-13930.03.patch, HIVE-13930.04.patch, HIVE-13930.05.patch, > HIVE-13930.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14323) Reduce number of FS permissions and redundant FS operations
[ https://issues.apache.org/jira/browse/HIVE-14323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15394167#comment-15394167 ] Ashutosh Chauhan commented on HIVE-14323: - +1 [~rajesh.balamohan] would you like to upload your RB patch on jira, so that Hive QA can run on it? > Reduce number of FS permissions and redundant FS operations > --- > > Key: HIVE-14323 > URL: https://issues.apache.org/jira/browse/HIVE-14323 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-14323.1.patch > > > Some examples are given below. > 1. When creating stage directory, FileUtils sets the directory permissions by > running a set of chgrp and chmod commands. In systems like S3, this would not > be relevant. > 2. In some cases, fs.delete() is followed by fs.exists(). In this case, it > might be redundant to check for exists() (lookup ops are expensive in systems > like S3). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14335) TaskDisplay's return value is not getting deserialized properly
[ https://issues.apache.org/jira/browse/HIVE-14335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15394183#comment-15394183 ] Szehon Ho commented on HIVE-14335: -- +1 fine with me. why does renaming it allow it to get deserialized? > TaskDisplay's return value is not getting deserialized properly > --- > > Key: HIVE-14335 > URL: https://issues.apache.org/jira/browse/HIVE-14335 > Project: Hive > Issue Type: Bug >Reporter: Rajat Khandelwal >Assignee: Rajat Khandelwal > Attachments: HIVE-14335.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14227) Investigate invalid SessionHandle and invalid OperationHandle
[
https://issues.apache.org/jira/browse/HIVE-14227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15394188#comment-15394188
]
Tao Li commented on HIVE-14227:
---
Hi [~aihuaxu], I just started looking at this issue and related code. Looks
like each instance of HiveConnection class holds a reference to a Transport
instance (representing the underlying tcp/http connection) and a reference to
SessionHandle. Like you mentioned, there is no binding between the Transport
instance and SessionHandle. But I don't see how a new HiveConnection can hold a
reference to an existing SessionHandle. Looks like when we need to create a new
HiveConnection, we create a new HiveSessionImpl instance and attach the handle
the HiveConnection. Also when we need to close a HiveConnection, we close the
Hive session first.
So can you elaborate how a new HiveConnection can use an existing HiveSession?
Thanks!
> Investigate invalid SessionHandle and invalid OperationHandle
> -
>
> Key: HIVE-14227
> URL: https://issues.apache.org/jira/browse/HIVE-14227
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 2.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-14227.1.patch
>
>
> There are the following warnings.
> {noformat}
> WARN org.apache.hive.service.cli.thrift.ThriftCLIService:
> [HiveServer2-Handler-Pool: Thread-55]: Error executing statement:
> org.apache.hive.service.cli.HiveSQLException: Invalid SessionHandle:
> SessionHandle [1bc00251-64e9-4a95-acb7-a7f53f773528]
> at
> org.apache.hive.service.cli.session.SessionManager.getSession(SessionManager.java:318)
> at
> org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:258)
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:506)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> {noformat}
> {noformat}
> WARN org.apache.hive.service.cli.thrift.ThriftCLIService:
> [HiveServer2-Handler-Pool: Thread-1060]: Error closing operation:
> org.apache.hive.service.cli.HiveSQLException: Invalid OperationHandle:
> OperationHandle [opType=EXECUTE_STATEMENT,
> getHandleIdentifier()=13d930dd-316c-4c09-9f44-fee5f483e73d]
> at
> org.apache.hive.service.cli.operation.OperationManager.getOperation(OperationManager.java:185)
> at
> org.apache.hive.service.cli.CLIService.closeOperation(CLIService.java:408)
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.CloseOperation(ThriftCLIService.java:664)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$CloseOperation.getResult(TCLIService.java:1513)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$CloseOperation.getResult(TCLIService.java:1498)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14227) Investigate invalid SessionHandle and invalid OperationHandle
[
https://issues.apache.org/jira/browse/HIVE-14227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15394197#comment-15394197
]
Aihua Xu commented on HIVE-14227:
-
[~taoli-hwx] This issue happens when you use thrift API directly, not from
JDBC. One workflow can be as follows. Notice you need to pass sessionHandle to
each request so you can pass in an existing sessionHandle created from another
connection. Then when you close the connection, it will also close the session
while the session is still used by another connection.
{noformat}
final TSocket tSocket = new TSocket(host, port);
tSocket.open();
final Client client = new Client(new TBinaryProtocol(tSocket));
TOpenSessionReq req = new TOpenSessionReq();
Map openConf = new HashMap();
req.setConfiguration(openConf);
TOpenSessionResp resp = client.OpenSession(req);
TGetTablesReq getTablesReq = new TGetTablesReq(resp.getSessionHandle());
client.GetTables(getTablesReq);
{noformat}
> Investigate invalid SessionHandle and invalid OperationHandle
> -
>
> Key: HIVE-14227
> URL: https://issues.apache.org/jira/browse/HIVE-14227
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 2.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-14227.1.patch
>
>
> There are the following warnings.
> {noformat}
> WARN org.apache.hive.service.cli.thrift.ThriftCLIService:
> [HiveServer2-Handler-Pool: Thread-55]: Error executing statement:
> org.apache.hive.service.cli.HiveSQLException: Invalid SessionHandle:
> SessionHandle [1bc00251-64e9-4a95-acb7-a7f53f773528]
> at
> org.apache.hive.service.cli.session.SessionManager.getSession(SessionManager.java:318)
> at
> org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:258)
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:506)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> {noformat}
> {noformat}
> WARN org.apache.hive.service.cli.thrift.ThriftCLIService:
> [HiveServer2-Handler-Pool: Thread-1060]: Error closing operation:
> org.apache.hive.service.cli.HiveSQLException: Invalid OperationHandle:
> OperationHandle [opType=EXECUTE_STATEMENT,
> getHandleIdentifier()=13d930dd-316c-4c09-9f44-fee5f483e73d]
> at
> org.apache.hive.service.cli.operation.OperationManager.getOperation(OperationManager.java:185)
> at
> org.apache.hive.service.cli.CLIService.closeOperation(CLIService.java:408)
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.CloseOperation(ThriftCLIService.java:664)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$CloseOperation.getResult(TCLIService.java:1513)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$CloseOperation.getResult(TCLIService.java:1498)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14338) Delete/Alter table calls failing with HiveAccessControlException
[ https://issues.apache.org/jira/browse/HIVE-14338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-14338: --- Attachment: HIVE-14338.01.patch > Delete/Alter table calls failing with HiveAccessControlException > > > Key: HIVE-14338 > URL: https://issues.apache.org/jira/browse/HIVE-14338 > Project: Hive > Issue Type: Bug >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-14338.01.patch > > > Many Hcatalog/Webhcat tests are failing with below error, when tests try to > alter/delete/describe tables. Error is thrown when the same user or a > different user (same group) who created the table is trying to run the > delete/alter table call. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14338) Delete/Alter table calls failing with HiveAccessControlException
[ https://issues.apache.org/jira/browse/HIVE-14338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-14338: --- Status: Patch Available (was: Open) > Delete/Alter table calls failing with HiveAccessControlException > > > Key: HIVE-14338 > URL: https://issues.apache.org/jira/browse/HIVE-14338 > Project: Hive > Issue Type: Bug >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-14338.01.patch > > > Many Hcatalog/Webhcat tests are failing with below error, when tests try to > alter/delete/describe tables. Error is thrown when the same user or a > different user (same group) who created the table is trying to run the > delete/alter table call. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14338) Delete/Alter table calls failing with HiveAccessControlException
[ https://issues.apache.org/jira/browse/HIVE-14338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15394201#comment-15394201 ] Pengcheng Xiong commented on HIVE-14338: [~thejas] and [~ashutoshc], could u take a look? It seems that I overdid the configuration change. Simply remove that from HCatCli is OK. I have run some tests locally and confirmed... > Delete/Alter table calls failing with HiveAccessControlException > > > Key: HIVE-14338 > URL: https://issues.apache.org/jira/browse/HIVE-14338 > Project: Hive > Issue Type: Bug >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-14338.01.patch > > > Many Hcatalog/Webhcat tests are failing with below error, when tests try to > alter/delete/describe tables. Error is thrown when the same user or a > different user (same group) who created the table is trying to run the > delete/alter table call. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14323) Reduce number of FS permissions and redundant FS operations
[ https://issues.apache.org/jira/browse/HIVE-14323?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15394202#comment-15394202 ] Chris Nauroth commented on HIVE-14323: -- +1 (non-binding) from me too. Thank you, Rajesh. > Reduce number of FS permissions and redundant FS operations > --- > > Key: HIVE-14323 > URL: https://issues.apache.org/jira/browse/HIVE-14323 > Project: Hive > Issue Type: Improvement >Reporter: Rajesh Balamohan >Assignee: Rajesh Balamohan >Priority: Minor > Attachments: HIVE-14323.1.patch > > > Some examples are given below. > 1. When creating stage directory, FileUtils sets the directory permissions by > running a set of chgrp and chmod commands. In systems like S3, this would not > be relevant. > 2. In some cases, fs.delete() is followed by fs.exists(). In this case, it > might be redundant to check for exists() (lookup ops are expensive in systems > like S3). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14342) Beeline output is garbled when executed from a remote shell
[
https://issues.apache.org/jira/browse/HIVE-14342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Naveen Gangam updated HIVE-14342:
-
Description:
{code}
use default;
create table clitest (key int, name String, value String);
insert into table clitest values
(1,"TRUE","1"),(2,"TRUE","1"),(3,"TRUE","1"),(4,"TRUE","1"),(5,"FALSE","0"),(6,"FALSE","0"),(7,"FALSE","0");
{code}
then run a select query
{code}
# cat /tmp/select.sql
set hive.execution.engine=mr;
select key,name,value
from clitest
where value="1" limit 1;
{code}
Then run beeline via a remote shell, for example
{code}
$ ssh -l root "sudo -u hive beeline -u
jdbc:hive2://localhost:1 -n hive -p hive --silent=true --outputformat=csv2
-f /tmp/select.sql"
root@'s password:
16/07/12 14:59:22 WARN mapreduce.TableMapReduceUtil: The hbase-prefix-tree
module jar containing PrefixTreeCodec is not present. Continuing without it.
nullkey,name,value
1,TRUE,1
null
$
{code}
In older releases that the output is as follows
{code}
$ ssh -l root "sudo -u hive beeline -u
jdbc:hive2://localhost:1 -n hive -p hive --silent=true --outputformat=csv2
-f /tmp/run.sql"
Are you sure you want to continue connecting (yes/no)? yes
root@'s password:
16/07/12 14:57:55 WARN mapreduce.TableMapReduceUtil: The hbase-prefix-tree
module jar containing PrefixTreeCodec is not present. Continuing without it.
key,name,value
1,TRUE,1
$
{code}
The output contains nulls instead of blank lines. This is due to the use of
-Djline.terminal=jline.UnsupportedTerminal introduced in HIVE-6758 to be able
to run beeline as a background process. But this is the unfortunate side effect
of that fix.
Running beeline in background also produces garbled output.
{code}
# beeline -u "jdbc:hive2://localhost:1" -n hive -p hive --silent=true
--outputformat=csv2 --showHeader=false -f /tmp/run.sql 2>&1 > /tmp/beeline.txt &
# cat /tmp/beeline.txt
null1,TRUE,1
#
{code}
So I think the use of jline.UnsupportedTerminal should be documented but not
used automatically by beeline under the covers.
was:
{code}
use default;
create table clitest (key int, name String, value String);
insert into table clitest values
(1,"TRUE","1"),(2,"TRUE","1"),(3,"TRUE","1"),(4,"TRUE","1"),(5,"FALSE","0"),(6,"FALSE","0"),(7,"FALSE","0");
{code}
then run a select query
{code}
# cat /tmp/select.sql
set hive.execution.engine=mr;
select key,name,value
from clitest
where value="1" limit 1;
{code}
Then run beeline via a remote shell, for example
$ ssh -l root "sudo -u hive beeline -u
jdbc:hive2://localhost:1 -n hive -p hive --silent=true --outputformat=csv2
-f /tmp/select.sql"
root@'s password:
16/07/12 14:59:22 WARN mapreduce.TableMapReduceUtil: The hbase-prefix-tree
module jar containing PrefixTreeCodec is not present. Continuing without it.
nullkey,name,value
1,TRUE,1
null
$
{code}
In older releases that the output is as follows
{code}
$ ssh -l root "sudo -u hive beeline -u
jdbc:hive2://localhost:1 -n hive -p hive --silent=true --outputformat=csv2
-f /tmp/run.sql"
Are you sure you want to continue connecting (yes/no)? yes
root@'s password:
16/07/12 14:57:55 WARN mapreduce.TableMapReduceUtil: The hbase-prefix-tree
module jar containing PrefixTreeCodec is not present. Continuing without it.
key,name,value
1,TRUE,1
$
{code}
The output contains nulls instead of blank lines. This is due to the use of
-Djline.terminal=jline.UnsupportedTerminal introduced in HIVE-6758 to be able
to run beeline as a background process. But this is the unfortunate side effect
of that fix.
Running beeline in background also produces garbled output.
{code}
# beeline -u "jdbc:hive2://localhost:1" -n hive -p hive --silent=true
--outputformat=csv2 --showHeader=false -f /tmp/run.sql 2>&1 > /tmp/beeline.txt &
# cat /tmp/beeline.txt
null1,TRUE,1
#
{code}
So I think the use of jline.UnsupportedTerminal should be documented but not
used automatically by beeline under the covers.
> Beeline output is garbled when executed from a remote shell
> ---
>
> Key: HIVE-14342
> URL: https://issues.apache.org/jira/browse/HIVE-14342
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 2.0.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
>
> {code}
> use default;
> create table clitest (key int, name String, value String);
> insert into table clitest values
> (1,"TRUE","1"),(2,"TRUE","1"),(3,"TRUE","1"),(4,"TRUE","1"),(5,"FALSE","0"),(6,"FALSE","0"),(7,"FALSE","0");
> {code}
> then run a select query
> {cod
[jira] [Comment Edited] (HIVE-14315) Implement StatsProvidingRecordReader for ParquetRecordReaderWrapper
[
https://issues.apache.org/jira/browse/HIVE-14315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15394270#comment-15394270
]
Chao Sun edited comment on HIVE-14315 at 7/26/16 6:25 PM:
--
Test failures unrelated. Pushed to master. Thanks Xuefu for the review!
was (Author: csun):
Pushed to master. Thanks Xuefu for the review!
> Implement StatsProvidingRecordReader for ParquetRecordReaderWrapper
> ---
>
> Key: HIVE-14315
> URL: https://issues.apache.org/jira/browse/HIVE-14315
> Project: Hive
> Issue Type: New Feature
> Components: Statistics
>Affects Versions: 2.1.0
>Reporter: Chao Sun
>Assignee: Chao Sun
> Fix For: 2.2.0
>
> Attachments: HIVE-14315.0.patch, HIVE-14315.1.patch
>
>
> Currently only ORC supports {{analyze table ... compute statistics noscan}}
> (via HIVE-6578) where stats such as # of rows, raw datasize, etc., can be
> obtained via the footer. The similar functionality should be implemented on
> Parquet since it also has the info in the metadata.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14315) Implement StatsProvidingRecordReader for ParquetRecordReaderWrapper
[
https://issues.apache.org/jira/browse/HIVE-14315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated HIVE-14315:
Resolution: Fixed
Fix Version/s: 2.2.0
Status: Resolved (was: Patch Available)
Pushed to master. Thanks Xuefu for the review!
> Implement StatsProvidingRecordReader for ParquetRecordReaderWrapper
> ---
>
> Key: HIVE-14315
> URL: https://issues.apache.org/jira/browse/HIVE-14315
> Project: Hive
> Issue Type: New Feature
> Components: Statistics
>Affects Versions: 2.1.0
>Reporter: Chao Sun
>Assignee: Chao Sun
> Fix For: 2.2.0
>
> Attachments: HIVE-14315.0.patch, HIVE-14315.1.patch
>
>
> Currently only ORC supports {{analyze table ... compute statistics noscan}}
> (via HIVE-6578) where stats such as # of rows, raw datasize, etc., can be
> obtained via the footer. The similar functionality should be implemented on
> Parquet since it also has the info in the metadata.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14035) Enable predicate pushdown to delta files created by ACID Transactions
[ https://issues.apache.org/jira/browse/HIVE-14035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saket Saurabh updated HIVE-14035: - Attachment: HIVE-14035.11.patch Add more UTs to specifically test AcidUtils and various compaction scenarios for split-update. > Enable predicate pushdown to delta files created by ACID Transactions > - > > Key: HIVE-14035 > URL: https://issues.apache.org/jira/browse/HIVE-14035 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Saket Saurabh >Assignee: Saket Saurabh > Attachments: HIVE-14035.02.patch, HIVE-14035.03.patch, > HIVE-14035.04.patch, HIVE-14035.05.patch, HIVE-14035.06.patch, > HIVE-14035.07.patch, HIVE-14035.08.patch, HIVE-14035.09.patch, > HIVE-14035.10.patch, HIVE-14035.11.patch, HIVE-14035.patch > > > In current Hive version, delta files created by ACID transactions do not > allow predicate pushdown if they contain any update/delete events. This is > done to preserve correctness when following a multi-version approach during > event collapsing, where an update event overwrites an existing insert event. > This JIRA proposes to split an update event into a combination of a delete > event followed by a new insert event, that can enable predicate push down to > all delta files without breaking correctness. To support backward > compatibility for this feature, this JIRA also proposes to add some sort of > versioning to ACID that can allow different versions of ACID transactions to > co-exist together. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14035) Enable predicate pushdown to delta files created by ACID Transactions
[ https://issues.apache.org/jira/browse/HIVE-14035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saket Saurabh updated HIVE-14035: - Status: Open (was: Patch Available) > Enable predicate pushdown to delta files created by ACID Transactions > - > > Key: HIVE-14035 > URL: https://issues.apache.org/jira/browse/HIVE-14035 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Saket Saurabh >Assignee: Saket Saurabh > Attachments: HIVE-14035.02.patch, HIVE-14035.03.patch, > HIVE-14035.04.patch, HIVE-14035.05.patch, HIVE-14035.06.patch, > HIVE-14035.07.patch, HIVE-14035.08.patch, HIVE-14035.09.patch, > HIVE-14035.10.patch, HIVE-14035.11.patch, HIVE-14035.patch > > > In current Hive version, delta files created by ACID transactions do not > allow predicate pushdown if they contain any update/delete events. This is > done to preserve correctness when following a multi-version approach during > event collapsing, where an update event overwrites an existing insert event. > This JIRA proposes to split an update event into a combination of a delete > event followed by a new insert event, that can enable predicate push down to > all delta files without breaking correctness. To support backward > compatibility for this feature, this JIRA also proposes to add some sort of > versioning to ACID that can allow different versions of ACID transactions to > co-exist together. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14035) Enable predicate pushdown to delta files created by ACID Transactions
[ https://issues.apache.org/jira/browse/HIVE-14035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saket Saurabh updated HIVE-14035: - Status: Patch Available (was: Open) > Enable predicate pushdown to delta files created by ACID Transactions > - > > Key: HIVE-14035 > URL: https://issues.apache.org/jira/browse/HIVE-14035 > Project: Hive > Issue Type: New Feature > Components: Transactions >Reporter: Saket Saurabh >Assignee: Saket Saurabh > Attachments: HIVE-14035.02.patch, HIVE-14035.03.patch, > HIVE-14035.04.patch, HIVE-14035.05.patch, HIVE-14035.06.patch, > HIVE-14035.07.patch, HIVE-14035.08.patch, HIVE-14035.09.patch, > HIVE-14035.10.patch, HIVE-14035.11.patch, HIVE-14035.patch > > > In current Hive version, delta files created by ACID transactions do not > allow predicate pushdown if they contain any update/delete events. This is > done to preserve correctness when following a multi-version approach during > event collapsing, where an update event overwrites an existing insert event. > This JIRA proposes to split an update event into a combination of a delete > event followed by a new insert event, that can enable predicate push down to > all delta files without breaking correctness. To support backward > compatibility for this feature, this JIRA also proposes to add some sort of > versioning to ACID that can allow different versions of ACID transactions to > co-exist together. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14343) HiveDriverRunHookContext's command is null in HS2 mode
[
https://issues.apache.org/jira/browse/HIVE-14343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated HIVE-14343:
Component/s: Query Processor
> HiveDriverRunHookContext's command is null in HS2 mode
> --
>
> Key: HIVE-14343
> URL: https://issues.apache.org/jira/browse/HIVE-14343
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 2.2.0
>Reporter: Chao Sun
>
> Looking at the {{Driver#runInternal(String command, boolean
> alreadyCompiled)}}:
> {code}
> HiveDriverRunHookContext hookContext = new
> HiveDriverRunHookContextImpl(conf, command);
> // Get all the driver run hooks and pre-execute them.
> List driverRunHooks;
> {code}
> The context is initialized with the {{command}} passed in to the method.
> However, this command is always null if {{alreadyCompiled}} is true, which is
> the case for HS2 mode.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14338) Delete/Alter table calls failing with HiveAccessControlException
[ https://issues.apache.org/jira/browse/HIVE-14338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15394291#comment-15394291 ] Ashutosh Chauhan commented on HIVE-14338: - +1 LGTM > Delete/Alter table calls failing with HiveAccessControlException > > > Key: HIVE-14338 > URL: https://issues.apache.org/jira/browse/HIVE-14338 > Project: Hive > Issue Type: Bug >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-14338.01.patch > > > Many Hcatalog/Webhcat tests are failing with below error, when tests try to > alter/delete/describe tables. Error is thrown when the same user or a > different user (same group) who created the table is trying to run the > delete/alter table call. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12727) refactor Hive strict checks to be more granular, allow order by no limit and no partition filter by default for now
[ https://issues.apache.org/jira/browse/HIVE-12727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15394322#comment-15394322 ] Sergey Shelukhin commented on HIVE-12727: - That makes sense. Although doesn't the above code explicitly use default == null as an argument? > refactor Hive strict checks to be more granular, allow order by no limit and > no partition filter by default for now > --- > > Key: HIVE-12727 > URL: https://issues.apache.org/jira/browse/HIVE-12727 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Blocker > Labels: TODOC2.0 > Fix For: 2.0.0 > > Attachments: HIVE-12727.01.patch, HIVE-12727.02.patch, > HIVE-12727.03.patch, HIVE-12727.04.patch, HIVE-12727.05.patch, > HIVE-12727.06.patch, HIVE-12727.07.patch, HIVE-12727.patch > > > Making strict mode the default recently appears to have broken many normal > queries, such as some TPCDS benchmark queries, e.g. Q85: > Response message: org.apache.hive.service.cli.HiveSQLException: Error while > compiling statement: FAILED: SemanticException [Error 10041]: No partition > predicate found for Alias "web_sales" Table "web_returns" > We should remove this restriction from strict mode, or change the default > back to non-strict. Perhaps make a 3-value parameter, nonstrict, semistrict, > and strict, for backward compat for people who are relying on strict already. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14227) Investigate invalid SessionHandle and invalid OperationHandle
[
https://issues.apache.org/jira/browse/HIVE-14227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15394327#comment-15394327
]
Mohit Sabharwal commented on HIVE-14227:
SessionHandle is getting passed in most (all?) calls to HS2:
https://github.com/apache/hive/blob/master/service-rpc/if/TCLIService.thrift
In TThreadPoolServer
(https://github.com/apache/thrift/blob/master/lib/java/src/org/apache/thrift/server/TThreadPoolServer.java#L300)
the call order seems to be:
createContext -> processContext -> openSession (where we capture and set
sessionHandle in the context as context.setSessionHandle(sessionHandle) ->
deleteContext
Similarly, could we do for, say ExecuteStatement:
createContext -> processContext -> ExecuteStatement (where we capture and set
sessionHandle like above) -> deleteContext
IOW, you can attach the session to any connection for every request
individually. And keep a count of number of connections for every session which
you decrement in deleteContext. When it reaches 0, you can delete the session.
Am I missing something ?
> Investigate invalid SessionHandle and invalid OperationHandle
> -
>
> Key: HIVE-14227
> URL: https://issues.apache.org/jira/browse/HIVE-14227
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 2.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-14227.1.patch
>
>
> There are the following warnings.
> {noformat}
> WARN org.apache.hive.service.cli.thrift.ThriftCLIService:
> [HiveServer2-Handler-Pool: Thread-55]: Error executing statement:
> org.apache.hive.service.cli.HiveSQLException: Invalid SessionHandle:
> SessionHandle [1bc00251-64e9-4a95-acb7-a7f53f773528]
> at
> org.apache.hive.service.cli.session.SessionManager.getSession(SessionManager.java:318)
> at
> org.apache.hive.service.cli.CLIService.executeStatementAsync(CLIService.java:258)
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.ExecuteStatement(ThriftCLIService.java:506)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1313)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$ExecuteStatement.getResult(TCLIService.java:1298)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> {noformat}
> {noformat}
> WARN org.apache.hive.service.cli.thrift.ThriftCLIService:
> [HiveServer2-Handler-Pool: Thread-1060]: Error closing operation:
> org.apache.hive.service.cli.HiveSQLException: Invalid OperationHandle:
> OperationHandle [opType=EXECUTE_STATEMENT,
> getHandleIdentifier()=13d930dd-316c-4c09-9f44-fee5f483e73d]
> at
> org.apache.hive.service.cli.operation.OperationManager.getOperation(OperationManager.java:185)
> at
> org.apache.hive.service.cli.CLIService.closeOperation(CLIService.java:408)
> at
> org.apache.hive.service.cli.thrift.ThriftCLIService.CloseOperation(ThriftCLIService.java:664)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$CloseOperation.getResult(TCLIService.java:1513)
> at
> org.apache.hive.service.cli.thrift.TCLIService$Processor$CloseOperation.getResult(TCLIService.java:1498)
> at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
> at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14344) Intermittent failures caused by leaking delegation tokens
[
https://issues.apache.org/jira/browse/HIVE-14344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Drome updated HIVE-14344:
---
Attachment: HIVE-14344.patch
> Intermittent failures caused by leaking delegation tokens
> -
>
> Key: HIVE-14344
> URL: https://issues.apache.org/jira/browse/HIVE-14344
> Project: Hive
> Issue Type: Bug
> Components: Tez
>Affects Versions: 1.2.1, 2.1.0
>Reporter: Chris Drome
>Assignee: Chris Drome
> Attachments: HIVE-14344.patch
>
>
> We have experienced random job failures caused by leaking delegation tokens.
> The Tez child task will fail because it is attempting to read from the
> delegation tokens directory of a different (related) task.
> Failure results in the following type of stack trace:
> {noformat}
> 2016-07-21 16:57:18,061 [FATAL] [TezChild] |tez.ReduceRecordSource|:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row (tag=0)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:370)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:292)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:249)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:148)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:137)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:362)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:179)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:171)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1738)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:171)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:167)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.RuntimeException: java.io.IOException: Exception reading
> file:/grid/4/tmp/yarn-local/usercache/.../appcache/application_1468602386465_489814/container_e02_1468602386465_489814_01_01/container_tokens
> at
> org.apache.hadoop.hive.ql.exec.persistence.RowContainer.first(RowContainer.java:237)
> at
> org.apache.hadoop.hive.ql.exec.persistence.RowContainer.first(RowContainer.java:74)
> at
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genUniqueJoinObject(CommonJoinOperator.java:650)
> at
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:756)
> at
> org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.joinObject(CommonMergeJoinOperator.java:316)
> at
> org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.joinOneGroup(CommonMergeJoinOperator.java:279)
> at
> org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.joinOneGroup(CommonMergeJoinOperator.java:272)
> at
> org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.process(CommonMergeJoinOperator.java:258)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:361)
> ... 17 more
> Caused by: java.lang.RuntimeException: java.io.IOException: Exception reading
> file:/grid/4/tmp/yarn-local/usercache/.../appcache/application_1468602386465_489814/container_e02_1468602386465_489814_01_01/container_tokens
> at
> org.apache.hadoop.mapreduce.security.TokenCache.mergeBinaryTokens(TokenCache.java:141)
> at
> org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:119)
> at
> org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:100)
> at
> org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:80)
> at
> org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:206)
> at
> org.apache.hadoop.mapred.SequenceFileInputFormat.listStatus(SequenceFileInputFormat.java:45
[jira] [Updated] (HIVE-14344) Intermittent failures caused by leaking delegation tokens
[
https://issues.apache.org/jira/browse/HIVE-14344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Drome updated HIVE-14344:
---
Attachment: HIVE-14344-branch-1.patch
> Intermittent failures caused by leaking delegation tokens
> -
>
> Key: HIVE-14344
> URL: https://issues.apache.org/jira/browse/HIVE-14344
> Project: Hive
> Issue Type: Bug
> Components: Tez
>Affects Versions: 1.2.1, 2.1.0
>Reporter: Chris Drome
>Assignee: Chris Drome
> Attachments: HIVE-14344-branch-1.patch, HIVE-14344.patch
>
>
> We have experienced random job failures caused by leaking delegation tokens.
> The Tez child task will fail because it is attempting to read from the
> delegation tokens directory of a different (related) task.
> Failure results in the following type of stack trace:
> {noformat}
> 2016-07-21 16:57:18,061 [FATAL] [TezChild] |tez.ReduceRecordSource|:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row (tag=0)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:370)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:292)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:249)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:148)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:137)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:362)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:179)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:171)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1738)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:171)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:167)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.RuntimeException: java.io.IOException: Exception reading
> file:/grid/4/tmp/yarn-local/usercache/.../appcache/application_1468602386465_489814/container_e02_1468602386465_489814_01_01/container_tokens
> at
> org.apache.hadoop.hive.ql.exec.persistence.RowContainer.first(RowContainer.java:237)
> at
> org.apache.hadoop.hive.ql.exec.persistence.RowContainer.first(RowContainer.java:74)
> at
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genUniqueJoinObject(CommonJoinOperator.java:650)
> at
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:756)
> at
> org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.joinObject(CommonMergeJoinOperator.java:316)
> at
> org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.joinOneGroup(CommonMergeJoinOperator.java:279)
> at
> org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.joinOneGroup(CommonMergeJoinOperator.java:272)
> at
> org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.process(CommonMergeJoinOperator.java:258)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:361)
> ... 17 more
> Caused by: java.lang.RuntimeException: java.io.IOException: Exception reading
> file:/grid/4/tmp/yarn-local/usercache/.../appcache/application_1468602386465_489814/container_e02_1468602386465_489814_01_01/container_tokens
> at
> org.apache.hadoop.mapreduce.security.TokenCache.mergeBinaryTokens(TokenCache.java:141)
> at
> org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:119)
> at
> org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:100)
> at
> org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:80)
> at
> org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:206)
> at
> org.apache.hadoop.mapred.SequenceFileInputFormat.listSt
[jira] [Updated] (HIVE-14344) Intermittent failures caused by leaking delegation tokens
[
https://issues.apache.org/jira/browse/HIVE-14344?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Drome updated HIVE-14344:
---
Target Version/s: 2.1.0, 1.2.1
Status: Patch Available (was: Open)
> Intermittent failures caused by leaking delegation tokens
> -
>
> Key: HIVE-14344
> URL: https://issues.apache.org/jira/browse/HIVE-14344
> Project: Hive
> Issue Type: Bug
> Components: Tez
>Affects Versions: 2.1.0, 1.2.1
>Reporter: Chris Drome
>Assignee: Chris Drome
> Attachments: HIVE-14344-branch-1.patch, HIVE-14344.patch
>
>
> We have experienced random job failures caused by leaking delegation tokens.
> The Tez child task will fail because it is attempting to read from the
> delegation tokens directory of a different (related) task.
> Failure results in the following type of stack trace:
> {noformat}
> 2016-07-21 16:57:18,061 [FATAL] [TezChild] |tez.ReduceRecordSource|:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row (tag=0)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:370)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:292)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:249)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:148)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:137)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:362)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:179)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable$1.run(TezTaskRunner.java:171)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1738)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:171)
> at
> org.apache.tez.runtime.task.TezTaskRunner$TaskRunnerCallable.callInternal(TezTaskRunner.java:167)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.RuntimeException: java.io.IOException: Exception reading
> file:/grid/4/tmp/yarn-local/usercache/.../appcache/application_1468602386465_489814/container_e02_1468602386465_489814_01_01/container_tokens
> at
> org.apache.hadoop.hive.ql.exec.persistence.RowContainer.first(RowContainer.java:237)
> at
> org.apache.hadoop.hive.ql.exec.persistence.RowContainer.first(RowContainer.java:74)
> at
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genUniqueJoinObject(CommonJoinOperator.java:650)
> at
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:756)
> at
> org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.joinObject(CommonMergeJoinOperator.java:316)
> at
> org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.joinOneGroup(CommonMergeJoinOperator.java:279)
> at
> org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.joinOneGroup(CommonMergeJoinOperator.java:272)
> at
> org.apache.hadoop.hive.ql.exec.CommonMergeJoinOperator.process(CommonMergeJoinOperator.java:258)
> at
> org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:361)
> ... 17 more
> Caused by: java.lang.RuntimeException: java.io.IOException: Exception reading
> file:/grid/4/tmp/yarn-local/usercache/.../appcache/application_1468602386465_489814/container_e02_1468602386465_489814_01_01/container_tokens
> at
> org.apache.hadoop.mapreduce.security.TokenCache.mergeBinaryTokens(TokenCache.java:141)
> at
> org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:119)
> at
> org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(TokenCache.java:100)
> at
> org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(TokenCache.java:80)
> at
> org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:206)
> at
> org.apache.
[jira] [Commented] (HIVE-14317) Make the print of COLUMN_STATS_ACCURATE more stable.
[
https://issues.apache.org/jira/browse/HIVE-14317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15394372#comment-15394372
]
Sergey Shelukhin commented on HIVE-14317:
-
Seems like lots of out files need to be updated, or the format needs to be
changed to be consistent... lots of braces changed.
> Make the print of COLUMN_STATS_ACCURATE more stable.
>
>
> Key: HIVE-14317
> URL: https://issues.apache.org/jira/browse/HIVE-14317
> Project: Hive
> Issue Type: Bug
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-14317.01.patch, HIVE-14317.02.patch
>
>
> based on different versions, we may have COLUMN_STATS_ACCURATE
> {"BASIC_STATS":"true","COLUMN_STATS":{"key":"true","value":"true"}} or
> COLUMN_STATS_ACCURATE
> {"COLUMN_STATS":{"key":"true","value":"true"},"BASIC_STATS":"true"}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14324) ORC PPD for floats is broken
[
https://issues.apache.org/jira/browse/HIVE-14324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15394374#comment-15394374
]
Prasanth Jayachandran commented on HIVE-14324:
--
Test failures doesn't look related to me.
> ORC PPD for floats is broken
>
>
> Key: HIVE-14324
> URL: https://issues.apache.org/jira/browse/HIVE-14324
> Project: Hive
> Issue Type: Bug
> Components: ORC
>Affects Versions: 1.3.0, 2.0.0, 2.1.0, 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Critical
> Attachments: HIVE-14324.1.patch, HIVE-14324.2.patch
>
>
> ORC stores min/max stats, bloom filters by passing floats as doubles using
> java's widening conversion. So if we write a float value of 0.22 to ORC file,
> the min/max stats and bloom filter will use 0.219988079071 double value.
> But when we do PPD, SARG creates literals by converting float to string and
> then to double which compares 0.22 to 0.219988079071 and fails PPD
> evaluation.
> {code}
> hive> create table orc_float (f float) stored as orc;
> hive> insert into table orc_float values(0.22);
> hive> set hive.optimize.index.filter=true;
> hive> select * from orc_float where f=0.22;
> OK
> hive> set hive.optimize.index.filter=false;
> hive> select * from orc_float where f=0.22;
> OK
> 0.22
> {code}
> This is not a problem for doubles and decimals.
> This issue was introduced in HIVE-8460 but back then there was no strict type
> check when SARGs are created and also PPD evaluation does not convert to
> column type. But now predicate leaf creation in SARG enforces strict type
> check for boxed literals and predicate type and PPD evaluation converts stats
> and constants to column type (predicate).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14324) ORC PPD for floats is broken
[
https://issues.apache.org/jira/browse/HIVE-14324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-14324:
-
Resolution: Fixed
Fix Version/s: 2.1.1
2.2.0
Status: Resolved (was: Patch Available)
Committed to branch-2.1 and master
> ORC PPD for floats is broken
>
>
> Key: HIVE-14324
> URL: https://issues.apache.org/jira/browse/HIVE-14324
> Project: Hive
> Issue Type: Bug
> Components: ORC
>Affects Versions: 1.3.0, 2.0.0, 2.1.0, 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Critical
> Fix For: 2.2.0, 2.1.1
>
> Attachments: HIVE-14324.1.patch, HIVE-14324.2.patch
>
>
> ORC stores min/max stats, bloom filters by passing floats as doubles using
> java's widening conversion. So if we write a float value of 0.22 to ORC file,
> the min/max stats and bloom filter will use 0.219988079071 double value.
> But when we do PPD, SARG creates literals by converting float to string and
> then to double which compares 0.22 to 0.219988079071 and fails PPD
> evaluation.
> {code}
> hive> create table orc_float (f float) stored as orc;
> hive> insert into table orc_float values(0.22);
> hive> set hive.optimize.index.filter=true;
> hive> select * from orc_float where f=0.22;
> OK
> hive> set hive.optimize.index.filter=false;
> hive> select * from orc_float where f=0.22;
> OK
> 0.22
> {code}
> This is not a problem for doubles and decimals.
> This issue was introduced in HIVE-8460 but back then there was no strict type
> check when SARGs are created and also PPD evaluation does not convert to
> column type. But now predicate leaf creation in SARG enforces strict type
> check for boxed literals and predicate type and PPD evaluation converts stats
> and constants to column type (predicate).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-10022) Authorization checks for non existent file/directory should not be recursive
[
https://issues.apache.org/jira/browse/HIVE-10022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15394408#comment-15394408
]
Hive QA commented on HIVE-10022:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12820068/HIVE-10022.9.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 11 failed/errored test(s), 10365 tests
executed
*Failed tests:*
{noformat}
TestMsgBusConnection - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.llap.daemon.impl.TestLlapTokenChecker.testCheckPermissions
org.apache.hadoop.hive.llap.daemon.impl.TestLlapTokenChecker.testGetToken
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.testConnections
org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler.org.apache.hive.hcatalog.hbase.TestPigHBaseStorageHandler
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/650/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/650/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-650/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 11 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12820068 - PreCommit-HIVE-MASTER-Build
> Authorization checks for non existent file/directory should not be recursive
>
>
> Key: HIVE-10022
> URL: https://issues.apache.org/jira/browse/HIVE-10022
> Project: Hive
> Issue Type: Bug
> Components: Authorization
>Affects Versions: 0.14.0
>Reporter: Pankit Thapar
>Assignee: Sushanth Sowmyan
> Attachments: HIVE-10022.2.patch, HIVE-10022.3.patch,
> HIVE-10022.4.patch, HIVE-10022.5.patch, HIVE-10022.6.patch,
> HIVE-10022.7.patch, HIVE-10022.8.patch, HIVE-10022.9.patch, HIVE-10022.patch
>
>
> I am testing a query like :
> set hive.test.authz.sstd.hs2.mode=true;
> set
> hive.security.authorization.manager=org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizerFactoryForTest;
> set
> hive.security.authenticator.manager=org.apache.hadoop.hive.ql.security.SessionStateConfigUserAuthenticator;
> set hive.security.authorization.enabled=true;
> set user.name=user1;
> create table auth_noupd(i int) clustered by (i) into 2 buckets stored as orc
> location '${OUTPUT}' TBLPROPERTIES ('transactional'='true');
> Now, in the above query, since authorization is true,
> we would end up calling doAuthorizationV2() which ultimately ends up calling
> SQLAuthorizationUtils.getPrivilegesFromFS() which calls a recursive method :
> FileUtils.isActionPermittedForFileHierarchy() with the object or the ancestor
> of the object we are trying to authorize if the object does not exist.
> The logic in FileUtils.isActionPermittedForFileHierarchy() is DFS.
> Now assume, we have a path as a/b/c/d that we are trying to authorize.
> In case, a/b/c/d does not exist, we would call
> FileUtils.isActionPermittedForFileHierarchy() with say a/b/ assuming a/b/c
> also does not exist.
> If under the subtree at a/b, we have millions of files, then
> FileUtils.isActionPermittedForFileHierarchy() is going to check file
> permission on each of those objects.
> I do not completely understand why do we have to check for file permissions
> in all the objects in branch of the tree that we are not trying to read
> from /write to.
> We could have checked file permission on the ancestor that exists and if it
> matches what we expect, the return true.
> Please confirm if this is a bug so that I can submit a patch else let me know
> what I am missing ?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-7239) Fix bug in HiveIndexedInputFormat implementation that causes incorrect query result when input backed by Sequence/RC files
[
https://issues.apache.org/jira/browse/HIVE-7239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15394412#comment-15394412
]
Hive QA commented on HIVE-7239:
---
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12820032/HIVE-7239.3.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/651/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/651/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-651/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ [[ -n /usr/java/jdk1.8.0_25 ]]
+ export JAVA_HOME=/usr/java/jdk1.8.0_25
+ JAVA_HOME=/usr/java/jdk1.8.0_25
+ export
PATH=/usr/java/jdk1.8.0_25/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/java/jdk1.8.0_25/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-MASTER-Build-651/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
dcfc31b..87bc121 master -> origin/master
5015e20..d87cb86 branch-2.1 -> origin/branch-2.1
+ git reset --hard HEAD
HEAD is now at dcfc31b HIVE-14123: Add beeline configuration option to show
database in the prompt (Peter Vary, reviewed by Aihua Xu)
+ git clean -f -d
Removing b/
Removing common/src/java/org/apache/hadoop/hive/common/FileUtils.java.orig
Removing ql/src/test/queries/clientpositive/authorization_load.q
Removing
ql/src/test/queries/clientpositive/exim_25_export_parentpath_has_inaccessible_children.q
Removing ql/src/test/results/clientpositive/authorization_load.q.out
Removing
ql/src/test/results/clientpositive/exim_25_export_parentpath_has_inaccessible_children.q.out
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 2 commits, and can be fast-forwarded.
(use "git pull" to update your local branch)
+ git reset --hard origin/master
HEAD is now at 87bc121 HIVE-14324: ORC PPD for floats is broken (Prasanth
Jayachandran reviewed by Gopal V)
+ git merge --ff-only origin/master
Already up-to-date.
+ git gc
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12820032 - PreCommit-HIVE-MASTER-Build
> Fix bug in HiveIndexedInputFormat implementation that causes incorrect query
> result when input backed by Sequence/RC files
> --
>
> Key: HIVE-7239
> URL: https://issues.apache.org/jira/browse/HIVE-7239
> Project: Hive
> Issue Type: Bug
> Components: Indexing
>Affects Versions: 2.1.0
>Reporter: Sumit Kumar
>Assignee: Illya Yalovyy
> Attachments: HIVE-7239.2.patch, HIVE-7239.3.patch, HIVE-7239.patch
>
>
> In case of sequence files, it's crucial that splits are calculated around the
> boundaries enforced by the input sequence file. However by default hadoop
> creates input splits depending on the configuration parameters which may not
> match the boundaries for the input sequence file. Hive provides
> HiveIndexedInputFormat that provides extra logic and recalculates the split
> boundaries for each split depending on the sequence file's boundaries.
> However we noticed this behavior of "over" reporting from data backed by
> sequence file. We've a sample data on which we experimented and fixed this
> bug, we have verified this fix by comparing the query output for input being
> sequence file format, rc file and regular format. However we have
[jira] [Updated] (HIVE-14233) Improve vectorization for ACID by eliminating row-by-row stitching
[ https://issues.apache.org/jira/browse/HIVE-14233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saket Saurabh updated HIVE-14233: - Attachment: HIVE-14233.04.patch Fix a NullPointerException bug that was being thrown when vectorized row batches were being used across subsequent nextBatch() calls. > Improve vectorization for ACID by eliminating row-by-row stitching > -- > > Key: HIVE-14233 > URL: https://issues.apache.org/jira/browse/HIVE-14233 > Project: Hive > Issue Type: New Feature > Components: Transactions, Vectorization >Reporter: Saket Saurabh >Assignee: Saket Saurabh > Attachments: HIVE-14233.01.patch, HIVE-14233.02.patch, > HIVE-14233.03.patch, HIVE-14233.04.patch > > > This JIRA proposes to improve vectorization for ACID by eliminating > row-by-row stitching when reading back ACID files. In the current > implementation, a vectorized row batch is created by populating the batch one > row at a time, before the vectorized batch is passed up along the operator > pipeline. This row-by-row stitching limitation was because of the fact that > the ACID insert/update/delete events from various delta files needed to be > merged together before the actual version of a given row was found out. > HIVE-14035 has enabled us to break away from that limitation by splitting > ACID update events into a combination of delete+insert. In fact, it has now > enabled us to create splits on delta files. > Building on top of HIVE-14035, this JIRA proposes to solve this earlier > bottleneck in the vectorized code path for ACID by now directly reading row > batches from the underlying ORC files and avoiding any stitching altogether. > Once a row batch is read from the split (which may be on a base/delta file), > the deleted rows will be found by cross-referencing them against a data > structure that will just keep track of deleted events (found in the > deleted_delta files). This will lead to a large performance gain when reading > ACID files in vectorized fashion, while enabling further optimizations in > future that can be done on top of that. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14316) TestLlapTokenChecker.testCheckPermissions, testGetToken fail
[ https://issues.apache.org/jira/browse/HIVE-14316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-14316: Resolution: Fixed Fix Version/s: 2.1.1 2.2.0 Status: Resolved (was: Patch Available) Committed. Thanks for the review! > TestLlapTokenChecker.testCheckPermissions, testGetToken fail > > > Key: HIVE-14316 > URL: https://issues.apache.org/jira/browse/HIVE-14316 > Project: Hive > Issue Type: Test >Reporter: Siddharth Seth >Assignee: Sergey Shelukhin > Fix For: 2.2.0, 2.1.1 > > Attachments: HIVE-14316.patch > > > cc [~sershe] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-10022) Authorization checks for non existent file/directory should not be recursive
[
https://issues.apache.org/jira/browse/HIVE-10022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15394522#comment-15394522
]
Sushanth Sowmyan commented on HIVE-10022:
-
None of the reported test failures are related to this patch. Based on Thejas'
prior +1, going ahead and committing.
> Authorization checks for non existent file/directory should not be recursive
>
>
> Key: HIVE-10022
> URL: https://issues.apache.org/jira/browse/HIVE-10022
> Project: Hive
> Issue Type: Bug
> Components: Authorization
>Affects Versions: 0.14.0
>Reporter: Pankit Thapar
>Assignee: Sushanth Sowmyan
> Attachments: HIVE-10022.2.patch, HIVE-10022.3.patch,
> HIVE-10022.4.patch, HIVE-10022.5.patch, HIVE-10022.6.patch,
> HIVE-10022.7.patch, HIVE-10022.8.patch, HIVE-10022.9.patch, HIVE-10022.patch
>
>
> I am testing a query like :
> set hive.test.authz.sstd.hs2.mode=true;
> set
> hive.security.authorization.manager=org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizerFactoryForTest;
> set
> hive.security.authenticator.manager=org.apache.hadoop.hive.ql.security.SessionStateConfigUserAuthenticator;
> set hive.security.authorization.enabled=true;
> set user.name=user1;
> create table auth_noupd(i int) clustered by (i) into 2 buckets stored as orc
> location '${OUTPUT}' TBLPROPERTIES ('transactional'='true');
> Now, in the above query, since authorization is true,
> we would end up calling doAuthorizationV2() which ultimately ends up calling
> SQLAuthorizationUtils.getPrivilegesFromFS() which calls a recursive method :
> FileUtils.isActionPermittedForFileHierarchy() with the object or the ancestor
> of the object we are trying to authorize if the object does not exist.
> The logic in FileUtils.isActionPermittedForFileHierarchy() is DFS.
> Now assume, we have a path as a/b/c/d that we are trying to authorize.
> In case, a/b/c/d does not exist, we would call
> FileUtils.isActionPermittedForFileHierarchy() with say a/b/ assuming a/b/c
> also does not exist.
> If under the subtree at a/b, we have millions of files, then
> FileUtils.isActionPermittedForFileHierarchy() is going to check file
> permission on each of those objects.
> I do not completely understand why do we have to check for file permissions
> in all the objects in branch of the tree that we are not trying to read
> from /write to.
> We could have checked file permission on the ancestor that exists and if it
> matches what we expect, the return true.
> Please confirm if this is a bug so that I can submit a patch else let me know
> what I am missing ?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12727) refactor Hive strict checks to be more granular, allow order by no limit and no partition filter by default for now
[
https://issues.apache.org/jira/browse/HIVE-12727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15394533#comment-15394533
]
Chao Sun commented on HIVE-12727:
-
Do you mean:
{code}
String mode = HiveConf.getVar(conf, ConfVars.HIVEMAPREDMODE, null);
{code}?
Yeah, I think that's a little strange. Inside {{HiveConf#initialize()}}, it
calls this method:
{code}
private static void applyDefaultNonNullConfVars(Configuration conf) {
for (ConfVars var : ConfVars.values()) {
String defaultValue = var.getDefaultValue();
if (defaultValue == null) {
// Don't override ConfVars with null values
continue;
}
conf.set(var.varname, defaultValue);
}
}
{code}
which initialize all conf vars whose default value are not null. So that code
doesn't work. I think we may have to change the default value to null.
> refactor Hive strict checks to be more granular, allow order by no limit and
> no partition filter by default for now
> ---
>
> Key: HIVE-12727
> URL: https://issues.apache.org/jira/browse/HIVE-12727
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Blocker
> Labels: TODOC2.0
> Fix For: 2.0.0
>
> Attachments: HIVE-12727.01.patch, HIVE-12727.02.patch,
> HIVE-12727.03.patch, HIVE-12727.04.patch, HIVE-12727.05.patch,
> HIVE-12727.06.patch, HIVE-12727.07.patch, HIVE-12727.patch
>
>
> Making strict mode the default recently appears to have broken many normal
> queries, such as some TPCDS benchmark queries, e.g. Q85:
> Response message: org.apache.hive.service.cli.HiveSQLException: Error while
> compiling statement: FAILED: SemanticException [Error 10041]: No partition
> predicate found for Alias "web_sales" Table "web_returns"
> We should remove this restriction from strict mode, or change the default
> back to non-strict. Perhaps make a 3-value parameter, nonstrict, semistrict,
> and strict, for backward compat for people who are relying on strict already.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-10022) Authorization checks for non existent file/directory should not be recursive
[
https://issues.apache.org/jira/browse/HIVE-10022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sushanth Sowmyan updated HIVE-10022:
Resolution: Fixed
Fix Version/s: 2.1.1
2.2.0
Status: Resolved (was: Patch Available)
Committed to branch-2.1 and master. Thanks [~pankit], for the issue report and
original patch, [~niklaus.xiao] & [~lirui] for the regex usecase and [~thejas]
for the reviews!
> Authorization checks for non existent file/directory should not be recursive
>
>
> Key: HIVE-10022
> URL: https://issues.apache.org/jira/browse/HIVE-10022
> Project: Hive
> Issue Type: Bug
> Components: Authorization
>Affects Versions: 0.14.0
>Reporter: Pankit Thapar
>Assignee: Sushanth Sowmyan
> Fix For: 2.2.0, 2.1.1
>
> Attachments: HIVE-10022.2.patch, HIVE-10022.3.patch,
> HIVE-10022.4.patch, HIVE-10022.5.patch, HIVE-10022.6.patch,
> HIVE-10022.7.patch, HIVE-10022.8.patch, HIVE-10022.9.patch, HIVE-10022.patch
>
>
> I am testing a query like :
> set hive.test.authz.sstd.hs2.mode=true;
> set
> hive.security.authorization.manager=org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizerFactoryForTest;
> set
> hive.security.authenticator.manager=org.apache.hadoop.hive.ql.security.SessionStateConfigUserAuthenticator;
> set hive.security.authorization.enabled=true;
> set user.name=user1;
> create table auth_noupd(i int) clustered by (i) into 2 buckets stored as orc
> location '${OUTPUT}' TBLPROPERTIES ('transactional'='true');
> Now, in the above query, since authorization is true,
> we would end up calling doAuthorizationV2() which ultimately ends up calling
> SQLAuthorizationUtils.getPrivilegesFromFS() which calls a recursive method :
> FileUtils.isActionPermittedForFileHierarchy() with the object or the ancestor
> of the object we are trying to authorize if the object does not exist.
> The logic in FileUtils.isActionPermittedForFileHierarchy() is DFS.
> Now assume, we have a path as a/b/c/d that we are trying to authorize.
> In case, a/b/c/d does not exist, we would call
> FileUtils.isActionPermittedForFileHierarchy() with say a/b/ assuming a/b/c
> also does not exist.
> If under the subtree at a/b, we have millions of files, then
> FileUtils.isActionPermittedForFileHierarchy() is going to check file
> permission on each of those objects.
> I do not completely understand why do we have to check for file permissions
> in all the objects in branch of the tree that we are not trying to read
> from /write to.
> We could have checked file permission on the ancestor that exists and if it
> matches what we expect, the return true.
> Please confirm if this is a bug so that I can submit a patch else let me know
> what I am missing ?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12727) refactor Hive strict checks to be more granular, allow order by no limit and no partition filter by default for now
[ https://issues.apache.org/jira/browse/HIVE-12727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15394544#comment-15394544 ] Sergey Shelukhin commented on HIVE-12727: - Yeah, we found it recently, I think there's a JIRA to remove that. Meanwhile it makes sense to change to null > refactor Hive strict checks to be more granular, allow order by no limit and > no partition filter by default for now > --- > > Key: HIVE-12727 > URL: https://issues.apache.org/jira/browse/HIVE-12727 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Blocker > Labels: TODOC2.0 > Fix For: 2.0.0 > > Attachments: HIVE-12727.01.patch, HIVE-12727.02.patch, > HIVE-12727.03.patch, HIVE-12727.04.patch, HIVE-12727.05.patch, > HIVE-12727.06.patch, HIVE-12727.07.patch, HIVE-12727.patch > > > Making strict mode the default recently appears to have broken many normal > queries, such as some TPCDS benchmark queries, e.g. Q85: > Response message: org.apache.hive.service.cli.HiveSQLException: Error while > compiling statement: FAILED: SemanticException [Error 10041]: No partition > predicate found for Alias "web_sales" Table "web_returns" > We should remove this restriction from strict mode, or change the default > back to non-strict. Perhaps make a 3-value parameter, nonstrict, semistrict, > and strict, for backward compat for people who are relying on strict already. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12727) refactor Hive strict checks to be more granular, allow order by no limit and no partition filter by default for now
[ https://issues.apache.org/jira/browse/HIVE-12727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15394548#comment-15394548 ] Chao Sun commented on HIVE-12727: - Cool. Do you know the JIRA number for that? I'll create a follow up JIRA for this to change it to null. Thanks. > refactor Hive strict checks to be more granular, allow order by no limit and > no partition filter by default for now > --- > > Key: HIVE-12727 > URL: https://issues.apache.org/jira/browse/HIVE-12727 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Blocker > Labels: TODOC2.0 > Fix For: 2.0.0 > > Attachments: HIVE-12727.01.patch, HIVE-12727.02.patch, > HIVE-12727.03.patch, HIVE-12727.04.patch, HIVE-12727.05.patch, > HIVE-12727.06.patch, HIVE-12727.07.patch, HIVE-12727.patch > > > Making strict mode the default recently appears to have broken many normal > queries, such as some TPCDS benchmark queries, e.g. Q85: > Response message: org.apache.hive.service.cli.HiveSQLException: Error while > compiling statement: FAILED: SemanticException [Error 10041]: No partition > predicate found for Alias "web_sales" Table "web_returns" > We should remove this restriction from strict mode, or change the default > back to non-strict. Perhaps make a 3-value parameter, nonstrict, semistrict, > and strict, for backward compat for people who are relying on strict already. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14346) Change the default value for hive.mapred.mode to null
[
https://issues.apache.org/jira/browse/HIVE-14346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated HIVE-14346:
Attachment: HIVE-14346.0.patch
> Change the default value for hive.mapred.mode to null
> -
>
> Key: HIVE-14346
> URL: https://issues.apache.org/jira/browse/HIVE-14346
> Project: Hive
> Issue Type: Bug
> Components: Configuration
>Affects Versions: 2.2.0
>Reporter: Chao Sun
>Assignee: Chao Sun
> Attachments: HIVE-14346.0.patch
>
>
> HIVE-12727 introduces three new configurations to replace the existing
> {{hive.mapred.mode}}, which is deprecated. However, the default value for the
> latter is 'nonstrict', which prevent the new configurations from being used
> (see comments in that JIRA for more details).
> This proposes to change the default value for {{hive.mapred.mode}} to null.
> Users can then set the three new configurations to get more fine-grained
> control over the strict checking. If user want to use the old configuration,
> they can set {{hive.mapred.mode}} to strict/nonstrict.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14346) Change the default value for hive.mapred.mode to null
[
https://issues.apache.org/jira/browse/HIVE-14346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chao Sun updated HIVE-14346:
Status: Patch Available (was: Open)
> Change the default value for hive.mapred.mode to null
> -
>
> Key: HIVE-14346
> URL: https://issues.apache.org/jira/browse/HIVE-14346
> Project: Hive
> Issue Type: Bug
> Components: Configuration
>Affects Versions: 2.2.0
>Reporter: Chao Sun
>Assignee: Chao Sun
> Attachments: HIVE-14346.0.patch
>
>
> HIVE-12727 introduces three new configurations to replace the existing
> {{hive.mapred.mode}}, which is deprecated. However, the default value for the
> latter is 'nonstrict', which prevent the new configurations from being used
> (see comments in that JIRA for more details).
> This proposes to change the default value for {{hive.mapred.mode}} to null.
> Users can then set the three new configurations to get more fine-grained
> control over the strict checking. If user want to use the old configuration,
> they can set {{hive.mapred.mode}} to strict/nonstrict.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14346) Change the default value for hive.mapred.mode to null
[
https://issues.apache.org/jira/browse/HIVE-14346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15394567#comment-15394567
]
Chao Sun commented on HIVE-14346:
-
cc [~sershe], [~xuefuz], [~leftylev].
> Change the default value for hive.mapred.mode to null
> -
>
> Key: HIVE-14346
> URL: https://issues.apache.org/jira/browse/HIVE-14346
> Project: Hive
> Issue Type: Bug
> Components: Configuration
>Affects Versions: 2.2.0
>Reporter: Chao Sun
>Assignee: Chao Sun
> Attachments: HIVE-14346.0.patch
>
>
> HIVE-12727 introduces three new configurations to replace the existing
> {{hive.mapred.mode}}, which is deprecated. However, the default value for the
> latter is 'nonstrict', which prevent the new configurations from being used
> (see comments in that JIRA for more details).
> This proposes to change the default value for {{hive.mapred.mode}} to null.
> Users can then set the three new configurations to get more fine-grained
> control over the strict checking. If user want to use the old configuration,
> they can set {{hive.mapred.mode}} to strict/nonstrict.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14346) Change the default value for hive.mapred.mode to null
[
https://issues.apache.org/jira/browse/HIVE-14346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15394566#comment-15394566
]
Xuefu Zhang commented on HIVE-14346:
+1 pending on test.
> Change the default value for hive.mapred.mode to null
> -
>
> Key: HIVE-14346
> URL: https://issues.apache.org/jira/browse/HIVE-14346
> Project: Hive
> Issue Type: Bug
> Components: Configuration
>Affects Versions: 2.2.0
>Reporter: Chao Sun
>Assignee: Chao Sun
> Attachments: HIVE-14346.0.patch
>
>
> HIVE-12727 introduces three new configurations to replace the existing
> {{hive.mapred.mode}}, which is deprecated. However, the default value for the
> latter is 'nonstrict', which prevent the new configurations from being used
> (see comments in that JIRA for more details).
> This proposes to change the default value for {{hive.mapred.mode}} to null.
> Users can then set the three new configurations to get more fine-grained
> control over the strict checking. If user want to use the old configuration,
> they can set {{hive.mapred.mode}} to strict/nonstrict.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14346) Change the default value for hive.mapred.mode to null
[
https://issues.apache.org/jira/browse/HIVE-14346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15394583#comment-15394583
]
Sergey Shelukhin commented on HIVE-14346:
-
+1; might break Q files. Is it accessed anywhere else without the default where
it might break?
> Change the default value for hive.mapred.mode to null
> -
>
> Key: HIVE-14346
> URL: https://issues.apache.org/jira/browse/HIVE-14346
> Project: Hive
> Issue Type: Bug
> Components: Configuration
>Affects Versions: 2.2.0
>Reporter: Chao Sun
>Assignee: Chao Sun
> Attachments: HIVE-14346.0.patch
>
>
> HIVE-12727 introduces three new configurations to replace the existing
> {{hive.mapred.mode}}, which is deprecated. However, the default value for the
> latter is 'nonstrict', which prevent the new configurations from being used
> (see comments in that JIRA for more details).
> This proposes to change the default value for {{hive.mapred.mode}} to null.
> Users can then set the three new configurations to get more fine-grained
> control over the strict checking. If user want to use the old configuration,
> they can set {{hive.mapred.mode}} to strict/nonstrict.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13688) Variable substitutions not working for shell commands in beeline
[
https://issues.apache.org/jira/browse/HIVE-13688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15394588#comment-15394588
]
Xuefu Zhang commented on HIVE-13688:
There are 4 kinds of variables, including system variables and env variables.
which can exist in both hosts and are not always defined by the user. Thus,
user query can potentially pick up the wrong value by substitution.
> Variable substitutions not working for shell commands in beeline
>
>
> Key: HIVE-13688
> URL: https://issues.apache.org/jira/browse/HIVE-13688
> Project: Hive
> Issue Type: Bug
> Components: Beeline
>Affects Versions: 1.1.0
>Reporter: Naveen Gangam
>Assignee: Naveen Gangam
>
> Appears that the variable substitutions work fine for DML and DDL commands.
> > set hivevar:v1=databases;
> No rows affected (0.004 seconds)
> > show ${v1};
> ++--+
> | database_name |
> ++--+
> | default|
> ++--+
> 1 row selected (0.243 seconds)
> > set hivevar:v1=tmp;
> No rows affected (0.061 seconds)
> > create table ${v1} (col1 string);
> No rows affected (0.065 seconds)
> > describe ${v1};
> INFO : Compiling
> command(queryId=hive_20160504105151_cb6ab7f1-26ac-4733-80ed-13a5134ba2b5):
> describe tmp
> INFO : Executing
> command(queryId=hive_20160504105151_cb6ab7f1-26ac-4733-80ed-13a5134ba2b5):
> describe tmp
> +---++--+--+
> | col_name | data_type | comment |
> +---++--+--+
> | col1 | string | |
> +---++--+--+
> 1 row selected (0.121 seconds)
> But it does not appear to work for shell commands (introduced in HIVE-6791 )
> > !sh sudo -u hdfs hdfs dfs -mkdir /user/root/${v1}
> mkdir: `/user/root/${v1}': No such file or directory <== problem
> Command failed with exit code = 1
> > !sh echo ${v1};
> ${v1};
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HIVE-14322) Postgres db issues after Datanucleus 4.x upgrade
[
https://issues.apache.org/jira/browse/HIVE-14322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin reassigned HIVE-14322:
---
Assignee: Sergey Shelukhin (was: Thejas M Nair)
> Postgres db issues after Datanucleus 4.x upgrade
>
>
> Key: HIVE-14322
> URL: https://issues.apache.org/jira/browse/HIVE-14322
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.0.0, 2.1.0, 2.0.1
>Reporter: Thejas M Nair
>Assignee: Sergey Shelukhin
> Attachments: HIVE-14322.1.patch
>
>
> With the upgrade to datanucleus 4.x versions in HIVE-6113, hive does not
> work properly with postgres.
> The nullable fields in the database have string "NULL::character varying"
> instead of real NULL values. This causes various issues.
> One example is -
> {code}
> hive> create table t(i int);
> OK
> Time taken: 1.9 seconds
> hive> create view v as select * from t;
> OK
> Time taken: 0.542 seconds
> hive> select * from v;
> FAILED: SemanticException Unable to fetch table v.
> java.net.URISyntaxException: Relative path in absolute URI:
> NULL::character%20varying
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14336) Make usage of VectorUDFAdaptor configurable
[
https://issues.apache.org/jira/browse/HIVE-14336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-14336:
Attachment: (was: HIVE-14336.01.patch)
> Make usage of VectorUDFAdaptor configurable
> ---
>
> Key: HIVE-14336
> URL: https://issues.apache.org/jira/browse/HIVE-14336
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-14336.01.patch
>
>
> Add a Hive configuration variable:
> {code}
> hive.vectorized.adaptor.usage.mode = {none, chosen, all}
> {code}
> for configuring whether to attempt vectorization using the VectorUDFAdaptor.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14336) Make usage of VectorUDFAdaptor configurable
[
https://issues.apache.org/jira/browse/HIVE-14336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-14336:
Attachment: HIVE-14336.01.patch
> Make usage of VectorUDFAdaptor configurable
> ---
>
> Key: HIVE-14336
> URL: https://issues.apache.org/jira/browse/HIVE-14336
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-14336.01.patch
>
>
> Add a Hive configuration variable:
> {code}
> hive.vectorized.adaptor.usage.mode = {none, chosen, all}
> {code}
> for configuring whether to attempt vectorization using the VectorUDFAdaptor.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14339) Fix UT failure for acid_globallimit.q
[ https://issues.apache.org/jira/browse/HIVE-14339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15394604#comment-15394604 ] Wei Zheng commented on HIVE-14339: -- Per discussion with [~pxiong] the reason why we get different explain content for data estimates is due to file system difference. Since that specific test is was just to make sure the query won't fail, I will remove the explain statement to avoid future mismatches. Btw, I couldn't reproduce the mismatch locally on my Mac laptop. > Fix UT failure for acid_globallimit.q > - > > Key: HIVE-14339 > URL: https://issues.apache.org/jira/browse/HIVE-14339 > Project: Hive > Issue Type: Bug >Affects Versions: 1.3.0, 2.1.0, 2.2.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14346) Change the default value for hive.mapred.mode to null
[
https://issues.apache.org/jira/browse/HIVE-14346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15394610#comment-15394610
]
Chao Sun commented on HIVE-14346:
-
It might. There're a few places which set this to nonstrict, mostly are in
tests, but one in SemanticAnalyzer. I don't think this will break those.
> Change the default value for hive.mapred.mode to null
> -
>
> Key: HIVE-14346
> URL: https://issues.apache.org/jira/browse/HIVE-14346
> Project: Hive
> Issue Type: Bug
> Components: Configuration
>Affects Versions: 2.2.0
>Reporter: Chao Sun
>Assignee: Chao Sun
> Attachments: HIVE-14346.0.patch
>
>
> HIVE-12727 introduces three new configurations to replace the existing
> {{hive.mapred.mode}}, which is deprecated. However, the default value for the
> latter is 'nonstrict', which prevent the new configurations from being used
> (see comments in that JIRA for more details).
> This proposes to change the default value for {{hive.mapred.mode}} to null.
> Users can then set the three new configurations to get more fine-grained
> control over the strict checking. If user want to use the old configuration,
> they can set {{hive.mapred.mode}} to strict/nonstrict.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14330) fix LockHandle TxnHandler.acquireLock(String key) retry logic
[
https://issues.apache.org/jira/browse/HIVE-14330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15394609#comment-15394609
]
Hive QA commented on HIVE-14330:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12820042/HIVE-14330.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 10366 tests
executed
*Failed tests:*
{noformat}
TestMsgBusConnection - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.llap.daemon.impl.TestLlapTokenChecker.testCheckPermissions
org.apache.hadoop.hive.llap.daemon.impl.TestLlapTokenChecker.testGetToken
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testDelayedLocalityNodeCommErrorImmediateAllocation
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.testConnections
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/652/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/652/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-652/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12820042 - PreCommit-HIVE-MASTER-Build
> fix LockHandle TxnHandler.acquireLock(String key) retry logic
> -
>
> Key: HIVE-14330
> URL: https://issues.apache.org/jira/browse/HIVE-14330
> Project: Hive
> Issue Type: Bug
> Components: Metastore, Transactions
>Affects Versions: 1.3.0, 2.1.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Critical
> Attachments: HIVE-14330.patch
>
>
> stupid bug: return statement is missing. See patch.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14322) Postgres db issues after Datanucleus 4.x upgrade
[
https://issues.apache.org/jira/browse/HIVE-14322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-14322:
Attachment: HIVE-14322.02.patch
[~thejas] the patch. I am not sure if we need to add this anywhere else... I
see configs in the source tree set autoCreateAll, but it seems like it should
work without such setting (and validateTables/etc. is not set)
> Postgres db issues after Datanucleus 4.x upgrade
>
>
> Key: HIVE-14322
> URL: https://issues.apache.org/jira/browse/HIVE-14322
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.0.0, 2.1.0, 2.0.1
>Reporter: Thejas M Nair
>Assignee: Sergey Shelukhin
> Attachments: HIVE-14322.02.patch, HIVE-14322.1.patch
>
>
> With the upgrade to datanucleus 4.x versions in HIVE-6113, hive does not
> work properly with postgres.
> The nullable fields in the database have string "NULL::character varying"
> instead of real NULL values. This causes various issues.
> One example is -
> {code}
> hive> create table t(i int);
> OK
> Time taken: 1.9 seconds
> hive> create view v as select * from t;
> OK
> Time taken: 0.542 seconds
> hive> select * from v;
> FAILED: SemanticException Unable to fetch table v.
> java.net.URISyntaxException: Relative path in absolute URI:
> NULL::character%20varying
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14333) ORC schema evolution from float to double changes precision and breaks filters
[
https://issues.apache.org/jira/browse/HIVE-14333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-14333:
-
Status: Patch Available (was: Open)
> ORC schema evolution from float to double changes precision and breaks filters
> --
>
> Key: HIVE-14333
> URL: https://issues.apache.org/jira/browse/HIVE-14333
> Project: Hive
> Issue Type: Bug
> Components: ORC
>Affects Versions: 2.1.0, 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Critical
> Attachments: HIVE-14333.1.patch
>
>
> ORC vs text schema evolution from float to double changes precision
> {code:title=Text Schema Evolution}
> hive> create table float_text(f float);
> hive> insert into float_text values(74.72);
> hive> select f from float_text;
> OK
> 74.72
> hive> alter table float_text change column f f double;
> hive> select f from float_text;
> OK
> 74.72
> {code}
> {code:title=Orc Schema Evolution}
> hive> create table float_orc(f float) stored as orc;
> hive> insert into float_orc values(74.72);
> hive> select f from float_orc;
> OK
> 74.72
> hive> alter table float_orc change column f f double;
> hive> select f from float_orc;
> OK
> 74.72000122070312
> {code}
> This will break all filters on the evolved column "f"
> {code:title=Filter returning no results}
> hive> set hive.optimize.index.filter=false;
> hive> select f from float_orc where f=74.72;
> OK
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14333) ORC schema evolution from float to double changes precision and breaks filters
[
https://issues.apache.org/jira/browse/HIVE-14333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran updated HIVE-14333:
-
Attachment: HIVE-14333.1.patch
> ORC schema evolution from float to double changes precision and breaks filters
> --
>
> Key: HIVE-14333
> URL: https://issues.apache.org/jira/browse/HIVE-14333
> Project: Hive
> Issue Type: Bug
> Components: ORC
>Affects Versions: 2.1.0, 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Critical
> Attachments: HIVE-14333.1.patch
>
>
> ORC vs text schema evolution from float to double changes precision
> {code:title=Text Schema Evolution}
> hive> create table float_text(f float);
> hive> insert into float_text values(74.72);
> hive> select f from float_text;
> OK
> 74.72
> hive> alter table float_text change column f f double;
> hive> select f from float_text;
> OK
> 74.72
> {code}
> {code:title=Orc Schema Evolution}
> hive> create table float_orc(f float) stored as orc;
> hive> insert into float_orc values(74.72);
> hive> select f from float_orc;
> OK
> 74.72
> hive> alter table float_orc change column f f double;
> hive> select f from float_orc;
> OK
> 74.72000122070312
> {code}
> This will break all filters on the evolved column "f"
> {code:title=Filter returning no results}
> hive> set hive.optimize.index.filter=false;
> hive> select f from float_orc where f=74.72;
> OK
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HIVE-14333) ORC schema evolution from float to double changes precision and breaks filters
[
https://issues.apache.org/jira/browse/HIVE-14333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prasanth Jayachandran reassigned HIVE-14333:
Assignee: Prasanth Jayachandran
> ORC schema evolution from float to double changes precision and breaks filters
> --
>
> Key: HIVE-14333
> URL: https://issues.apache.org/jira/browse/HIVE-14333
> Project: Hive
> Issue Type: Bug
> Components: ORC
>Affects Versions: 2.1.0, 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Critical
> Attachments: HIVE-14333.1.patch
>
>
> ORC vs text schema evolution from float to double changes precision
> {code:title=Text Schema Evolution}
> hive> create table float_text(f float);
> hive> insert into float_text values(74.72);
> hive> select f from float_text;
> OK
> 74.72
> hive> alter table float_text change column f f double;
> hive> select f from float_text;
> OK
> 74.72
> {code}
> {code:title=Orc Schema Evolution}
> hive> create table float_orc(f float) stored as orc;
> hive> insert into float_orc values(74.72);
> hive> select f from float_orc;
> OK
> 74.72
> hive> alter table float_orc change column f f double;
> hive> select f from float_orc;
> OK
> 74.72000122070312
> {code}
> This will break all filters on the evolved column "f"
> {code:title=Filter returning no results}
> hive> set hive.optimize.index.filter=false;
> hive> select f from float_orc where f=74.72;
> OK
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14336) Make usage of VectorUDFAdaptor configurable
[
https://issues.apache.org/jira/browse/HIVE-14336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15394631#comment-15394631
]
Sergey Shelukhin commented on HIVE-14336:
-
Some feedback on rb
> Make usage of VectorUDFAdaptor configurable
> ---
>
> Key: HIVE-14336
> URL: https://issues.apache.org/jira/browse/HIVE-14336
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-14336.01.patch
>
>
> Add a Hive configuration variable:
> {code}
> hive.vectorized.adaptor.usage.mode = {none, chosen, all}
> {code}
> for configuring whether to attempt vectorization using the VectorUDFAdaptor.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14339) Fix UT failure for acid_globallimit.q
[ https://issues.apache.org/jira/browse/HIVE-14339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-14339: - Attachment: HIVE-14339.patch [~pxiong] Can you please review? > Fix UT failure for acid_globallimit.q > - > > Key: HIVE-14339 > URL: https://issues.apache.org/jira/browse/HIVE-14339 > Project: Hive > Issue Type: Bug >Affects Versions: 1.3.0, 2.1.0, 2.2.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-14339.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14339) Fix UT failure for acid_globallimit.q
[ https://issues.apache.org/jira/browse/HIVE-14339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15394645#comment-15394645 ] Pengcheng Xiong commented on HIVE-14339: LGTM +!. [~wzheng], for test, you can directly commit. > Fix UT failure for acid_globallimit.q > - > > Key: HIVE-14339 > URL: https://issues.apache.org/jira/browse/HIVE-14339 > Project: Hive > Issue Type: Bug >Affects Versions: 1.3.0, 2.1.0, 2.2.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-14339.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14336) Make usage of VectorUDFAdaptor configurable
[
https://issues.apache.org/jira/browse/HIVE-14336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-14336:
Attachment: HIVE-14336.01.patch
> Make usage of VectorUDFAdaptor configurable
> ---
>
> Key: HIVE-14336
> URL: https://issues.apache.org/jira/browse/HIVE-14336
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-14336.01.patch
>
>
> Add a Hive configuration variable:
> {code}
> hive.vectorized.adaptor.usage.mode = {none, chosen, all}
> {code}
> for configuring whether to attempt vectorization using the VectorUDFAdaptor.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14339) Fix UT failure for acid_globallimit.q
[ https://issues.apache.org/jira/browse/HIVE-14339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zheng updated HIVE-14339: - Attachment: HIVE-14339.branch21.patch > Fix UT failure for acid_globallimit.q > - > > Key: HIVE-14339 > URL: https://issues.apache.org/jira/browse/HIVE-14339 > Project: Hive > Issue Type: Bug >Affects Versions: 1.3.0, 2.1.0, 2.2.0 >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-14339.branch21.patch, HIVE-14339.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14336) Make usage of VectorUDFAdaptor configurable
[
https://issues.apache.org/jira/browse/HIVE-14336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt McCline updated HIVE-14336:
Attachment: (was: HIVE-14336.01.patch)
> Make usage of VectorUDFAdaptor configurable
> ---
>
> Key: HIVE-14336
> URL: https://issues.apache.org/jira/browse/HIVE-14336
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-14336.01.patch
>
>
> Add a Hive configuration variable:
> {code}
> hive.vectorized.adaptor.usage.mode = {none, chosen, all}
> {code}
> for configuring whether to attempt vectorization using the VectorUDFAdaptor.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14336) Make usage of VectorUDFAdaptor configurable
[
https://issues.apache.org/jira/browse/HIVE-14336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15394671#comment-15394671
]
Matt McCline commented on HIVE-14336:
-
[~sershe] Thank you very much for the review comments. I posted a new diff.
> Make usage of VectorUDFAdaptor configurable
> ---
>
> Key: HIVE-14336
> URL: https://issues.apache.org/jira/browse/HIVE-14336
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-14336.01.patch
>
>
> Add a Hive configuration variable:
> {code}
> hive.vectorized.adaptor.usage.mode = {none, chosen, all}
> {code}
> for configuring whether to attempt vectorization using the VectorUDFAdaptor.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14052) Cleanup of structures required when LLAP access from external clients completes
[ https://issues.apache.org/jira/browse/HIVE-14052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Dere updated HIVE-14052: -- Status: Open (was: Patch Available) > Cleanup of structures required when LLAP access from external clients > completes > --- > > Key: HIVE-14052 > URL: https://issues.apache.org/jira/browse/HIVE-14052 > Project: Hive > Issue Type: Bug > Components: llap >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-14052.1.patch > > > Per [~sseth]: There's no cleanup at the moment, and structures used in LLAP > to track a query will keep building up slowly over time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13930) upgrade Hive to latest Hadoop version
[
https://issues.apache.org/jira/browse/HIVE-13930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15394675#comment-15394675
]
Sahil Takiar commented on HIVE-13930:
-
I downloaded the attached patch and applied it locally. I was able to get the
TestSparkCliDriver to work for at least one q test (input14). I'm not sure why
it is failing on the Pre-Commit build. Looking at the logs it seems to be
failing due to the same error {{NoClassDefFoundError:
org/apache/hadoop/crypto/key/KeyProvider}} as before. Is it possible the Spark
tar-ball is getting cached somewhere? I noticed that mvn clean doesn't actually
delete the Spark tar-ball. However, it does seem that each new Pre-Commit job
does a fresh git clone; although, I don't know enough about the Jenkins job to
be sure. I also don't have permission on the Jenkins machine to manually check.
Anyone with more access permissions have time to double check this?
> upgrade Hive to latest Hadoop version
> -
>
> Key: HIVE-13930
> URL: https://issues.apache.org/jira/browse/HIVE-13930
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-13930.01.patch, HIVE-13930.02.patch,
> HIVE-13930.03.patch, HIVE-13930.04.patch, HIVE-13930.05.patch,
> HIVE-13930.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14270) Write temporary data to HDFS when doing inserts on tables located on S3
[ https://issues.apache.org/jira/browse/HIVE-14270?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15394679#comment-15394679 ] Sergio Peña commented on HIVE-14270: [~poeppt] [~ste...@apache.org] I submitted another patch to the RB. I will skip the attachment here to avoid unit tests run for now. One question: - Isn't better that Hadoop can return a list of blobstore scheme it supports? I think this is better for 2 reasons: 1. Future versions of hadoop might add other blobstore scheme, and this way we will avoid changing Hive but just adding more test coverage. 2. Other non-hive components may want to get a list of currently supported blobstore from hadoop. I can add the configuration variable, but I was wondering about that. I'm still working on the testing part to run S3 tests. I'm still thinking whether to use q-test or write Junit code. Both have different complications. I uploaded the code to RB in the meantime so you can help me review. Btw, thanks for your help on reviewing it. > Write temporary data to HDFS when doing inserts on tables located on S3 > --- > > Key: HIVE-14270 > URL: https://issues.apache.org/jira/browse/HIVE-14270 > Project: Hive > Issue Type: Sub-task >Reporter: Sergio Peña >Assignee: Sergio Peña > Attachments: HIVE-14270.1.patch > > > Currently, when doing INSERT statements on tables located at S3, Hive writes > and reads temporary (or intermediate) files to S3 as well. > If HDFS is still the default filesystem on Hive, then we can keep such > temporary files on HDFS to keep things run faster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11943) Set old CLI as the default Client when using hive script
[
https://issues.apache.org/jira/browse/HIVE-11943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15394681#comment-15394681
]
Vihang Karajgaonkar commented on HIVE-11943:
Hi [~Ferd] I am trying to understand the patch for this issue. In the patch the
cli.sh is modified such that if USE_DEPRECATED_CLI is not set it is set to true
but in cli.cmd file it is set to true
{code}
+# Set old CLI as the default client
+if [ -n '$USE_DEPRECATED_CLI' ]; then
+ USE_DEPRECATED_CLI="true"
+fi
+
{code}
{code}
+ if [%USE_DEPRECATED_CLI%] == [] (
+ set USE_DEPRECATED_CLI=false
+ )
+
{code}
Is that intentional? or it should be set to true in cli.cmd also?
> Set old CLI as the default Client when using hive script
>
>
> Key: HIVE-11943
> URL: https://issues.apache.org/jira/browse/HIVE-11943
> Project: Hive
> Issue Type: Sub-task
> Components: CLI
>Affects Versions: beeline-cli-branch
>Reporter: Ferdinand Xu
>Assignee: Ferdinand Xu
> Fix For: beeline-cli-branch
>
> Attachments: HIVE-11943.1-beeline-cli.patch
>
>
> Since we have some concerns about deprecating the current CLI, we will set
> the old CLI as default. Once we resolve the problems, we will set the new CLI
> as default.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HIVE-11943) Set old CLI as the default Client when using hive script
[
https://issues.apache.org/jira/browse/HIVE-11943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15394681#comment-15394681
]
Vihang Karajgaonkar edited comment on HIVE-11943 at 7/26/16 10:31 PM:
--
Hi [~Ferd] I am trying to understand the patch for this issue. In the patch the
cli.sh is modified such that if USE_DEPRECATED_CLI is not set it is set to true
but in cli.cmd file it is set to false
{code}
+# Set old CLI as the default client
+if [ -n '$USE_DEPRECATED_CLI' ]; then
+ USE_DEPRECATED_CLI="true"
+fi
+
{code}
{code}
+ if [%USE_DEPRECATED_CLI%] == [] (
+ set USE_DEPRECATED_CLI=false
+ )
+
{code}
Is that intentional? or it should be set to true in cli.cmd also?
was (Author: vihangk1):
Hi [~Ferd] I am trying to understand the patch for this issue. In the patch the
cli.sh is modified such that if USE_DEPRECATED_CLI is not set it is set to true
but in cli.cmd file it is set to true
{code}
+# Set old CLI as the default client
+if [ -n '$USE_DEPRECATED_CLI' ]; then
+ USE_DEPRECATED_CLI="true"
+fi
+
{code}
{code}
+ if [%USE_DEPRECATED_CLI%] == [] (
+ set USE_DEPRECATED_CLI=false
+ )
+
{code}
Is that intentional? or it should be set to true in cli.cmd also?
> Set old CLI as the default Client when using hive script
>
>
> Key: HIVE-11943
> URL: https://issues.apache.org/jira/browse/HIVE-11943
> Project: Hive
> Issue Type: Sub-task
> Components: CLI
>Affects Versions: beeline-cli-branch
>Reporter: Ferdinand Xu
>Assignee: Ferdinand Xu
> Fix For: beeline-cli-branch
>
> Attachments: HIVE-11943.1-beeline-cli.patch
>
>
> Since we have some concerns about deprecating the current CLI, we will set
> the old CLI as default. Once we resolve the problems, we will set the new CLI
> as default.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14333) ORC schema evolution from float to double changes precision and breaks filters
[
https://issues.apache.org/jira/browse/HIVE-14333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15394682#comment-15394682
]
Matt McCline commented on HIVE-14333:
-
+1 LGTM tests pending.
> ORC schema evolution from float to double changes precision and breaks filters
> --
>
> Key: HIVE-14333
> URL: https://issues.apache.org/jira/browse/HIVE-14333
> Project: Hive
> Issue Type: Bug
> Components: ORC
>Affects Versions: 2.1.0, 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Prasanth Jayachandran
>Priority: Critical
> Attachments: HIVE-14333.1.patch
>
>
> ORC vs text schema evolution from float to double changes precision
> {code:title=Text Schema Evolution}
> hive> create table float_text(f float);
> hive> insert into float_text values(74.72);
> hive> select f from float_text;
> OK
> 74.72
> hive> alter table float_text change column f f double;
> hive> select f from float_text;
> OK
> 74.72
> {code}
> {code:title=Orc Schema Evolution}
> hive> create table float_orc(f float) stored as orc;
> hive> insert into float_orc values(74.72);
> hive> select f from float_orc;
> OK
> 74.72
> hive> alter table float_orc change column f f double;
> hive> select f from float_orc;
> OK
> 74.72000122070312
> {code}
> This will break all filters on the evolved column "f"
> {code:title=Filter returning no results}
> hive> set hive.optimize.index.filter=false;
> hive> select f from float_orc where f=74.72;
> OK
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13930) upgrade Hive to latest Hadoop version
[ https://issues.apache.org/jira/browse/HIVE-13930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15394689#comment-15394689 ] Sergio Peña commented on HIVE-13930: [~stakiar] What do you need to look on jenkins? > upgrade Hive to latest Hadoop version > - > > Key: HIVE-13930 > URL: https://issues.apache.org/jira/browse/HIVE-13930 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-13930.01.patch, HIVE-13930.02.patch, > HIVE-13930.03.patch, HIVE-13930.04.patch, HIVE-13930.05.patch, > HIVE-13930.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14330) fix LockHandle TxnHandler.acquireLock(String key) retry logic
[ https://issues.apache.org/jira/browse/HIVE-14330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-14330: -- Resolution: Fixed Fix Version/s: 2.1.1 2.2.0 1.3.0 Status: Resolved (was: Patch Available) Thanks Wei for the review Committed to branch-1, branch-2.1,master > fix LockHandle TxnHandler.acquireLock(String key) retry logic > - > > Key: HIVE-14330 > URL: https://issues.apache.org/jira/browse/HIVE-14330 > Project: Hive > Issue Type: Bug > Components: Metastore, Transactions >Affects Versions: 1.3.0, 2.1.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Critical > Fix For: 1.3.0, 2.2.0, 2.1.1 > > Attachments: HIVE-14330.patch > > > stupid bug: return statement is missing. See patch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14259) FileUtils.isSubDir may return incorrect result
[
https://issues.apache.org/jira/browse/HIVE-14259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-14259:
Attachment: HIVE-14259.2.patch
[~ashutoshc] [~lirui] yes, I was going to...at first it seemed that most of
these file logistic method group can be moved to movetask...but I realized it
wont be that simple...i'm still thinking on it...having them in the {{Hive}}
class is a bit unexpected to me.
In the end i've removed the original method - and used the more sophisticated
one - it's still a bit strange for me why a "following" statTask will make the
source file local...the old method threated the dest file also as being on the
same fs as the oldpath (not sure about the probability of this scenario) - but
anyway i think {{Hive.isSubDir}} will handle that case also correctly.
> FileUtils.isSubDir may return incorrect result
> --
>
> Key: HIVE-14259
> URL: https://issues.apache.org/jira/browse/HIVE-14259
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Minor
> Attachments: HIVE-14259.1.patch, HIVE-14259.2.patch
>
>
> while I was working on HIVE-12244 i've looked around for utility
> methods...i've found this method; but it considers path: `/dir12` inside
> `/dir1`
> which is not true
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
