[
https://issues.apache.org/jira/browse/HIVE-21496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16801290#comment-16801290
]
Hive QA commented on HIVE-21496:
--------------------------------
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
33s{color} | {color:blue} ql in master has 2255 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 27m 17s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-16678/dev-support/hive-personality.sh
|
| git revision | master / 80998ad |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-16678/yetus.txt |
| Powered by | Apache Yetus http://yetus.apache.org |
This message was automatically generated.
> Automatic sizing of unordered buffer can overflow
> -------------------------------------------------
>
> Key: HIVE-21496
> URL: https://issues.apache.org/jira/browse/HIVE-21496
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer
> Affects Versions: 4.0.0
> Reporter: Prasanth Jayachandran
> Assignee: Jesus Camacho Rodriguez
> Priority: Major
> Attachments: HIVE-21496.01.patch, hive.log
>
>
> HIVE-21329 added automatic sizing of tez unordered partitioned KV buffer
> based on group by statistics. However, some corner cases for group by
> statistics sets Long.MAX for data size. This ends up setting Integer.MAX for
> unordered KV buffer size. This buffer size is expected to be in MB.
> Converting Integer.MAX value from MB to bytes will overflow and following
> exception is thrown.
> {code:java}
> 2019-03-23T01:35:17,760 INFO [Dispatcher thread {Central}]
> HistoryEventHandler.criticalEvents:
> [HISTORY][DAG:dag_1553330105749_0001_1][Event:TASK_ATTEMPT_FINISHED]:
> vertexName=Map 1, taskAttemptId=attempt_1553330105749_0001_1_00_000000_0,
> creationTime=1553330117468, allocationTime=1553330117524,
> startTime=1553330117562, finishTime=1553330117755, timeTaken=193,
> status=FAILED, taskFailureType=NON_FATAL, errorEnum=FRAMEWORK_ERROR,
> diagnostics=Error: Error while running task ( failure ) :
> attempt_1553330105749_0001_1_00_000000_0:java.lang.IllegalArgumentException
> at com.google.common.base.Preconditions.checkArgument(Preconditions.java:108)
> at
> org.apache.tez.runtime.common.resources.MemoryDistributor.registerRequest(MemoryDistributor.java:177)
> at
> org.apache.tez.runtime.common.resources.MemoryDistributor.requestMemory(MemoryDistributor.java:110)
> at
> org.apache.tez.runtime.api.impl.TezTaskContextImpl.requestInitialMemory(TezTaskContextImpl.java:214)
> at
> org.apache.tez.runtime.library.output.UnorderedPartitionedKVOutput.initialize(UnorderedPartitionedKVOutput.java:76)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask$InitializeOutputCallable._callInternal(LogicalIOProcessorRuntimeTask.java:537)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask$InitializeOutputCallable.callInternal(LogicalIOProcessorRuntimeTask.java:520)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask$InitializeOutputCallable.callInternal(LogicalIOProcessorRuntimeTask.java:505)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745){code}
>
> Stats for GBY operator is getting Long.MAX_VALUE as seen below
> {code:java}
> 2019-03-23T01:35:16,466 DEBUG [c779e956-b3b9-451a-8248-6ae7c669854f main]
> annotation.StatsRulesProcFactory: [0] STATS-TS[0] (logs): numRows: 1795
> dataSize: 4443078 basicStatsState: PARTIAL colStatsState: NONE colStats:
> {severity= colName: severity colType: string countDistincts: 359 numNulls: 89
> avgColLen: 100.0 numTrues: 0 numFalses: 0 isPrimaryKey: false isEstimated:
> true}
> 2019-03-23T01:35:16,466 DEBUG [c779e956-b3b9-451a-8248-6ae7c669854f main]
> annotation.StatsRulesProcFactory: Estimating row count for
> GenericUDFOPEqual(Column[severity], Const string ERROR) Original num rows:
> 1795 New num rows: 5
> 2019-03-23T01:35:16,467 DEBUG [c779e956-b3b9-451a-8248-6ae7c669854f main]
> annotation.StatsRulesProcFactory: [1] STATS-FIL[8]: numRows: 5 dataSize:
> 12376 basicStatsState: PARTIAL colStatsState: NONE colStats: {severity=
> colName: severity colType: string countDistincts: 359 numNulls: 89 avgColLen:
> 100.0 numTrues: 0 numFalses: 0 isPrimaryKey: false isEstimated: true}
> 2019-03-23T01:35:16,467 DEBUG [c779e956-b3b9-451a-8248-6ae7c669854f main]
> exec.FilterOperator: Setting stats (Num rows: 5 Data size: 12376 Basic stats:
> PARTIAL Column stats: NONE) on: FIL[8]
> 2019-03-23T01:35:16,468 DEBUG [c779e956-b3b9-451a-8248-6ae7c669854f main]
> exec.SelectOperator: Setting stats (Num rows: 5 Data size: 12376 Basic stats:
> PARTIAL Column stats: NONE) on: SEL[2]
> 2019-03-23T01:35:16,468 DEBUG [c779e956-b3b9-451a-8248-6ae7c669854f main]
> annotation.StatsRulesProcFactory: [1] STATS-SEL[2]: numRows: 5 dataSize:
> 12376 basicStatsState: PARTIAL colStatsState: NONE colStats: {severity=
> colName: severity colType: string countDistincts: 359 numNulls: 89 avgColLen:
> 100.0 numTrues: 0 numFalses: 0 isPrimaryKey: false isEstimated: true}
> 2019-03-23T01:35:16,471 DEBUG [c779e956-b3b9-451a-8248-6ae7c669854f main]
> annotation.StatsRulesProcFactory: STATS-GBY[3]: inputSize: 4443078
> maxSplitSize: 256000000 parallelism: 1 containsGroupingSet: false
> sizeOfGroupingSet: 1
> 2019-03-23T01:35:16,471 DEBUG [c779e956-b3b9-451a-8248-6ae7c669854f main]
> annotation.StatsRulesProcFactory: [Case 1] STATS-GBY[3]: cardinality: 5
> 2019-03-23T01:35:16,472 DEBUG [c779e956-b3b9-451a-8248-6ae7c669854f main]
> exec.GroupByOperator: Setting stats (Num rows: 1 Data size:
> 9223372036854775807 Basic stats: PARTIAL Column stats: NONE) on: GBY[3]
> 2019-03-23T01:35:16,472 DEBUG [c779e956-b3b9-451a-8248-6ae7c669854f main]
> annotation.StatsRulesProcFactory: [0] STATS-GBY[3]: numRows: 1 dataSize:
> 9223372036854775807 basicStatsState: PARTIAL colStatsState: NONE colStats:
> {severity= colName: severity colType: string countDistincts: 1 numNulls: 18
> avgColLen: 100.0 numTrues: 0 numFalses: 0 isPrimaryKey: false isEstimated:
> true, _col0= colName: _col0 colType: bigint countDistincts: 1 numNulls: 0
> avgColLen: 8.0 numTrues: 0 numFalses: 0 isPrimaryKey: false isEstimated:
> false}
> 2019-03-23T01:35:16,473 DEBUG [c779e956-b3b9-451a-8248-6ae7c669854f main]
> exec.ReduceSinkOperator: Setting stats (Num rows: 1 Data size:
> 9223372036854775807 Basic stats: PARTIAL Column stats: NONE) on: RS[4]
> 2019-03-23T01:35:16,474 DEBUG [c779e956-b3b9-451a-8248-6ae7c669854f main]
> annotation.StatsRulesProcFactory: [0] STATS-RS[4]: numRows: 1 dataSize:
> 9223372036854775807 basicStatsState: PARTIAL colStatsState: NONE colStats:
> {severity= colName: severity colType: string countDistincts: 1 numNulls: 18
> avgColLen: 100.0 numTrues: 0 numFalses: 0 isPrimaryKey: false isEstimated:
> true, _col0= colName: _col0 colType: bigint countDistincts: 1 numNulls: 0
> avgColLen: 8.0 numTrues: 0 numFalses: 0 isPrimaryKey: false isEstimated:
> false}
> 2019-03-23T01:35:16,474 DEBUG [c779e956-b3b9-451a-8248-6ae7c669854f main]
> annotation.StatsRulesProcFactory: STATS-GBY[5]: inputSize: 1 maxSplitSize:
> 256000000 parallelism: 1 containsGroupingSet: false sizeOfGroupingSet: 1
> 2019-03-23T01:35:16,474 DEBUG [c779e956-b3b9-451a-8248-6ae7c669854f main]
> annotation.StatsRulesProcFactory: [Case 7] STATS-GBY[5]: cardinality: 0
> 2019-03-23T01:35:16,474 DEBUG [c779e956-b3b9-451a-8248-6ae7c669854f main]
> stats.StatsUtils: STATS-GBY[5]: Equals 0 in number of rows. 0 rows will be
> set to 1
> 2019-03-23T01:35:16,474 DEBUG [c779e956-b3b9-451a-8248-6ae7c669854f main]
> exec.GroupByOperator: Setting stats (Num rows: 1 Data size:
> 9223372036854775807 Basic stats: PARTIAL Column stats: NONE) on: GBY[5]
> 2019-03-23T01:35:16,474 DEBUG [c779e956-b3b9-451a-8248-6ae7c669854f main]
> annotation.StatsRulesProcFactory: [0] STATS-GBY[5]: numRows: 1 dataSize:
> 9223372036854775807 basicStatsState: PARTIAL colStatsState: NONE colStats:
> {severity= colName: severity colType: string countDistincts: 1 numNulls: 18
> avgColLen: 100.0 numTrues: 0 numFalses: 0 isPrimaryKey: false isEstimated:
> true, _col0= colName: _col0 colType: bigint countDistincts: 1 numNulls: 0
> avgColLen: 8.0 numTrues: 0 numFalses: 0 isPrimaryKey: false isEstimated:
> false}
> 2019-03-23T01:35:16,474 DEBUG [c779e956-b3b9-451a-8248-6ae7c669854f main]
> annotation.StatsRulesProcFactory: [0] STATS-FS[7]: numRows: 1 dataSize:
> 9223372036854775807 basicStatsState: PARTIAL colStatsState: NONE colStats:
> {severity= colName: severity colType: string countDistincts: 1 numNulls: 36
> avgColLen: 100.0 numTrues: 0 numFalses: 0 isPrimaryKey: false isEstimated:
> true, _col0= colName: _col0 colType: bigint countDistincts: 1 numNulls: 0
> avgColLen: 8.0 numTrues: 0 numFalses: 0 isPrimaryKey: false isEstimated:
> false}{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
