[
https://issues.apache.org/jira/browse/HIVE-15221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-15221:
----------------------------------
Labels: pull-request-available (was: )
> Improvement for MapJoin checkMemoryStatus, adding gc before throwing Exception
> ------------------------------------------------------------------------------
>
> Key: HIVE-15221
> URL: https://issues.apache.org/jira/browse/HIVE-15221
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
> Affects Versions: 2.0.1, 2.1.0
> Reporter: Hui Fei
> Assignee: Hui Fei
> Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-15221.1.patch, stat_gc.png
>
>
> i see in the current master version
> {code:title=MapJoinMemoryExhaustionHandler.java|borderStyle=solid}
> public void checkMemoryStatus(long tableContainerSize, long numRows)
> throws MapJoinMemoryExhaustionException {
> long usedMemory = memoryMXBean.getHeapMemoryUsage().getUsed();
> double percentage = (double) usedMemory / (double) maxHeapSize;
> String msg = Utilities.now() + "\tProcessing rows:\t" + numRows +
> "\tHashtable size:\t"
> + tableContainerSize + "\tMemory usage:\t" + usedMemory +
> "\tpercentage:\t" + percentageNumberFormat.format(percentage);
> console.printInfo(msg);
> if(percentage > maxMemoryUsage) {
> throw new MapJoinMemoryExhaustionException(msg);
> }
> }
> {code}
> if {{percentage > maxMemoryUsage}}, then throw
> MapJoinMemoryExhaustionException
> in my opinion, running is better than fail. after System.gc, ' if percentage
> > maxMemoryUsage, then throw MapJoinMemoryExhaustionException' maybe better
> And original checking way has a problem: 1) consuming much memory cause gc
> (e.g young gc), then check after adding row and pass. 2) consuming much
> memory does not cause gc, then check after adding rows but throw Exception
> sometimes 2) occurs, but it contians less rows than 1).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
