[
https://issues.apache.org/jira/browse/HIVE-13934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15384574#comment-15384574
]
Gunther Hagleitner commented on HIVE-13934:
-------------------------------------------
Looked at it more closely. There are still a few things we should change.
a) Let's create a "ReservedMemoryMB" field in BaseWork. We'll use this for map
join now, but hopefully can extend that in future to include other memory
sensitive operations. Also let's not record the fraction but the actual memory
(in mb). Default should be -1, to indicate leaving it up to Tez.
b) Can you move the string function and the adjustment into DagUtils? It's not
really a string util function anyways, and we can convert just before we set
the Tez value. Once we have a proper Tez API we can nuke that stuff.
c) Recording the memory right in GenTezPlan is fragile. I think it would be
better if you just set the reserved memory for each mapjoin in TezCompiler.
There's a map (mj -> work), you can iterate through it and set the reserved
memory. (right before we handle the filesink and union operators for instance).
d) You have two new variables fraction and fraction_max. Can you make it min
and max? It would also be nice to have a third fraction (apart from min, max).
You can leave that one null by default. We can use it to overwrite the memory
requested from Tez. If it's set we use it for every task.
> Configure Tez to make nocondiional task size memory available for the
> Processor
> -------------------------------------------------------------------------------
>
> Key: HIVE-13934
> URL: https://issues.apache.org/jira/browse/HIVE-13934
> Project: Hive
> Issue Type: Bug
> Reporter: Wei Zheng
> Assignee: Wei Zheng
> Attachments: HIVE-13934.1.patch, HIVE-13934.2.patch,
> HIVE-13934.3.patch, HIVE-13934.4.patch, HIVE-13934.6.patch,
> HIVE-13934.7.patch, HIVE-13934.8.patch, HIVE-13934.9.patch
>
>
> Currently, noconditionaltasksize is not validated against the container size,
> the reservations made in the container by Tez for Inputs / Outputs etc.
> Check this at compile time to see if enough memory is available, or set up
> the vertex to reserve additional memory for the Processor.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
