[
https://issues.apache.org/jira/browse/HIVE-24201?focusedWorklogId=590493&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-590493
]
ASF GitHub Bot logged work on HIVE-24201:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 28/Apr/21 16:16
Start Date: 28/Apr/21 16:16
Worklog Time Spent: 10m
Work Description: sankarh commented on a change in pull request #2065:
URL: https://github.com/apache/hive/pull/2065#discussion_r622334824
##########
File path: ql/src/java/org/apache/hadoop/hive/ql/exec/tez/WorkloadManager.java
##########
@@ -682,11 +685,8 @@ private void processCurrentEvents(EventState e,
WmThreadSyncWork syncWork) throw
// as possible
Map<WmTezSession, WmEvent> recordMoveEvents = new HashMap<>();
for (MoveSession moveSession : e.moveSessions) {
- if (HiveConf.getBoolVar(conf, ConfVars.HIVE_SERVER2_WM_DELAYED_MOVE)) {
- handleMoveSessionOnMasterThread(moveSession, syncWork,
poolsToRedistribute, e.toReuse, recordMoveEvents, true);
- } else {
- handleMoveSessionOnMasterThread(moveSession, syncWork,
poolsToRedistribute, e.toReuse, recordMoveEvents, false);
- }
+ handleMoveSessionOnMasterThread(moveSession, syncWork,
poolsToRedistribute, e.toReuse,
Review comment:
Shall store the HiveConf.getBoolVar(conf,
ConfVars.HIVE_SERVER2_WM_DELAYED_MOVE) in a boolean variable outside the "for"
loop and use it here.
##########
File path: common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
##########
@@ -3753,11 +3753,12 @@ private static void
populateLlapDaemonVarsSet(Set<String> llapDaemonVarsSetLocal
"Determines behavior of the wm move trigger when destination pool is
full.\n" +
"If true, the query will run in source pool as long as possible if
destination pool is full;\n" +
"if false, the query will be killed if destination pool is full."),
-
HIVE_SERVER2_WM_DELAYED_MOVE_TIMEOUT("hive.server2.wm.delayed.move.timeout",
"600",
+
HIVE_SERVER2_WM_DELAYED_MOVE_TIMEOUT("hive.server2.wm.delayed.move.timeout",
"3600",
new TimeValidator(TimeUnit.SECONDS),
"The amount of time a delayed move is allowed to run in the source
pool,\n" +
- "when a delayed move session times out, the session is moved to the
destination pool.\n"),
-
HIVE_SERVER2_WM_DELAYED_MOVE_VALIDATOR_INTERVAL("hive.server2.wm.delayed.move.validator.interval",
"10",
+ "when a delayed move session times out, the session is moved to the
destination pool.\n" +
+ "A value of 0 indicates no timeout"),
+
HIVE_SERVER2_WM_DELAYED_MOVE_VALIDATOR_INTERVAL("hive.server2.wm.delayed.move.validator.interval",
"60",
new TimeValidator(TimeUnit.SECONDS),
"Interval for checking for expired delayed moves and retries. Value of
0 indicates no checks."),
Review comment:
I think, we shouldn't allow 0 for interval. It creates confusion as we
set timeout but it doesn't work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 590493)
Time Spent: 1h 50m (was: 1h 40m)
> WorkloadManager can support delayed move if destination pool does not have
> enough sessions
> ------------------------------------------------------------------------------------------
>
> Key: HIVE-24201
> URL: https://issues.apache.org/jira/browse/HIVE-24201
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2, llap
> Affects Versions: 4.0.0
> Reporter: Adesh Kumar Rao
> Assignee: Pritha Dawn
> Priority: Minor
> Labels: pull-request-available
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> To reproduce, create a resource plan with move trigger, like below:
> {code:java}
> +----------------------------------------------------+
> | line |
> +----------------------------------------------------+
> | experiment[status=DISABLED,parallelism=null,defaultPool=default] |
> | + default[allocFraction=0.888,schedulingPolicy=null,parallelism=1] |
> | | mapped for default |
> | + pool2[allocFraction=0.1,schedulingPolicy=fair,parallelism=1] |
> | | trigger t1: if (ELAPSED_TIME > 20) { MOVE TO pool1 } |
> | | mapped for users: abcd |
> | + pool1[allocFraction=0.012,schedulingPolicy=null,parallelism=1] |
> | | mapped for users: efgh |
>
> {code}
> Now, run two queries in pool1 and pool2 using different users. The query
> running in pool2 will tried to move to pool1 and it will get killed because
> pool1 will not have session to handle the query.
> Currently, the Workload management move trigger kills the query being moved
> to a different pool if destination pool does not have enough capacity. We
> could have a "delayed move" configuration which lets the query run in the
> source pool as long as possible, if the destination pool is full. It will
> attempt the move to destination pool only when there is claim upon the source
> pool. If the destination pool is not full, delayed move behaves as normal
> move i.e. the move will happen immediately.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
