Joe McDonnell created IMPALA-14784:
--------------------------------------
Summary: Switch end to end parallel tests to python-xdist's
--dist=worksteal
Key: IMPALA-14784
URL: https://issues.apache.org/jira/browse/IMPALA-14784
Project: IMPALA
Issue Type: Task
Components: Infrastructure
Affects Versions: Impala 5.0.0
Reporter: Joe McDonnell
Python-xdist added a "worksteal" distribution mode that can rebalance work
between the threads when a thread goes idle. This is mainly useful towards the
end of a parallel test run to reduce skew. On exhaustive end-to-end parallel
tests, the last 2% of tests take hours:
{noformat}
02:26:10
..............................xxx....................................... [ 98%]
03:38:48
........................................................................ [ 98%]
04:42:39
........................................................................ [ 99%]
05:08:04
........................................................................ [ 99%]
05:08:21 ....ss
[100%]{noformat}
Looking at the report log, there is massive skew. At the end, a single worker
is working alone on its remaining tests while other workers are idle. Work
stealing would make a big difference here.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
