John Zhuge created SPARK-56158:
----------------------------------
Summary: [CORE] Add ActiveProcessorCount JVM option support for
local mode
Key: SPARK-56158
URL: https://issues.apache.org/jira/browse/SPARK-56158
Project: Spark
Issue Type: New Feature
Components: Spark Core
Affects Versions: 4.2.0
Reporter: John Zhuge
SPARK-53209 added spark.driver.limitActiveProcessorCount.enabled to inject
-XX:ActiveProcessorCount into YARN JVM launch commands.
In local mode, Spark runs entirely in a single in-process JVM, so
-XX:ActiveProcessorCount cannot be injected at runtime. This ticket
investigates the right mechanism (e.g., auto-prepending to driver JVM options
at submit time via SparkSubmit) so that local mode users on machines with many
cores can benefit from the same CPU limiting behavior when each app only uses a
few cores.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
