Yang Jie created SPARK-58857:
--------------------------------
Summary: LocalDirsFeatureStep discards the result of
Utils.randomize, so configured local dirs are never shuffled
Key: SPARK-58857
URL: https://issues.apache.org/jira/browse/SPARK-58857
Project: Spark
Issue Type: Bug
Components: Kubernetes
Affects Versions: 5.0.0
Reporter: Yang Jie
`LocalDirsFeatureStep.configurePod` calls `Utils.randomize` on the local dirs
resolved from configuration but discards the result, so that branch never
randomizes.
{code}
val resolvedLocalDirs = Option(conf.sparkConf.getenv("SPARK_LOCAL_DIRS"))
.orElse(conf.getOption("spark.local.dir"))
.getOrElse(defaultLocalDir)
.split(",")
randomize(resolvedLocalDirs) // result dropped
localDirs = resolvedLocalDirs.toImmutableArraySeq // original order
{code}
`Utils.randomize[T](seq: IterableOnce[T]): Seq[T]` is
`randomizeInPlace(seq.iterator.toArray).toImmutableArraySeq` - it shuffles a
fresh copy and returns it, so the caller observes no change to the argument.
The in-place variant is `Utils.randomizeInPlace`. The sibling call fifteen
lines earlier, for the pod-template branch, binds the result correctly:
{code}
var localDirs = randomize(pod.container.getVolumeMounts.asScala
.filter(_.getName.startsWith("spark-local-dir-"))
.map(_.getMountPath))
{code}
Both calls arrived in the same commit, 9f9af2a7bf6 (SPARK-39755, "Improve
LocalDirsFeatureStep to randomize local directories"). An earlier revision of
that PR used `randomizeInPlace`; review suggested `Utils.randomize` instead,
and the next revision adopted it at both sites, binding the result at one and
dropping it at the other. During that review a committer twice noted the test
did not fail as expected, which is the symptom of this bug, but the discrepancy
was attributed to force-pushes and the assertions were loosened to accept
either order.
So `SPARK_LOCAL_DIRS`, the emptyDir volume list, and the `spark-local-dir-N` to
path pairing are emitted in configured order, identically for every driver and
executor pod in an application.
Scope of the observable effect, so this is not oversold: no measurable I/O or
capacity skew follows. `DiskBlockManager.getFile` picks a directory by
`nonNegativeHash(filename) % localDirs.length`, which spreads files
near-uniformly within each executor regardless of the order it receives, and in
this branch every path is backed by an emptyDir volume this step creates, so
all of them share one node filesystem (or RAM under
`spark.kubernetes.local.dirs.tmpfs`). What is real is that the randomization
SPARK-39755 added, and its JIRA and release note describe, does not happen on
this branch, and the suite's `// SPARK-39755 : Changes the method to test
randomization` comment describes behavior that never occurs.
Fix: bind the result, matching the sibling call. Deleting the call as dead code
is a defensible alternative if the randomization is not considered worth having.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
