[
https://issues.apache.org/jira/browse/SPARK-58057?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-58057:
----------------------------------
Fix Version/s: 4.2.0
(was: 4.2.1)
> Make Kubernetes VolumeSuite OnDemand PVC test robust to slow container startup
> ------------------------------------------------------------------------------
>
> Key: SPARK-58057
> URL: https://issues.apache.org/jira/browse/SPARK-58057
> Project: Spark
> Issue Type: Bug
> Components: Kubernetes, Tests
> Affects Versions: 5.0.0
> Reporter: Hyukjin Kwon
> Assignee: Hyukjin Kwon
> Priority: Major
> Labels: pull-request-available
> Fix For: 4.2.0, 4.3.0
>
>
> The Kubernetes integration test VolumeSuite "A driver-only Spark job with an
> OnDemand PVC volume" is flaky (observed on the scheduled build_java21
> workflow on branch-4.0).
> checkDisk() runs 'df <path>' inside the driver container via
> Utils.executeCommand, wrapped in eventually(...), but used a short fixed
> timeout of 10 seconds. For the OnDemand PVC cases, dynamic PVC provisioning
> and mounting can keep the driver container in ContainerCreating longer than
> that, so the in-pod command is not yet runnable and the assertion never
> passes within 10s. The same test passes on the in-run retry, which is
> characteristic of a too-tight readiness timeout rather than a product bug.
> In addition, logForFailedTest() fetches the driver pod log with .getLog
> without a guard. When a test fails while the driver pod is still in
> ContainerCreating, that .getLog throws KubernetesClientException (HTTP 400,
> "container ... is waiting to start: ContainerCreating"), which then surfaces
> as the reported failure and masks the actual cause. The executor-pod log
> fetch right below already wraps .getLog in a try/catch.
> Fix: use the shared TIMEOUT (3 minutes) for checkDisk, matching the other
> in-pod readiness waits, and apply the same try/catch guard to the driver-pod
> log fetch in logForFailedTest.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
