Hi csmith,
Just be too late here. We just realize this bug recently. Here's the fix
https://github.com/apache/spark/pull/28257. And I believe we're going to
backport it into 2.4.x.
Best,
Yi Wu
--
Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/
---
I'm using barrier execution in my spark job but am occasionally seeing
deadlocks where the task scheduler is unable to place all the tasks. The
failure is logged but the job hangs indefinitely. I have 2 executors with 16
cores each, using standalone mode (I think? I'm using databricks). The
dataset
