Hi all,

I’d like to ask for a review and feedback on SPARK-58126, implemented in PR 
#57257.

The PR introduces an opt-in extension point for customizing the ordering of the 
scheduler’s root pool through a configurable 
java.util.Comparator[SchedulableInfo].

The motivation is to allow use cases such as priority-based, deadline-aware, or 
SLA-driven scheduling without requiring users to patch Spark itself.

The design has evolved based on the feedback received so far. The current 
approach deliberately keeps the API surface minimal and avoids exposing Spark’s 
internal, mutable scheduling implementation:

* A single new immutable @DeveloperApi type, SchedulableInfo, exposes the 
scheduling-relevant information needed by custom comparators.
* A new configuration property, spark.scheduler.rootPool.comparator.class, 
allows users to provide their own comparator.
* Existing scheduling behavior (FAIR/FIFO) remains unchanged unless the new 
configuration is explicitly set.
* Internal scheduler abstractions remain internal.

The implementation is complete and intentionally minimal: only six files are 
touched, with a very small addition to the Developer API.

I’d appreciate feedback on the overall design, API surface, and implementation. 
In particular, I’d like to get broader community feedback on whether this is a 
suitable extension point for Spark’s scheduler.

Thanks in advance for taking the time to review!

Best regards,
Christian
---------------------------------------------------------------------
To unsubscribe e-mail: [email protected]

Reply via email to