andygrove commented on code in PR #2038:
URL:
https://github.com/apache/datafusion-ballista/pull/2038#discussion_r3610707296
##########
ballista/core/src/config.rs:
##########
@@ -335,6 +339,20 @@ static CONFIG_ENTRIES: LazyLock<HashMap<String,
ConfigEntry>> = LazyLock::new(||
DataType::Utf8,
Some("lz4".to_string()),
),
+ ConfigEntry::new(
+ BALLISTA_SCHEDULER_MAX_PARTITIONS_PER_TASK.to_string(),
+ "Upper bound on the number of input partitions packed into a
single \
+ task's `partition_slice`. `0` (default) means unbounded — the
scheduler \
+ fills each task up to the executor's free vcore count. Set to `1`
to \
+ restore the pre-multi-partition-tasks execution model (one task
per \
+ partition), matching the Spark user persona at the cost of the \
+ parallel-sort / parallel-join wins this branch enables. Does not
apply \
+ to collapse stages, which must pack their full pending queue into
a \
+ single task for correctness."
+ .to_string(),
+ DataType::UInt64,
+ Some(0.to_string()),
Review Comment:
Could we default this to 1 for now, so that this is not a breaking change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
