andygrove opened a new issue, #2128: URL: https://github.com/apache/datafusion-ballista/issues/2128
## Is your feature request related to a problem or challenge? `SessionConfig::new_with_ballista()` overrides `datafusion.execution.target_partitions` to a hard-coded `16`: https://github.com/apache/datafusion-ballista/blob/main/ballista/core/src/extension.rs#L370-L377 DataFusion's own default is the number of CPU cores available to the process, which is not meaningful for Ballista because the work runs on executors rather than in the client process, so some override is needed. But a fixed `16` is arbitrary and, as noted in https://github.com/apache/datafusion-ballista/pull/2125#discussion_r3623918647, it may be too low for the clusters people actually run. `target_partitions` is the main lever on scan parallelism in Ballista. File listing chunks a table's files into at most `target_partitions` file groups, and Ballista disables round robin repartitioning, so a scan will never produce more partitions than this setting no matter how much executor capacity is available. With the default, a 200 file table scanned on a cluster with 64 total vcores still produces only 16 partitions, and because `ballista.scheduler.max_partitions_per_task` defaults to `1`, that stage runs as 16 tasks and leaves most of the cluster idle. Users have to know to raise the setting before their cluster is used. ## Describe the solution you'd like Revisit the default. A few directions worth considering: 1. Raise the constant to something larger that better matches typical cluster sizes. 2. Derive the default from cluster capacity (total executor vcores) at session creation time, rather than using a constant. This matches what the scheduler tests already do (`SessionConfig::new_with_ballista().with_target_partitions(total_vcores)`), but the scheduler would need to decide how to handle sessions created before executors register and clusters that scale up or down mid-session. 3. Keep a constant, but pick it deliberately and document the reasoning plus the interaction with `ballista.scheduler.max_partitions_per_task` in the tuning guide. Whichever direction is chosen, the change should come with TPC-H numbers, since raising `target_partitions` increases shuffle partition counts as well as scan parallelism and that is not uniformly a win. ## Describe alternatives you've considered Leaving the default at `16` and relying on documentation, which is the status quo after #2125. ## Additional context Raised by @milenkovicm in review of #2125. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
