phillipleblanc commented on issue #2128: URL: https://github.com/apache/datafusion-ballista/issues/2128#issuecomment-5040611144
In Spice we set this to the total size of the cluster: `target_partitions = sum(executor.specification.task_slots)` (i.e. your option 2) Whether that is a good idea depends on the cluster size, but it seems to work well enough in practice for us. Ideally this would be set based on some estimate of how "hard" the query is. A very small query will not benefit from more partitions, whereas a large scan would be. Ref: https://github.com/spiceai/spiceai/pull/9100 > Ballista can run tasks within a stage in parallel across the number of partitions that are configured for the query. The default target_partitions count is set to 16 - ideally this can be set to utilize the entire executor fleet. This PR updates it to set the target_partitions to be equal to target_partitions = sum(executor.task_slots). This allows queries to utilize the full potential of the cluster, without the overhead that DistributeFileScanOptimizer was creating. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
