andygrove commented on code in PR #2125: URL: https://github.com/apache/datafusion-ballista/pull/2125#discussion_r3625611083
########## docs/source/user-guide/tuning-guide.md: ########## @@ -25,18 +25,31 @@ The goal of any distributed compute engine is to parallelize work as much as pos by adding more compute resource. The basic unit of concurrency and parallelism in Ballista is the concept of a partition. The leaf nodes of a query -are typically table scans that read from files on disk and Ballista currently treats each file within a table as a -single partition (in the future, Ballista will support splitting files into partitions but this is not implemented yet). - -For example, if there is a table "customer" that consists of 200 Parquet files, that table scan will naturally have -200 partitions and the table scan and certain subsequent operations will also have 200 partitions. Conversely, if the -table only has a single Parquet file then there will be a single partition and the work will not be able to scale even -if the cluster has resource available. Ballista supports repartitioning within a query to improve parallelism. -The configuration setting `datafusion.execution.target_partitions`can be set to the desired number of partitions. This is -currently a global setting for the entire context. The default value for this setting is 16. - -Note that Ballista will never decrease the number of partitions based on this setting and will only repartition if -the source operation has fewer partitions than this setting. +are typically table scans that read files from object storage, and the number of partitions such a scan produces is +driven by `datafusion.execution.target_partitions`. This is currently a global setting for the entire context. + +DataFusion's own default for this setting is the number of CPU cores available to the process. Ballista overrides it +to 16 in `SessionConfig::new_with_ballista()`, so a context created with `SessionContext::remote()` or Review Comment: I filed https://github.com/apache/datafusion-ballista/issues/2128 to discuss this -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
