andygrove opened a new pull request, #2510: URL: https://github.com/apache/datafusion-ballista/pull/2510
# Which issue does this PR close? No issue. This is the config follow-up to #2498 [promised in review](https://github.com/apache/datafusion-ballista/pull/2498#issuecomment-5897209313). # Rationale for this change #2498 made the scheduler share one file statistics cache across sessions, so planning no longer reads every Parquet footer again for every job. A cached entry is only reused while the file's size and modification time match the job's own listing, and that check misses a file rewritten in place without changing either (apache/datafusion#25841 tracks comparing e-tags and versions). @milenkovicm asked for a `ballista.` setting to turn sharing off should corner cases like that turn up. # What changes are included in this PR? - A new setting, `ballista.scheduler.share_file_statistics_cache`, true by default, and a `BallistaConfig::share_file_statistics_cache()` getter. - `share_file_statistics_cache` reads the setting from the config each session is built with. A session with it set to false is returned as the session builder built it, which with the default builder means a statistics cache of its own, as before #2498. It never becomes the shared cache, so sessions left at the default keep sharing. - The doc comments on `share_file_statistics_cache` and `BallistaCluster::new_memory` and the generated config reference describe the setting. A new test goes through `BallistaCluster::new_memory` and builds a session with the setting off, then two with the default. The two default sessions share a cache and the opted-out one keeps its own. Without the change it fails, because the opted-out session is built first and its cache becomes the shared one. # Are there any user-facing changes? There is a new setting. Running `SET ballista.scheduler.share_file_statistics_cache = false`, or setting it on the client's `SessionConfig`, keeps that session's jobs off the shared cache. It applies per session. Clients send every `ballista.*` setting with each job, so the client's value wins over any default set on the scheduler side, and there is no scheduler-wide switch. The default keeps the current behavior, and no existing API changes. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
