andygrove opened a new pull request, #7342: URL: https://github.com/apache/datafusion-comet/pull/7342
## Which issue does this PR close? Part of #7108, which stays open: it asks whether release builds should use a different allocator. ## Rationale for this change Executors on the default glibc build can be OOMKilled by their container while Comet's pools and allocation counter stay well inside their limits, because glibc keeps memory that Comet has freed in its arenas (#7108, #7304). The JDK already has a fix users can apply today without upgrading Comet: `-XX:TrimNativeHeapInterval`, which calls `malloc_trim` on a timer. #7304 suggested documenting it, and the measurements in https://github.com/apache/datafusion-comet/pull/7304#issuecomment-6528220294 show it working on a real workload. On nested TPC-H at SF1000, a 5 s interval took 11 OOMKilled executors to none, for about a 4% cost on the queries that did not need it. ## What changes are included in this PR? A new section in the memory tuning guide, "Returning Freed Native Memory to the Operating System". It covers: - Why freed memory stays resident with glibc, and how to recognize it from the memory usage log and the kill reason. - The flag, which JDKs have it, and `-Xlog:trimnative` to confirm it is in effect. - Merging the flag with existing `spark.executor.extraJavaOptions`, and what happens on an older JDK. - `jcmd <pid> System.trim_native_heap` to check whether a trim helps before changing the configuration. - What a trim costs, with the measured numbers. - That it doesn't apply to jemalloc or mimalloc builds, and that `MALLOC_ARENA_MAX` is the slower alternative. The overhead section also links to the new section: raising the overhead only delays this kind of kill. ## How are these changes tested? It's docs only. `prettier --check` passes. I took the flag's behavior, its log format and the numbers from a JDK 17.0.20 cluster run on Comet 1.2.0. I haven't built the docs site locally. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
