comphead commented on PR #7304: URL: https://github.com/apache/datafusion-comet/pull/7304#issuecomment-6526208763
First round of cluster data for the comparison you asked for. It is all on glibc and covers the idle trim, no trim, and `MALLOC_ARENA_MAX=2`. The timer arm (`-XX:TrimNativeHeapInterval`), a jemalloc build and this PR's own build have not run yet. **Setup.** A wide `GROUP BY` shaped like a production job that gets OOMKilled: 87 grouping keys with `SUM` of a bigint and `MIN`/`MAX` of two strings, over 121M rows from Iceberg, with 40 map tasks and 50 result tasks. Spark plans sort aggregates for it, and the whole plan runs natively. There are 4 executors with 12 task slots, an 8 GiB heap, 32 GiB off-heap and a 49.16 GiB pod limit. Comet is main at 7f73d86 (DataFusion 55.2, system allocator) on Spark 3.4 and JDK 17. A plugin samples cgroup v2 memory and Comet's `Native.getMemoryUsage` every second. **How the trim was applied.** The same plugin calls `malloc_trim(0)`, through the JVM's `System.trim_native_heap`, when an executor's last running task ends. It runs on that task's thread before the task reports, which is about the moment this PR's hook fires, so the next stage cannot start on the executor first. At all 52 such points across five Comet runs, there were 0 live native plans and at most 2.3 MiB allocated. That includes executors where Spark killed 24 tasks when a job aborted. So `live_plan_count() == 0` would have held each time. **Results.** | Run | Outcome | Time | |---|---|---| | Spark | passed | 135 s | | Comet, no trim, `maxBufferBytes=256m` | aborted, 13 executors OOMKilled | 547 s | | Comet, idle trim, `maxBufferBytes=256m` | passed | 195 s | | Comet, idle trim, no buffer cap, two runs | passed | 203 s, 214 s | | Comet, no trim, `MALLOC_ARENA_MAX=2` | passed | 296 s | - **Retention without a trim.** After its map tasks, an executor holds 28-34.5 GiB of anonymous memory while Comet's allocation counter reads at most 2.3 MiB. Spark's executors hold 6.2 GiB at that point. The result wave reused only 1-7 GiB of it, and every result wave was OOMKilled 14-29 s in, with 18-26 GiB of live native memory. - **What the trim returns.** After the map wave, 27-46 GiB of resident memory went down to about 1 GiB. After the result wave, 39-45 GiB went down to 4.5-5.5 GiB. Each call took 1.8-3.8 s, and because it runs on the last task's thread, each stage ended 2-4 s later. - **Arenas.** With `MALLOC_ARENA_MAX=2` and no trim, the result wave reused the retained memory instead of adding to it. That fits the explanation in the description: the map plan frees on Tokio workers, while the result plan allocates on task threads. It was 38% slower than the trimmed run with the same buffer setting, close to the 37-39% in #7108, and 35-40 GiB stayed resident after the job. - **What a trim cannot fix.** Starting from about 1 GiB, a result wave still climbs to 42-45.5 GiB on its own, 3.7 GiB below the limit at worst. Memory that builds up inside a wave is never released at an idle point. Still to run on the same setup: `-XX:TrimNativeHeapInterval=5000`, this PR's build, and a jemalloc build. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
