dwsmith1983 commented on code in PR #6336:
URL: https://github.com/apache/datafusion-comet/pull/6336#discussion_r4380962173
##########
docs/source/user-guide/latest/tuning/memory.md:
##########
@@ -182,11 +187,16 @@ alongside the JVM's own non-heap memory. To size the
overhead from it:
non-heap memory, and add the most untracked memory seen on any executor.
3. Add a margin on top. The log can miss the true peak between samples, and
none of the figures
includes the allocator's fragmentation and retained pages, or memory
allocated by native C
- libraries such as zstd. With native Arrow UDFs, budget the embedded Python
interpreter and
- PyArrow allocations in addition to the untracked memory calculated from the
log. A PyArrow
- result buffer later imported into the JVM can appear in the `imported from
native` figure that
- the estimate subtracts, even though Rust's `allocated` does not count it.
Use the executor's
- peak resident memory to size this additional overhead.
+ libraries such as zstd. The zstd context of the native shuffle writer is
the exception: when
+ `spark.comet.shuffle.compression.codec` is `zstd`, a shuffle task writing
to more than one local
+ partition reserves its context while spilling or writing its output, so
that memory is in
Review Comment:
The sizing guide and the contributor guide now say the context is reserved
only up to level 8. Above that it exceeds the 8 MiB retention cap,
`release_zstd_if_oversized` frees it after each block, and it appears in
neither `reserved` nor `allocated`, so the guide asks to budget one per task
writing shuffle output, about 15 MiB at level 9 and 90 MiB at level 19. The
stale comment next to `release_zstd_if_oversized` in `shuffle_block_writer.rs`
is fixed too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
