cetra3 commented on code in PR #23565:
URL: https://github.com/apache/datafusion/pull/23565#discussion_r3755018057
##########
datafusion/physical-plan/src/spill/mod.rs:
##########
@@ -420,6 +420,20 @@ pub(crate) fn gc_view_arrays(batch: &RecordBatch) ->
Result<RecordBatch> {
}
}
+/// Garbage collect and deduplicate a string view before writing it out to disk
+///
+/// This is to avoid inflating disk usage and also to ensure that
deduplication reduces memory pressure when reading back.
+fn gc_dedup_view<T: ByteViewType>(
+ array: &GenericByteViewArray<T>,
+) -> GenericByteViewArray<T> {
+ let mut builder = GenericByteViewBuilder::<T>::with_capacity(array.len())
Review Comment:
Yes this is the trade off. However, I believe the size of this is actually
dictated by the configured batch size so it's never going to be more than that.
So with `16384` as the default batch size this could use ~ `16384 * 64 =
1MiB <+ hash internals>` RAM to write this out + CPU overhead of hash lookups
which I think is ~ `log(n)`. So yeah high cardinality string views will
probably cause increased memory/cpu but not by much. Keeping in mind that we
are spilling here to reduce RAM usage, and so when we re-hydrate the spilled
file, the memory usage is *much* more reduced for low cardinality views.
In high cardinality scenarios this will add overhead, but it's hard to
measure without doing a second pass to work out what the cardinality is. Maybe
there is a better way to gauge the cardinality of the input view and use that
to decide? Not sure there is a good method for this.
But, in production we have seen a good reduction in file usage/memory
pressure with gc & dedup (i.e, we have applied this as a patch to our version
of DF), so I feel that for real use cases the trade off is worth it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
