kazantsev-maksim commented on code in PR #5042:
URL: https://github.com/apache/datafusion-comet/pull/5042#discussion_r4007674830
##########
native/spark-expr/src/string_funcs/levenshtein.rs:
##########
@@ -26,67 +26,188 @@ use datafusion::common::{cast::as_generic_string_array,
DataFusionError, Result}
use datafusion::physical_plan::ColumnarValue;
use std::sync::Arc;
+// Thread-local scratch buffers to avoid heap allocations in the row
processing loop
+thread_local! {
+ static LEVENSHTEIN_SCRATCH: std::cell::RefCell<(Vec<i32>, Vec<i32>)> =
+ std::cell::RefCell::new((Vec::with_capacity(64),
Vec::with_capacity(64)));
Review Comment:
Thanks. That finding applied to the `prepare_scratch` revision (8456bc5). On
the current head the scratch handling was replaced by `with_scratch_buffers`,
which addresses all four points directly:
- Rows larger than `MAX_RETAINED_CAPACITY` (1024) do not use the
thread-local buffers at all; they allocate call-local `vec![..; len]` that are
dropped when the call returns. So an oversized row never attaches to the thread.
- Both the ASCII and the Unicode paths, thresholded and not, go through the
same helper, so Unicode no longer diverges.
- Empty inputs return before touching the buffers, and because oversized
cells never enter TLS, there is nothing left to clean up on that path.
The maximum retained TLS is therefore bounded at `2 * 1024 * 4 B = 8 KB` per
worker thread. I added a capacity-probe test that runs a 16,000,000-character
row and asserts the cached capacities stay within the cap, which captures your
replacement scenario.
If you were reviewing 8456bc5 specifically, could you re-check the current
head?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
