github-actions[bot] commented on code in PR #67962:
URL: https://github.com/apache/doris/pull/67962#discussion_r4045263736
##########
be/src/core/column/column_struct.cpp:
##########
@@ -218,24 +219,92 @@ void ColumnStruct::update_hash_with_value(size_t n,
SipHash& hash) const {
}
}
+namespace {
+
+// Invoke `func(run_start, run_end)` for every maximal run of rows in [start,
end) that are not
+// NULL according to `null_data`. An all-zero mask yields the single run
[start, end), so the
+// masked and unmasked range paths produce identical hashes and the same
number of calls.
+template <typename Func>
+void for_each_non_null_run(const uint8_t* __restrict null_data, size_t start,
size_t end,
+ Func&& func) {
+ size_t i = start;
+ while (i < end) {
+ while (i < end && null_data[i] != 0) {
+ ++i;
+ }
+ size_t run_start = i;
+ while (i < end && null_data[i] == 0) {
+ ++i;
+ }
+ if (run_start < i) {
+ func(run_start, i);
+ }
+ }
+}
+
+// Batch-hash every row through the field columns, then restore the rows that
are NULL at the
+// outer level. Surviving rows keep each field's batch hash contract (which
may differ from the
+// single/range contract, e.g. the width of a NULL default), and the field
columns are still
+// called once per column instead of once per row.
+template <typename HashType, typename Func>
+void batch_hash_skipping_null_rows(HashType* __restrict hashes, size_t rows,
+ const uint8_t* __restrict null_data, Func&&
hash_all_rows) {
Review Comment:
[P2] Use allocator-aware storage for the hash scratch
This full-row copy is allocated on every masked crc32c batch hash (about 32
KiB at the default 8,160 rows and 256 KiB at the maximum configured 65,535
rows), and nested `Nullable(Struct(...))` layers keep ancestor copies live
while recursively hashing child structs. The generic jemalloc replacement
selects the backing allocator, but it does not execute `Allocator<false>`'s
Doris memory-limit checks or thread-tracker consume/release path. Please use
`DorisVector<HashType>` (or reusable allocator-aware scratch) so this hot
BE-owned buffer follows the repository's memory-control and allocation-failure
contract.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
