This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/master by this push:
new 49b0bee1691 [test](be) Drop the std::set::end() dereference in
load_data_from_csv (#69695)
49b0bee1691 is described below
commit 49b0bee1691bf3f380ab3f4f8db5dc5fbeffd499
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Thu Oct 29 07:07:38 2026 +0000
[test](be) Drop the std::set::end() dereference in load_data_from_csv
(#69695)
### What problem does this PR solve?
Issue Number: None
Related PR: #48788
Problem Summary:
**In short.** `load_data_from_csv()` in the BE test utilities
dereferences `std::set::end()` in an assertion. With libc++ (macOS) that
reads past the set on the stack, so under ASAN every BE unit test suite
that loads a CSV fixture aborts while loading its fixtures and takes the
rest of the `doris_be_test` run with it. This removes the assertion; it
only repeated the size check on the line above.
**Background.** #48788 added `load_data_from_csv()`
(`be/test/testutil/test_util.cpp`), which the column, data type and
SerDe unit tests use to load their CSV fixtures. It checks that the
SerDes, the columns and the requested column indexes line up:
`serders.size() == columns.size()`, `serders.size() == idxes.size()`,
and then `serders.size() == *idxes.end()`.
**The problem, and what it cost.** Dereferencing the end iterator of a
`std::set` is undefined behavior. In libstdc++ the end iterator points
at the tree header, whose value slot overlaps the element count, so on
Linux the last assertion compares against `idxes.size()` again and
passes. In libc++ it reads outside the set object: ASAN reports a
stack-buffer-overflow, and with `abort_on_error` the whole
`doris_be_test` process stops. The suites of all 24 test files that load
CSV fixtures (`DataTypeDateTimeV2SerDeTest`, `DataTypeNumberSerDeTest`,
`ColumnStringTest`, ...) cannot run on macOS, and a test filter that
includes one of them ends the run there. CI runs on Linux only, so it
never failed there.
**How this PR fixes it.** Remove the assertion. The size check right
above it already guarantees one index per SerDe. No production code
changes. #63474, open since May, carries the same removal among its
macOS build fixes; this PR takes only this line, so the test fix does
not wait for the rest.
**Results.**
| | macOS arm64, ASAN | Linux |
|---|---|---|
| Before | `DataTypeDateTimeV2SerDeTest` aborts in `SetUpTestSuite` with
a stack-buffer-overflow | passes (the assertion repeats the size check)
|
| After | the 24 CSV-fixture suites (364 tests) pass | unchanged |
---
be/test/testutil/test_util.cpp | 2 --
1 file changed, 2 deletions(-)
diff --git a/be/test/testutil/test_util.cpp b/be/test/testutil/test_util.cpp
index 9615c6e9a10..0ca65a7b4ba 100644
--- a/be/test/testutil/test_util.cpp
+++ b/be/test/testutil/test_util.cpp
@@ -189,8 +189,6 @@ void load_data_from_csv(const DataTypeSerDeSPtrs serders,
MutableColumns& column
<< "serder size: " << serders.size() << " column size: " <<
columns.size();
ASSERT_EQ(serders.size(), idxes.size())
<< "serder size: " << serders.size() << " idxes size: " <<
idxes.size();
- ASSERT_EQ(serders.size(), *idxes.end())
- << "serder size: " << serders.size() << " idxes size: " <<
*idxes.end();
std::ifstream file(file_path);
if (!file) {
throw doris::Exception(ErrorCode::INVALID_ARGUMENT, "can not open the
file: {} ",
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
