raghav-reglobe opened a new issue, #69858: URL: https://github.com/apache/doris/issues/69858
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Version - FE `doris-4.1.0-39053bed`, BE `doris-4.1.0-6a680508` — our builds from `apache/doris` master as of 2026-10-20 with local patches that do not touch the Arrow reader or the serdes. Cloud mode (compute groups), aarch64 (Graviton), Ubuntu 22.04, Kubernetes. `enable_arrow_input_validation = true` (the default). - The lines involved are byte-identical on `apache/doris` **master today (2026-10-29)**: `be/src/core/data_type_serde/data_type_number_serde.cpp` 957–966 and 968–974 (the HALF_FLOAT contrast), `be/src/core/data_type_serde/data_type_decimal_serde.cpp` 921–923 and 948–951. ### What's Wrong? A Stream Load with `format: arrow` whose IPC stream carries a column of an Arrow type the target column's serde does not expect **crashes the BE (SIGSEGV)** instead of rejecting the load. Two serdes observed in production, same shape: **1. BOOLEAN target, Int32 Arrow array** — `DataTypeNumberSerDe<TYPE_BOOLEAN>::read_column_from_arrow` (`data_type_number_serde.cpp`): ```cpp if constexpr (T == TYPE_BOOLEAN) { const auto* concrete_array = dynamic_cast<const arrow::BooleanArray*>(arrow_array); // 958: nullptr for an Int32Array if (config::enable_arrow_input_validation) { check_arrow_boolean_buffer(*concrete_array); // 960: dereferences nullptr → SIGSEGV } for (int64_t bool_i = start; bool_i != end; ++bool_i) { col_data.emplace_back(concrete_array->Value(bool_i)); // 963: same, with validation off } return Status::OK(); } ``` Stack (BE container stdout, 2026-10-21 23:13:41 UTC; the faulting read is `array.data()` at offset 8 of a null object): ``` 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at ../src/common/signal_handler.h:418 1# 0x0000FFFFAA3A7838 in linux-vdso.so.1 2# doris::check_arrow_validity_bitmap(arrow::Array const&) at ../src/core/data_type_serde/arrow_validation.h:125 3# doris::DataTypeNumberSerDe<(doris::PrimitiveType)2>::read_column_from_arrow(doris::IColumn&, arrow::Array const*, long, long, cctz::time_zone const&) const at ./be/build_Release/../src/core/data_type_serde/data_type_number_serde.cpp:960 4# doris::DataTypeNullableSerDe::read_column_from_arrow(doris::IColumn&, arrow::Array const*, long, long, cctz::time_zone const&) const at ./be/src/core/data_type_serde/data_type_nullable_serde.cpp:425 5# doris::ArrowStreamReader::_do_get_next_block(doris::Block*, unsigned long*, bool*) at ./be/build_Release/../src/format/arrow/arrow_stream_reader.cpp:117 6# doris::GenericReader::get_next_block(doris::Block*, unsigned long*, bool*) at ../src/format/generic_reader.h:115 7# doris::FileScanner::_get_block_wrapped(doris::RuntimeState*, doris::Block*, bool*) at ./be/src/exec/scan/file_scanner.cpp:586 8# doris::FileScanner::_get_block_impl(doris::RuntimeState*, doris::Block*, bool*) at ./be/src/exec/scan/file_scanner.cpp:515 9# doris::Scanner::get_block(doris::RuntimeState*, doris::Block*, bool*) at ./be/src/exec/scan/scanner.cpp:192 10# doris::Scanner::get_block_after_projects(doris::RuntimeState*, doris::Block*, bool*) in /opt/apache-doris/be/lib/doris_be 11# doris::ScannerScheduler::_scanner_scan(std::shared_ptr<doris::ScannerContext>, std::shared_ptr<doris::ScanTask>) at ./be/src/exec/scan/scanner_scheduler.cpp:193 12# doris::ScannerScheduler::execute_scan_task(std::shared_ptr<doris::ScannerContext> const&, std::shared_ptr<doris::ScanTask> const&) at ./be/src/exec/scan/scanner_scheduler.cpp:101 13# doris::ThreadPoolSimplifiedScanScheduler::_run_context(std::shared_ptr<doris::ScannerContext>) at ./be/src/exec/scan/simplified_scan_scheduler.cpp:140 14# std::_Function_handler<void (), doris::ThreadPoolSimplifiedScanScheduler::schedule_scan_task(std::shared_ptr<doris::ScannerContext>, std::shared_ptr<doris::ScanTask>, std::unique_lock<std::mutex>&)::$_0>::_M_invoke(std::_Any_data const&) 15# doris::ThreadPool::dispatch_thread() at ./be/src/util/threadpool.cpp:628 16# doris::Thread::supervise_thread(void*) at ./be/src/util/thread.cpp:461 17# 0x0000FFFFA8DA0398 in /lib/aarch64-linux-gnu/libc.so.6 18# 0x0000FFFFA8E09E9C in /lib/aarch64-linux-gnu/libc.so.6 ``` Frame 4 shows the nullable wrapper had already validated the array (`data_type_nullable_serde.cpp:425`), so the input was a well-formed Arrow column — just not a `BooleanArray` (the client typed a MySQL `tinyint(1)` column as `int32`). **2. DECIMAL target, Utf8 Arrow array** — `DataTypeDecimalSerDe<TYPE_DECIMAL64>::read_column_from_arrow` (`data_type_decimal_serde.cpp`), target column `DECIMAL(10,2)` (decimalv3), the client sent the value as a utf8 string: ```cpp } else if constexpr (T == TYPE_DECIMAL32 || T == TYPE_DECIMAL64 || T == TYPE_DECIMAL128I) { const auto* concrete_array = dynamic_cast<const arrow::DecimalArray*>(arrow_array); // 949: nullptr for a StringArray for (auto value_i = start; value_i < end; ++value_i) { const auto* value = concrete_array->Value(value_i); // 951: SIGSEGV ``` (The `TYPE_DECIMALV2` branch at 921–923 has the same unchecked cast.) Stack (2026-10-25 19:19:06 UTC): ``` 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at ../src/common/signal_handler.h:418 1# 0x0000FFFFBD9E2838 in linux-vdso.so.1 2# doris::DataTypeDecimalSerDe<(doris::PrimitiveType)29>::read_column_from_arrow(doris::IColumn&, arrow::Array const*, long, long, cctz::time_zone const&) const at ./be/src/core/data_type_serde/data_type_decimal_serde.cpp:951 3# doris::DataTypeNullableSerDe::read_column_from_arrow(doris::IColumn&, arrow::Array const*, long, long, cctz::time_zone const&) const at ./be/src/core/data_type_serde/data_type_nullable_serde.cpp:425 4# doris::ArrowStreamReader::_do_get_next_block(doris::Block*, unsigned long*, bool*) at ./be/build_Release/../src/format/arrow/arrow_stream_reader.cpp:117 5# doris::GenericReader::get_next_block(doris::Block*, unsigned long*, bool*) at ../src/format/generic_reader.h:115 6# doris::FileScanner::_get_block_wrapped(doris::RuntimeState*, doris::Block*, bool*) at ./be/src/exec/scan/file_scanner.cpp:586 7# doris::FileScanner::_get_block_impl(doris::RuntimeState*, doris::Block*, bool*) at ./be/src/exec/scan/file_scanner.cpp:515 8# doris::Scanner::get_block(doris::RuntimeState*, doris::Block*, bool*) at ./be/src/exec/scan/scanner.cpp:192 9# doris::Scanner::get_block_after_projects(doris::RuntimeState*, doris::Block*, bool*) in /opt/apache-doris/be/lib/doris_be 10# doris::ScannerScheduler::_scanner_scan(std::shared_ptr<doris::ScannerContext>, std::shared_ptr<doris::ScanTask>) at ./be/src/exec/scan/scanner_scheduler.cpp:193 11# doris::ScannerScheduler::execute_scan_task(std::shared_ptr<doris::ScannerContext> const&, std::shared_ptr<doris::ScanTask> const&) at ./be/src/exec/scan/scanner_scheduler.cpp:101 12# doris::ThreadPoolSimplifiedScanScheduler::_run_context(std::shared_ptr<doris::ScannerContext>) at ./be/src/exec/scan/simplified_scan_scheduler.cpp:140 13# std::_Function_handler<void (), doris::ThreadPoolSimplifiedScanScheduler::schedule_scan_task(std::shared_ptr<doris::ScannerContext>, std::shared_ptr<doris::ScanTask>, std::unique_lock<std::mutex>&)::$_0>::_M_invoke(std::_Any_data const&) 14# doris::ThreadPool::dispatch_thread() at ./be/src/util/threadpool.cpp:628 15# doris::Thread::supervise_thread(void*) at ./be/src/util/thread.cpp:461 16# 0x0000FFFFBC3E0398 in /lib/aarch64-linux-gnu/libc.so.6 17# 0x0000FFFFBC449E9C in /lib/aarch64-linux-gnu/libc.so.6 ``` **Blast radius.** The client sees a reset connection, not a load error, so it retries the same batch on the next BE: in our case one bad batch crashed all three BEs of the compute group in turn within three minutes (identical stack on each), and every later commit of that table re-armed the loop until the Arrow path was switched off client-side. **Contrast inside the same function.** The HALF_FLOAT branch directly below the BOOLEAN branch does it right (`data_type_number_serde.cpp` 969–974): ```cpp const auto* concrete_array = dynamic_cast<const arrow::HalfFloatArray*>(arrow_array); if (concrete_array == nullptr) { return Status::InvalidArgument("Expected Arrow HalfFloatArray, got {}", arrow_array->type()->name()); } ``` and a *width* mismatch (a float32 array into a DOUBLE column) is already rejected cleanly by the validation layer (`[E33] Invalid Arrow float: data buffer too small: 32 bytes available, 64 required`). A *type* mismatch should fail the same way. ### What You Expected? The Stream Load fails with an error that names the column and the Arrow type it received (an `InvalidArgument` of the HALF_FLOAT shape), the HTTP response carries `Status: Fail` with that message, and the BE stays up. ### How to Reproduce? **Only on a scratch cluster — this kills the BE process.** One BE, default config (`enable_arrow_input_validation=true`; the crash also happens with it off, one line later). ```sql CREATE DATABASE IF NOT EXISTS repro; CREATE TABLE repro.t_bool (k INT, b BOOLEAN NULL) DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1"); CREATE TABLE repro.t_dec (k INT, v DECIMAL(10,2) NULL) DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1"); ``` ```python # make_streams.py — pyarrow IPC streams (pip install pyarrow) import pyarrow as pa, pyarrow.ipc as ipc from decimal import Decimal def write(path, table): with pa.OSFile(path, "wb") as f, ipc.new_stream(f, table.schema) as w: w.write_table(table) write("bool_crash.arrow", pa.table({"k": pa.array([1], pa.int32()), "b": pa.array([1], pa.int32())})) # int32 → BOOLEAN: crash write("bool_ok.arrow", pa.table({"k": pa.array([1], pa.int32()), "b": pa.array([True], pa.bool_())})) # control: loads write("dec_crash.arrow", pa.table({"k": pa.array([1], pa.int32()), "v": pa.array(["1.50"], pa.utf8())})) # utf8 → DECIMAL(10,2): crash write("dec_ok.arrow", pa.table({"k": pa.array([1], pa.int32()), "v": pa.array([Decimal("1.50")], pa.decimal128(10, 2))})) # control: loads ``` ```bash FE=http://<fe-host>:8030 # controls first — both return "Status": "Success" curl --location-trusted -u root: -H "format: arrow" -H "columns: k,b" -T bool_ok.arrow $FE/api/repro/t_bool/_stream_load curl --location-trusted -u root: -H "format: arrow" -H "columns: k,v" -T dec_ok.arrow $FE/api/repro/t_dec/_stream_load # the crashes — each returns "Empty reply from server" / connection reset, be.out prints the stack above, the BE exits curl --location-trusted -u root: -H "format: arrow" -H "columns: k,b" -T bool_crash.arrow $FE/api/repro/t_bool/_stream_load curl --location-trusted -u root: -H "format: arrow" -H "columns: k,v" -T dec_crash.arrow $FE/api/repro/t_dec/_stream_load ``` Expected on both crash cases: a rejected load. Actual: BE SIGSEGV at `data_type_number_serde.cpp:960` / `data_type_decimal_serde.cpp:951` (frames above), process exit, the transaction left in PREPARE until the stream-load timeout. ### Anything Else? - Proposed fix (minimal, the HALF_FLOAT pattern): in every `read_column_from_arrow` that `dynamic_cast`s the array — BOOLEAN, DECIMALV2, DECIMAL32/64/128I, and an audit of the other serdes — check the result and return `Status::InvalidArgument("Expected Arrow {}Array for column type {}, got {}", ...)`. Accepting Int8/Int16/Int32/UInt8 arrays for BOOLEAN (a cast, since MySQL `tinyint(1)` arrives that way from most Arrow writers) would be a reasonable follow-up, but the reject alone closes the crash. - A BE unit test for each: an `arrow::Int32Array` into a BOOLEAN column and an `arrow::StringArray` into a DECIMAL64 column both expect `InvalidArgument`, with `enable_arrow_input_validation` on and off. - Our fork does not yet carry the fix; the two incidents were stopped client-side (the writer now types BOOLEAN targets as `bool` and DECIMAL targets as `decimal128(p,s)`). We are willing to submit the PR. ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
