vchag opened a new issue, #66911: URL: https://github.com/apache/doris/issues/66911
## Version Apache Doris **4.0.5-rc01** ## What happened During a sustained bulk-load workload (`INSERT INTO ... FROM LOCAL(...)` loading Parquet files via the local-disk TVF path, ~20 files / batch, `parallel_pipeline_task_num=20`), the BE process crashed with `SIGSEGV` twice over a ~2 hour window, both times with the same signature. The crash occurs on the **write path**, not a query/scan path: the crashing thread's preceding log lines show it actively flushing and finalizing freshly-written segments (`segment_creator.cpp:271] tablet_id:..., flushing rowset_dir:...`) for the table being bulk-inserted into, immediately before the fault. The table being written to has one `VARIANT`-typed column among a mix of standard-typed columns, and 3 inverted indexes (on other, non-variant columns). Process memory was healthy at crash time (~49-50 GB used, well under both the 216 GB soft limit and 240 GB hard limit) — this does **not** look like a memory-pressure crash. ## Backtrace (top frames) ``` 0# doris::signal::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*) at .../common/signal_handler.h:420 1# PosixSignals::chained_handler(int, siginfo*, void*) [clone .part.0] in libjvm.so 2# JVM_handle_linux_signal in libjvm.so 3# 0x00007FDDF0128520 in libc.so.6 4# doris::segment_v2::Segment::new_column_iterator(doris::TabletColumn const&, std::unique_ptr<doris::segment_v2::ColumnIterator>*, doris::StorageReadOptions const*, std::unordered_map<int, std::unique_ptr<std::unordered_map<std::string, std::shared_ptr<doris::segment_v2::SparseColumnCache>>>> const*) at .../olap/rowset/segment_v2/segment.cpp:683 5# doris::vectorized::VStatisticsIterator::init(doris::StorageReadOptions const&) at .../vec/olap/vgeneric_iterators.cpp:54 6# doris::segment_v2::Segment::new_iterator(...) in /opt/apache-doris/be/lib/doris_be 7# doris::segment_v2::LazyInitSegmentIterator::init(doris::StorageReadOptions const&) at .../olap/rowset/segment_v2/lazy_init_segment_iterator.cpp:48 ``` ## What appears to trigger it Immediately preceding the crash, the same thread ID logged repeated `segment_creator.cpp:271] tablet_id:<N>, flushing rowset_dir: ..., rowset_id:...` lines for several different tablets in quick succession (bulk-load segment finalization), interleaved with `variant_statistics.h:37] num subcolumns 0, num sparse columns 0` lines. The crash happens in the call chain that builds a column iterator to compute embedded zone-map/min-max statistics for a just-flushed segment, specifically going through the `SparseColumnCache`-aware overload of `new_column_iterator`, which strongly suggests the fault is in statistics construction for the `VARIANT` column's sparse-subcolumn handling at segment-finalize time — not in the write of the segment data itself (which completed and closed cleanly per the preceding `load_stream_writer.cpp:174] file ... closed` lines). This crashed **twice** in one day under the same general workload (bulk INSERT into this table), both times during segment finalization, both times with process memory well within limits. ### Cluster topology - 4 BE nodes (each on its own host, ~1.6 TB local SSD storage per BE, `storage_format=V3`), 3 FE nodes (1 leader + 2 followers) - `replication_allocation=1` for this table specifically — every tablet has exactly one physical copy, no cross-BE redundancy - BEs configured with a memory soft limit of 216 GB / hard limit of 240 GB (not hit at crash time) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
