Doris-Breakwater commented on issue #69858: URL: https://github.com/apache/doris/issues/69858#issuecomment-6502937489
Breakwater-GitHub-Analysis-Slot: slot_381a558cb6cd This is a source-confirmed error-handling defect in Arrow ingestion and merits high-priority BE/load triage. A valid Arrow array with an unsupported target-type pairing reaches an unchecked cast and can terminate the BE instead of failing the load. The reported retry-driven compute-group outage increases the operational impact. The issue is currently open, with no labels or prior comments at review time. **Verified source evidence** I checked the configured local checkout, then fetched the relevant upstream files at the reported BE commit, `6a680508899258b93f78a15407a5ef93fb56fe9f`, and current master, `2a4191c1dc94e789e1725d6cb345f46c31a17272`. The BOOLEAN and DECIMAL Arrow-reading functions are unchanged between those two upstream revisions. This is source inspection; I did not run the crashing reproducer or inspect the locally patched production binary. - [ArrowStreamReader](https://github.com/apache/doris/blob/6a680508899258b93f78a15407a5ef93fb56fe9f/be/src/format/arrow/arrow_stream_reader.cpp#L97) checks column names and passes each array to the destination serde without establishing type compatibility. The nullable wrapper checks the read range and validity bitmap before delegating. Those checks do **not** establish that the array matches the destination type or fully validate every value buffer; reaching the nested serde alone is not proof of full Arrow validity. - [BOOLEAN conversion](https://github.com/apache/doris/blob/6a680508899258b93f78a15407a5ef93fb56fe9f/be/src/core/data_type_serde/data_type_number_serde.cpp#L957) casts to `BooleanArray` without checking the result. An `Int32Array` produces a null pointer: validation enabled dereferences it through `check_arrow_boolean_buffer`; validation disabled reaches `Value()` for the supplied nonempty batch. Disabling validation is therefore not a workaround. The adjacent HALF_FLOAT branch already demonstrates checked rejection. - [DECIMAL conversion](https://github.com/apache/doris/blob/6a680508899258b93f78a15407a5ef93fb56fe9f/be/src/core/data_type_serde/data_type_decimal_serde.cpp#L921) has the same unchecked cast in DECIMALV2 and DECIMAL32/64/128I. DECIMALV2 also accesses cast type metadata before reading values, so rejection must precede that access. DECIMAL256 has an equivalent unchecked `Decimal256Array` cast and belongs in the fix/audit scope. The source mechanism matches the supplied stacks. There is no architecture- or cloud-mode condition in these failing branches, although behavior on other deployments has not been independently reproduced. The three-BE outage and transaction timeout remain reporter-observed facts, not independently verified results. **Recommended next steps** 1. Proceed with the offered PR: return `InvalidArgument` immediately after failed array casts, before validation, metadata access, or value reads, regardless of `enable_arrow_input_validation`. Keep integer-to-BOOLEAN coercion as a separately specified behavior change. Preserve existing supported conversions. 2. Add the column name and destination type at `ArrowStreamReader`, which has that context; serde messages alone cannot identify the table column. Preserve the error code while adding context and include the received Arrow type. The reader already propagates returned errors. The ordinary Stream Load failure path attempts rollback, and [response serialization](https://github.com/apache/doris/blob/6a680508899258b93f78a15407a5ef93fb56fe9f/be/src/load/stream_load/stream_load_context.cpp#L67) produces `Status: Fail` and a message. Verify the complete HTTP/transaction path with an integration test. 3. Add direct serde tests for both reported mismatches with validation on/off, nullable and nonnullable targets, and successful correctly typed controls. Extend coverage to DECIMALV2/32/128I/256, empty and all-null arrays, and nested mismatches. Integration tests should assert an informative failed load, no committed rows, and successful subsequent loading on the same BE. 4. Audit remaining Arrow casts against the target revision. [MAP](https://github.com/apache/doris/blob/6a680508899258b93f78a15407a5ef93fb56fe9f/be/src/core/data_type_serde/data_type_map_serde.cpp#L546) and [STRUCT](https://github.com/apache/doris/blob/6a680508899258b93f78a15407a5ef93fb56fe9f/be/src/core/data_type_serde/data_type_struct_serde.cpp#L546) also contain unchecked incoming-array casts. These are source-level audit findings, not additional reproduced incidents. ARRAY already checks its supported list casts in this revision. Buffer-size validation alone is not a general type-compatibility check, so the float-width rejection should not be treated as proof that all other mismatches are handled. **Remaining evidence and immediate mitigation** The supplied DDL, generators, commands, versions, and stacks are sufficient to start the fix; a query profile is unnecessary for this deterministic type-dispatch failure. To close verification, record the PyArrow version and scratch-cluster results on unmodified upstream at the pinned commit, then attach the post-fix responses and BE-survival results. Production logs/core details are only needed if that reproduction diverges. Affected release branches and backport targets still require separate verification; the `4.1.0` build string alone does not establish release-wide coverage. Continue emitting Arrow `bool` and `decimal128(p,s)` matching these target columns, as in the supplied controls. Quarantine a failing batch instead of repeatedly sending the unchanged payload to other BEs; retain the client-side Arrow fallback until a tested fix is deployed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
