matanper opened a new issue, #69855: URL: https://github.com/apache/doris/issues/69855
### Search before asking - [x] I had searched in the [issues](https://github.com/apache/doris/issues?q=is%3Aissue) and found no similar issues. ### Description A Stream Load with `format=arrow` and no `columns` header cannot load into a table that has a column the Arrow stream does not carry. JSON and CSV loads load such a column as its default (NULL for a nullable column). The usual case is a nullable column added to the table after the sender was built: every Arrow load from that sender then fails until the sender is updated. - **branch-4.1** (`2e225ac5ac1`, also 4.1.4): `ArrowStreamReader::get_next_block` (`be/src/format/arrow/arrow_stream_reader.cpp:105-121`) fills only the block columns the record batch carries. The FE plans a source slot for every table column, so the trailing block columns stay empty, and the scanner fails with an internal error: `out of bound in ColumnVector<T>::insert_range_from (data.size() = 0)`. A batch with *more* columns than the block expects reaches an out-of-range block position instead of a clear error. - **master** (`2a4191c1dc9`): the `format_v2` reader matches columns by name and fails the load for any expected column the stream lacks (`be/src/format_v2/arrow/arrow_stream_reader.cpp:417`, `Arrow stream requires exactly one column named '…'`). That error is clear, but the column still cannot be filled with its default. ### Solution Treat a table column absent from the Arrow schema the way the JSON and CSV readers treat a missing field: - a nullable column loads as NULL; - a column with a default value loads its default; - a NOT NULL column without a default fails the load with InvalidArgument naming the column. Report extra Arrow columns the load does not expect as InvalidArgument, not as an out-of-range access. We have a patch on 4.1.4: trailing nullable columns missing from a batch load as NULL, a missing NOT NULL column or an over-wide batch returns InvalidArgument, plus reader unit tests. We have run it on our Arrow Stream Load traffic since mid-October. For master the same rule would go where the `format_v2` reader resolves the requested columns. ### How to Reproduce 1. `CREATE TABLE t (k INT, v STRING) DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) BUCKETS 1 PROPERTIES ("replication_num" = "1");` 2. Stream Load an Arrow IPC stream whose schema has `k` and `v` (`format=arrow`). It succeeds. 3. `ALTER TABLE t ADD COLUMN extra STRING NULL;` 4. Stream Load the same Arrow stream again, with no `columns` header. Expected: the rows load with `extra` = NULL, as a JSON load without that field does. Actual: - on 4.1.4 the load fails with `out of bound in ColumnVector<T>::insert_range_from (data.size() = 0)`; - on master, reading the `format_v2` reader, it fails with `Arrow stream requires exactly one column named 'extra'`. We have not run master. ### Anything Else? Part of the VARIANT + Arrow Stream Load tracking issue #68976. ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
