matanper opened a new pull request, #70729:
URL: https://github.com/apache/doris/pull/70729

   ### What problem does this PR solve?
   
   Issue Number: close #69855
   
   Related PR: None
   
   Problem Summary:
   
   A Stream Load with `format=arrow` and no `columns` header could not load 
into a
   table that has a column the Arrow stream does not carry. The usual case is a
   nullable column added with `ALTER TABLE ... ADD COLUMN` after the sender was
   built: every Arrow load from that sender failed until the sender was changed.
   JSON loads instead load such a column as its default.
   
   Reproduce:
   
   1. `CREATE TABLE t (k INT, v STRING) DUPLICATE KEY(k) DISTRIBUTED BY HASH(k) 
BUCKETS 1 PROPERTIES ("replication_num" = "1");`
   2. Stream Load an Arrow IPC stream with columns `k` and `v` 
(`format=arrow`). It succeeds.
   3. `ALTER TABLE t ADD COLUMN extra STRING NULL;`
   4. Stream Load the same stream again.
   
   Before: the load fails with
   `Arrow stream requires exactly one column named 'extra'`.
   After: the rows load with `extra` = NULL.
   
   Root cause: the FE plans a source slot for every table column. The format_v2
   `ArrowStreamReader::_open_next_stream` looked each requested slot up by name 
in
   the IPC schema and failed the load when the name was absent.
   
   Fix: a requested column that the current IPC stream does not carry is now
   treated as a missing field, in the same way the format_v2 JSON reader treats 
a
   field absent from a row. `get_block()` fills the column with a NULL 
placeholder
   and marks every row of it in `FileReader::_missing_fields`. Then the existing
   `LoadTableReader::_materialize_missing_values` swaps in the slot's default
   expression. The result:
   
   - a nullable column without a default loads NULL;
   - a column with a default value loads that default;
   - a NOT NULL column without a default stays NULL, and the row is rejected as
     `Column <name> is NULL but is not nullable`. That row counts toward
     `max_filter_ratio`, so with the default ratio the load fails and the error 
log
     names the column. This is the same result as a JSON row missing that field.
   
   Presence is decided per IPC stream, because one load body can concatenate
   several streams with different schemas. A column holding an explicit NULL in 
a
   stream that carries it stays NULL; it is not replaced by the default. A 
schema
   with two fields of the same name still fails, now with
   `Arrow stream has N columns named '<name>'`. Extra Arrow columns the load 
does
   not request were already ignored by name matching on master.
   
   The format_v2 reader matches columns by name, so it never reaches the
   out-of-range block position that the issue reports for branch-4.1's 
positional
   reader.
   
   ### Release note
   
   Stream Load with `format=arrow` now loads a table column that is missing from
   the Arrow stream as its default value (NULL for a nullable column), as JSON
   loads do, instead of failing the load.
   
   ### Check List (For Author)
   
   - Test: Unit Test / Regression test
       - BE UT (`be/test/format_v2/table/load_reader_test.cpp`): replaced
         `RejectsMissingRequestedColumn` with tests for a missing column with a
         default, with an explicit NULL default, a NOT NULL column without a
         default (rows filtered), concatenated streams with different schemas, 
and
         duplicate field names.
       - Regression test 
`load_p0/stream_load/test_arrow_stream_load_missing_columns`:
         defaults, concatenated streams, NOT NULL without default, and the
         ALTER TABLE ADD COLUMN reproduction. Its .out file was generated by the
         regression script on an ASAN FE+BE cluster built from this branch.
       - Ran on ASAN: all 80 tests of LoadArrowPipeTest, LoadJsonPipeTest and
         LoadReaderTest pass; test_arrow_stream_load_missing_columns,
         test_arrow_stream_load_uppercase_column and
         test_arrow_stream_load_type_validation pass.
   - Behavior changed: Yes. An Arrow Stream Load that used to fail on a table
     column missing from the stream now loads that column's default.
   - Does this need documentation: No
   
   Performance: not a performance change. The per-block hot path adds one map 
clear and one branch per requested column, so no benchmark was run.
   
   🤖 Generated with [Claude Code](https://claude.com/claude-code)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to