Gabriel39 opened a new pull request, #70808:
URL: https://github.com/apache/doris/pull/70808

   ### Background
   
   Doris separates physical file decoding from external table semantics in 
FileScannerV2. Hive supplies column names and partitions; Iceberg additionally 
supplies field IDs, snapshots and delete files. Paimon uses its SDK when raw 
file reads cannot preserve primary-key merges.
   
   ### What problem does this PR solve?
   
   Issue Number: close #70807
   Related PR: None
   PR Type: Feature
   
   Problem Summary:
   
   Master rejects Avro in TVFs and native Hive/Iceberg file scans. Add an Avro 
object-container reader to the existing filesystem and column-mapping pipeline, 
while retaining Paimon's SDK merge path. The implementation also preserves 
absolute row positions, schema evolution and Iceberg delete semantics, and 
admits decoder buffers against query memory limits. Other physical formats keep 
their existing readers.
   
   ### Details
   
   - **Change map:** 1,265 additions/35 deletions in production code; 1,828/97 
in tests; 423 generated-result lines; 615/1 in build, dependency patches and 
notices. About 1,190 production lines implement the physical reader and native 
routing; about 110 add FE admission and per-file format propagation. The 
dependency/build changes supply bounded codec allocation and reproducible 
third-party builds. Tests and generated results account for 2,348 of 4,264 
changed lines.
   - **Scope:** Avro object-container files through TVFs, Hive tables and 
Iceberg data/equality-delete/position-delete files, including the 
position-deletes system table. Paimon Avro files continue through the SDK. 
Support null, deflate, Snappy and Zstandard codecs, scalar/logical/nested 
values and nullable unions with one non-null branch. Recursive records and 
multi-value unions return explicit unsupported errors; Avro output and raw Avro 
messages are outside the scope. Direct Stream Load coverage uses known-length 
file uploads.
   - **Design:** Apache Avro C++ 1.12.1 implements the physical format below 
TableReader and TableColumnMapper. Native filesystem I/O retains existing 
credentials, cache and cancellation behavior. Restoring the removed JNI scanner 
would duplicate the retired scan path; routing Paimon directly through raw 
files would lose SDK merge semantics. Proposed in #70807, opened with this PR; 
no earlier design agreement is claimed.
   - **Usage:** Use `format='avro'` in an existing TVF. Hive input-format 
detection and Iceberg file-format planning select Avro automatically; Paimon 
requires no new setting.
   
   ```sql
   SELECT * FROM S3(
       'uri'='s3://bucket/data.avro',
       's3.endpoint'='https://object-store.example',
       's3.region'='us-east-1',
       's3.access_key'='<access-key>',
       's3.secret_key'='<secret-key>',
       'format'='avro');
   ```
   
   - **Behavior change:** Calls that previously rejected Avro now read 
supported object-container files. Other formats retain their existing readers.
   - **Default behavior:** FileScannerV2 remains enabled by default. 
Compression is read from the container header; no new session variable or 
catalog setting is required.
   - **Compatibility:** Keep the existing `FORMAT_AVRO=12` wire value and 
persisted catalog representation. Reading Avro needs upgraded FE/BE nodes and 
an Avro-enabled third-party build; older BE nodes retain their 
unsupported-format error. No Avro files or lake metadata are rewritten. The 
reader uses the common scan path in shared-nothing and cloud deployments.
   
   ### Testing
   
   - Locally, 228 BE tests across the Avro, schema, Hive, Iceberg, scanner and 
load suites pass on ASAN. They cover every adjacent byte-range cut, 
absolute/selected row IDs, logical/nested values, hidden equality keys, 
position deletes, schema promotion/defaults, DST folds, corrupt/truncated input 
and memory-limit failures.
   - Locally, 505 FE/Java tests pass with zero failures, errors or skips across 
format/TVF admission, Hive recognition, Iceberg planning and Paimon scan 
planning/scanning.
   - Locally, all three new regression suites pass in ordinary comparison mode. 
All 97 generated result blocks match independent fixture literals and Iceberg 
delete rules; the Java Iceberg reader independently verifies every data/delete 
combination. Coverage includes S3 TVFs, known-length Stream Load, Hive 
partitions/reordered/missing fields, Iceberg snapshots/deletes/system tables, 
and Paimon updates/NULLs/deletes. Both Hive version labels use the same local 
Hive 2.3.10 metastore/SerDe; a genuine Hive 3 run is not claimed.
   - The final-range test returns FORMAT_PARQUET before its fix and FORMAT_AVRO 
afterward; the mixed-format planner test now checks actual Thrift descriptors. 
The Hive mapping test aborts before its fix. The nested-array test fails before 
its fix on both retained-memory accounting and 8198 full-tree accounting 
passes; it passes afterward with a bound of six passes for two rows of 4096 
nullable subarrays.
   - External pipeline configuration selects and enables the new suites. Remote 
community CI, Hive 3, remote HDFS and cloud deployments have not been executed 
locally.
   
   ### Notes for Reviewers
   
   The third-party patches validate block/collection bounds, admit allocation 
and codec workspace before growth, accept Java Zstandard frames without a 
content size, and stop before decompressing another range's block. Collection 
resets account only their own subtrees to avoid quadratic rescanning. Paimon 
native eligibility remains restricted to its existing supported formats so 
update/delete merges stay in the SDK.
   
   ### Release note
   
   Support reading Avro object-container files through TVFs and Hive/Iceberg 
external tables in FileScannerV2, with null, deflate, Snappy and Zstandard 
compression. Paimon Avro reads retain SDK merge semantics.
   
   ### Check List (For Author)
   
   - Behavior changed: Yes; previously rejected Avro reads are now accepted. 
Existing readers for other formats keep their behavior.
   - Does this need documentation: Yes; follow up in apache/doris-website with 
TVF syntax, supported codecs/types and upgrade requirements.
   - Pre-submission status: Partially verified: full clang-tidy analysis of the 
modified test files is still running; all changed production files have passed.
   
   ### AI Disclosure
   
   - Tool: Codex (GPT-6 family)
   - Generated: Implementation and tests continued from an existing working 
tree; this description.
   - Human review: not stated
   - Prompts: Support Avro reads on master through TVFs and Hive/Iceberg/Paimon 
lake files, cover them with sufficient tests, and submit a PR.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to