Gabriel39 opened a new pull request, #70808:
URL: https://github.com/apache/doris/pull/70808
### Background
Doris separates physical file decoding from external table semantics in
FileScannerV2. Hive supplies column names and partitions; Iceberg additionally
supplies field IDs, snapshots and delete files. Paimon uses its SDK when raw
file reads cannot preserve primary-key merges.
### What problem does this PR solve?
Issue Number: close #70807
Related PR: None
PR Type: Feature
Problem Summary:
Master rejects Avro in TVFs and native Hive/Iceberg file scans. Add an Avro
object-container reader to the existing filesystem and column-mapping pipeline,
while retaining Paimon's SDK merge path. The implementation also preserves
absolute row positions, schema evolution and Iceberg delete semantics, and
admits decoder buffers against query memory limits. Other physical formats keep
their existing readers.
### Details
- **Change map:** 1,265 additions/35 deletions in production code; 1,828/97
in tests; 423 generated-result lines; 615/1 in build, dependency patches and
notices. About 1,190 production lines implement the physical reader and native
routing; about 110 add FE admission and per-file format propagation. The
dependency/build changes supply bounded codec allocation and reproducible
third-party builds. Tests and generated results account for 2,348 of 4,264
changed lines.
- **Scope:** Avro object-container files through TVFs, Hive tables and
Iceberg data/equality-delete/position-delete files, including the
position-deletes system table. Paimon Avro files continue through the SDK.
Support null, deflate, Snappy and Zstandard codecs, scalar/logical/nested
values and nullable unions with one non-null branch. Recursive records and
multi-value unions return explicit unsupported errors; Avro output and raw Avro
messages are outside the scope. Direct Stream Load coverage uses known-length
file uploads.
- **Design:** Apache Avro C++ 1.12.1 implements the physical format below
TableReader and TableColumnMapper. Native filesystem I/O retains existing
credentials, cache and cancellation behavior. Restoring the removed JNI scanner
would duplicate the retired scan path; routing Paimon directly through raw
files would lose SDK merge semantics. Proposed in #70807, opened with this PR;
no earlier design agreement is claimed.
- **Usage:** Use `format='avro'` in an existing TVF. Hive input-format
detection and Iceberg file-format planning select Avro automatically; Paimon
requires no new setting.
```sql
SELECT * FROM S3(
'uri'='s3://bucket/data.avro',
's3.endpoint'='https://object-store.example',
's3.region'='us-east-1',
's3.access_key'='<access-key>',
's3.secret_key'='<secret-key>',
'format'='avro');
```
- **Behavior change:** Calls that previously rejected Avro now read
supported object-container files. Other formats retain their existing readers.
- **Default behavior:** FileScannerV2 remains enabled by default.
Compression is read from the container header; no new session variable or
catalog setting is required.
- **Compatibility:** Keep the existing `FORMAT_AVRO=12` wire value and
persisted catalog representation. Reading Avro needs upgraded FE/BE nodes and
an Avro-enabled third-party build; older BE nodes retain their
unsupported-format error. No Avro files or lake metadata are rewritten. The
reader uses the common scan path in shared-nothing and cloud deployments.
### Testing
- Locally, 228 BE tests across the Avro, schema, Hive, Iceberg, scanner and
load suites pass on ASAN. They cover every adjacent byte-range cut,
absolute/selected row IDs, logical/nested values, hidden equality keys,
position deletes, schema promotion/defaults, DST folds, corrupt/truncated input
and memory-limit failures.
- Locally, 505 FE/Java tests pass with zero failures, errors or skips across
format/TVF admission, Hive recognition, Iceberg planning and Paimon scan
planning/scanning.
- Locally, all three new regression suites pass in ordinary comparison mode.
All 97 generated result blocks match independent fixture literals and Iceberg
delete rules; the Java Iceberg reader independently verifies every data/delete
combination. Coverage includes S3 TVFs, known-length Stream Load, Hive
partitions/reordered/missing fields, Iceberg snapshots/deletes/system tables,
and Paimon updates/NULLs/deletes. Both Hive version labels use the same local
Hive 2.3.10 metastore/SerDe; a genuine Hive 3 run is not claimed.
- The final-range test returns FORMAT_PARQUET before its fix and FORMAT_AVRO
afterward; the mixed-format planner test now checks actual Thrift descriptors.
The Hive mapping test aborts before its fix. The nested-array test fails before
its fix on both retained-memory accounting and 8198 full-tree accounting
passes; it passes afterward with a bound of six passes for two rows of 4096
nullable subarrays.
- External pipeline configuration selects and enables the new suites. Remote
community CI, Hive 3, remote HDFS and cloud deployments have not been executed
locally.
### Notes for Reviewers
The third-party patches validate block/collection bounds, admit allocation
and codec workspace before growth, accept Java Zstandard frames without a
content size, and stop before decompressing another range's block. Collection
resets account only their own subtrees to avoid quadratic rescanning. Paimon
native eligibility remains restricted to its existing supported formats so
update/delete merges stay in the SDK.
### Release note
Support reading Avro object-container files through TVFs and Hive/Iceberg
external tables in FileScannerV2, with null, deflate, Snappy and Zstandard
compression. Paimon Avro reads retain SDK merge semantics.
### Check List (For Author)
- Behavior changed: Yes; previously rejected Avro reads are now accepted.
Existing readers for other formats keep their behavior.
- Does this need documentation: Yes; follow up in apache/doris-website with
TVF syntax, supported codecs/types and upgrade requirements.
- Pre-submission status: Partially verified: full clang-tidy analysis of the
modified test files is still running; all changed production files have passed.
### AI Disclosure
- Tool: Codex (GPT-6 family)
- Generated: Implementation and tests continued from an existing working
tree; this description.
- Human review: not stated
- Prompts: Support Avro reads on master through TVFs and Hive/Iceberg/Paimon
lake files, cover them with sufficient tests, and submit a PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]