Gabriel39 opened a new pull request, #68027:
URL: https://github.com/apache/doris/pull/68027
### What problem does this PR solve?
Hive CSV tables created by Trino can store `separatorChar`, `quoteChar`, and
`escapeChar` only in table parameters. The CSV scan-property extractor read
only SerDe parameters, so custom CSV files were parsed with default delimiters,
producing merged, shifted, or empty columns.
Read CSV settings with the existing table-over-SerDe precedence and default
fallback, including deriving quote trimming from the effective quote character.
### Release note
Fix incorrect results when reading Hive CSV tables whose custom CSV settings
are stored in table properties.
### Check List (For Author)
- Test:
- [x] Unit Test: six property tests and one scan-planning test covering
both partitioned and unpartitioned tables. Six tests fail before the fix; all
42 targeted tests pass after it.
- [x] Regression test added: four CSV file layouts, custom `s/q/e`
settings, commas, escaped quotes, escape characters, empty strings, full-row
reads, aggregates, and filters. Results are compared with the source data
queried through Hive.
- Full Hive connector suite: 469 tests passed; 1,239 tests passed
including upstream modules.
- FE Checkstyle and `git diff --check` passed.
- Hive end-to-end execution remains pending: no reachable HiveServer2 in
the local test configuration.
- Behavior changed:
- [x] Yes. CSV scans honor table parameters before SerDe parameters
instead of silently using defaults.
- Does this need documentation?
- [x] No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
