taowenjun opened a new pull request, #13129:
URL: https://github.com/apache/seatunnel/pull/13129

   ## Purpose of this change
   
   This contribution to Apache SeaTunnel extends the opt-in Hive 
table-compression inheritance introduced by #13085 to existing Parquet tables. 
A table declaring `parquet.compression=SNAPPY` can otherwise receive 
uncompressed output unless every job repeats the codec.
   
   Closes #13128. This is a separate follow-up requested in #13087, not an 
expansion of that ORC issue.
   
   ## Resulting behavior
   
   - `use_table_compression` stays disabled by default. For existing Parquet 
tables, explicit `compress_codec` (including `NONE`) takes precedence over 
`parquet.compression`, then the existing default.
   - Trim and normalize inherited values; map Parquet `UNCOMPRESSED` to `NONE`. 
Missing, null and blank properties keep existing defaults.
   - Reject unknown values and inherited `ZLIB`, whose Parquet mapping would 
silently write uncompressed output. Explicit codec behavior and ORC behavior 
are unchanged.
   - Skip automatic table creation and unrelated formats. Do not change Hive 
metadata.
   - Document behavior and add local footer/data checks and Hive integration 
scenarios for inherited SNAPPY and explicit NONE.
   
   ## Dependency and validation
   
   **Draft: depends on #13085.** This branch currently includes that unmerged 
ORC prerequisite rebased onto dev at `2eccdc412`. The Parquet delta is isolated 
in commit `6543fe8c9`. After #13085 lands, rebase onto dev to remove 
prerequisite commits before marking this PR ready for review. Full Hive E2E 
execution is also pending.
   
   JDK 11 and JDK 17 targeted Maven verify runs each executed 36 Hive tests 
with zero failures, errors or skipped tests. Actual local Parquet files verify 
footer codecs and exact three-row round trips including NULL, covering SNAPPY 
inheritance, NONE override, UNCOMPRESSED alias and GZIP inheritance. These 
local writer tests use the resolved Hive configuration with Avro's Parquet 
writer; they do not substitute for the full production Hive sink pipeline.
   
   Hive Docker integration scenarios are included. The Hive E2E reactor passed 
Maven verify with `-DskipTests -Dskip.ui=true`, including compilation of the 
new HiveIT scenarios. This is compilation/package evidence only: Docker is 
unavailable locally, so integration execution is not verified. The user offered 
the existing Azkaban `upgrade_validation` project, but its saved session is 
expired (`{"error":"session"}`), confirmed again on request; no project was 
uploaded or executed.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to