goutamadwant opened a new pull request, #12777: URL: https://github.com/apache/seatunnel/pull/12777
### Purpose of this pull request Part of #12763 - Phase 1 - Iceberg 1.6.1 -> 1.10.2, the first line with the finalized v3 spec (1.11+ requires Java 17). Parquet 1.13.1 → 1.16.0 and Avro 1.11.3 → 1.12.1 to match. - `format-version` 3 is accepted when the sink creates a table. 2 stays the default; values outside 1–3 fail before the table is created. - Types: `timestamp_ns` and `timestamptz_ns` map to TIMESTAMP and TIMESTAMP_TZ with scale 9, `unknown` reads as null, and `variant`, `geometry` and `geography` fail with a clear error. - Upsert into v3 tables is rejected before any data is written: v3 requires deletion vectors for position deletes, which the delta writer does not produce yet. - Parquet 1.16 opens files with `FileSystem.openFile` (Hadoop 3.3+). Zeta ships Hadoop 3.1.4 and the Flink images Hadoop 2.7, so after the upgrade every Parquet read failed, including v1 and v2 tables. `HadoopStreamFileIO` falls back to a plain stream for Parquet reads only when that method is missing. Open question: keep this fallback, or raise the Hadoop runtime to 3.3+? ### Does this PR introduce _any_ user-facing change? Yes. v3 tables can be created, read and appended to; v1 and v2 behaviour is unchanged. On Spark, timestamps keep microsecond (TIMESTAMP) and millisecond (TIMESTAMP_TZ) precision because of the Spark translation; this is documented. The upgrade also picks up fixes for CVE-2024-47561 (Avro), CVE-2025-30065 and CVE-2025-46762 (parquet-avro) and CVE-2025-67721 (aircompressor). ### How was this patch tested? | Test | Result | |---|---| | Unit tests (`IcebergFormatVersionTest`, type mapper, row converter), JDK 11 and 17 | 129 run, 0 failures | | `IcebergSinkIT` with v3 write/read/copy, v3 upsert rejection and format-version 4 rejection, on Zeta, Flink 1.18, Flink 1.20, Spark 3.3 | 24/24; the v3 write failed on all engines before the change | | `IcebergSourceIT`, `IcebergSinkWithBranchIT`, `IcebergValidationIT`, `JdbcToIcebergTimestampIT`, `PostgresCdcToIcebergRecipeIT`, `IcebergSinkCDCIT` | Pass; one `IcebergSinkCDCIT` Flink 1.20 timeout passed 5/5 on rerun | | hadoop3 module, S3/MinIO module | 4/4, 1/1 | | `IcebergSinkParallelCommitIT` | 5/6; `testRecoveryFromSavepointProducesNoDuplicateSnapshots` also fails locally on unchanged dev (fixed 9 s wait vs. about 10 s job start) | Not in this PR: deletion vectors, row lineage, default values, multi-argument transforms. ### Check list - [x] New jars in the shaded connector: failsafe, jts-core, parquet-variant, javax.annotation-api. Let me know if they need LICENSE entries. - [x] Updated English and Chinese documentation. - [x] v1 and v2 behaviour unchanged; no incompatible-changes entry. - [x] Added E2E coverage. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
