ricky2129 opened a new pull request, #12956:
URL: https://github.com/apache/seatunnel/pull/12956
### Purpose of this pull request
Fixes #12954.
The Parquet file sink cut every TIMESTAMP to milliseconds, so
`DATETIME(4..6)` / `TIMESTAMP(4..6)` values lost digits that the sources
(MySQL-CDC `MicroTimestamp`, Jdbc `Timestamp#getNanos`) had delivered.
- `ParquetWriteStrategy`
- INT64: the column is declared `TIMESTAMP(MICROS, isAdjustedToUTC=true)`
instead of `TIMESTAMP_MILLIS`, and the value is epoch micros. It is computed
from `getEpochSecond()` (floored) and `getNano()`, so pre-1970 values are
exact, and with `multiplyExact` so the whole `LocalDateTime` range cannot
overflow silently.
- INT96 (`parquet_avro_write_timestamp_as_int96`): the nanos of the day
take `LocalDateTime#getNano()` instead of `Calendar.MILLISECOND`.
- `ParquetReadStrategy`
- Maps INT64 `TIMESTAMP_MICROS` to TIMESTAMP (it was BIGINT) and registers
Avro `TimestampMicrosConversion`. `resolveObject` takes the `Instant` from that
conversion, and legacy `TIMESTAMP_MILLIS` still comes as epoch millis.
- The Group (non-Avro) path converts `TIMESTAMP_MICROS` too.
- `int96ToLocalDateTime` keeps the INT96 nanos. The Avro INT96 branch now
uses that same method instead of a copy of it.
`isAdjustedToUTC` stays `true`, as with `TIMESTAMP_MILLIS` today, so
Spark/Hive still see a TIMESTAMP (not `TIMESTAMP_NTZ`).
### Does this PR introduce _any_ user-facing change?
Yes, recorded in `incompatible-changes.md` (en/zh):
- New files have a `timestamp[us]` column where earlier files had
`timestamp[ms]`. The SeaTunnel reader resolves the type per file and reads a
directory holding both. `pyarrow.dataset` with default options takes one schema
from the first file and fails on such a directory (`Casting from timestamp[us,
tz=UTC] to timestamp[ms, tz=UTC] would lose data`); an explicit microsecond
schema works.
- A `TIMESTAMP_MICROS` column written by another tool (e.g. Spark) is now
read as TIMESTAMP instead of BIGINT.
The S3File/OssFile source type tables list `TIMESTAMP_MICROS`.
### How was this patch tested?
Unit tests added:
- `ParquetWriteStrategyTest#testParquetWriteTimestampKeepsMicroseconds`:
checks the column annotation, reads the raw INT64 back with `GroupReadSupport`,
and round-trips through the reader. Values: `2026-04-27T20:50:29.006733`,
`1970-01-01T00:00:00.000001`, `1969-12-31T23:59:59.999999`,
`1900-06-15T12:30:45.123456`, `9999-12-31T23:59:59.999999`, and a whole second.
-
`ParquetWriteStrategyTest#testParquetWriteTimestampTruncatesBelowMicroseconds`
-
`ParquetWriteStrategyTest#testParquetWriteInt96KeepsSubMillisecondPrecision`
- `ParquetReadStrategyTest#testParquetReadTimestampMicrosAndMillis`: one
file with an INT64 MICROS and an INT64 MILLIS column, read through both the
Avro path and the Group path.
Each production change was reverted on its own to check that a test fails:
MILLIS annotation, epoch millis, Calendar INT96, no
`TimestampMicrosConversion`, no Group-path micros case, no INT96 `setNanos`, no
MICROS schema mapping, no `Instant` branch. Every one of these turns at least
one of the tests above red.
End to end on 2.3.12 with this patch: MySQL 8.4 → MySQL-CDC (snapshot +
binlog) and Jdbc → LocalFile parquet, INT64 and INT96, cell-by-cell against
MySQL.
- All `DATETIME/TIMESTAMP(0..6)` values match.
- Every other column type gives the same output as before.
- SeaTunnel LocalFile source reads back old millis files, new micros files,
a directory with both, and INT96 files with the full precision of each file.
### Check list
* [ ] If any new Jar binary package adding in your PR, please add License
Notice according
[New License
Guide](https://github.com/apache/seatunnel/blob/dev/docs/en/developer/new-license.md)
* [x] If necessary, please update the documentation to describe the new
feature. https://github.com/apache/seatunnel/tree/dev/docs
* [x] If necessary, please update `incompatible-changes.md` to describe the
incompatibility caused by this PR.
* [ ] If you are contributing the connector code, please check that the
following files are updated:
1. Update
[plugin-mapping.properties](https://github.com/apache/seatunnel/blob/dev/plugin-mapping.properties)
and add new connector information in it
2. Update the pom file of
[seatunnel-dist](https://github.com/apache/seatunnel/blob/dev/seatunnel-dist/pom.xml)
3. Add ci label in
[label-scope-conf](https://github.com/apache/seatunnel/blob/dev/.github/workflows/labeler/label-scope-conf.yml)
4. Add e2e testcase in
[seatunnel-e2e](https://github.com/apache/seatunnel/tree/dev/seatunnel-e2e/seatunnel-connector-v2-e2e/)
5. Update connector
[plugin_config](https://github.com/apache/seatunnel/blob/dev/config/plugin_config)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
