ump45nose commented on code in PR #7227:
URL: https://github.com/apache/datafusion-comet/pull/7227#discussion_r4412922356
##########
native/core/src/parquet/schema_adapter.rs:
##########
@@ -673,18 +683,45 @@ fn spark_has_updater(
| (DataType::Date32, DataType::Timestamp(_, Some(_)))
// Timestamp -> Date.
| (DataType::Timestamp(_, _), DataType::Date32)
+ // Timestamp -> narrower int, floating point or decimal.
+ | (
+ DataType::Timestamp(_, _),
+ DataType::Int8
+ | DataType::Int16
+ | DataType::Int32
+ | DataType::Float32
+ | DataType::Float64
+ | DataType::Decimal128(_, _)
Review Comment:
Good catch — addressed in 5714377.
ArrowWriter writes `Timestamp(Second, _)` as unannotated INT64 (Parquet has
no second unit), and an `ARROW:schema` hint can restore Timestamp. Spark still
reads the Parquet type and accepts `decimal(20,0)` under the INT64
`canReadAsDecimal` rule.
Changes:
- `Timestamp → Decimal*` rejection now applies only to annotated units
(millis/micros/nanos); `Timestamp(Second, _)` follows the INT64 precision floor
(`>= 20`).
- Footer `mark_parquet_leaves` rewrites an ARROW-restored Timestamp over
unannotated INT64 back to `Int64`, so the conversion check keys on the Parquet
type.
- Unit test covers Second→`decimal(20,0)` accept / `decimal(19,0)` reject,
micros still rejected, and Int64 after rewrite; runtime roundtrip via
ArrowWriter Second+UTC → `decimal(20,0)` checks the values.
(AI-assisted reply.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
