andygrove opened a new issue, #7212:
URL: https://github.com/apache/datafusion-comet/issues/7212
### Describe the bug
The native Iceberg scan returns NULL for an INT96 timestamp column when the
data file has no Iceberg field ids, the table reads it through a name mapping
(`schema.name-mapping.default`), and the query also selects other columns.
Spark returns the values, and nothing fails, so the wrong result is silent.
- `SELECT ts` on its own returns the right values.
- The same file read without a name mapping returns the right values.
- With micros (`TIMESTAMP_MICROS`) timestamps, the name-mapping path is
correct.
Comet 1.1.0 returns the same NULLs (release jar, Spark 3.5.7, Iceberg
1.8.1), so this is not new in 1.2. Files like this come from `add_files`,
`migrate` or `snapshot` over Parquet that Spark wrote with
`spark.sql.parquet.outputTimestampType=INT96`.
### Steps to reproduce
```scala
// Hadoop catalog `c`
spark.sql("CREATE TABLE c.db.t (id BIGINT, ts TIMESTAMP, s STRING) USING
iceberg")
val table = Spark3Util.loadIcebergTable(spark, "c.db.t")
table.updateProperties()
.set("schema.name-mapping.default",
NameMappingParser.toJson(MappingUtil.create(table.schema())))
.commit()
spark.conf.set("spark.sql.parquet.outputTimestampType", "INT96")
spark.range(5).selectExpr("id", "timestamp_micros(id * 1000003) AS ts",
"CAST(id AS STRING) AS s")
.coalesce(1).write.parquet(dataPath)
val file = /* the single part file under dataPath */
table.newAppend().appendFile(
DataFiles.builder(table.spec()).withPath(file.getPath.toString)
.withFileSizeInBytes(file.getLen).withRecordCount(5)
.withFormat(FileFormat.PARQUET).build()).commit()
spark.sql("REFRESH TABLE c.db.t")
spark.sql("SELECT id, ts, s FROM c.db.t").show()
```
- Spark: `ts` holds the five timestamps.
- Comet: `ts` is NULL in every row. `SELECT ts FROM c.db.t` is correct.
### Expected behavior
The same values as Spark.
### Additional context
Found in the branch-1.2 release audit (#7204). It is not a 1.2 regression.
Related: #6790 (NULL nested fields through a name mapping) and #6794 (nested
fields with no ids and no mapping). Those cover nested columns; this one is a
top-level INT96 column.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
