andygrove commented on PR #5048: URL: https://github.com/apache/datafusion-comet/pull/5048#issuecomment-5876384652
This is a light fully automated review since there are so many PRs open. The footer cache at `spark/src/main/scala/org/apache/spark/sql/comet/CometScanUtils.scala:49` holds at most 32,768 entries, shared by every scan in the driver JVM. A scan over more files than that can never be fully cached, so every time it is planned the footers of all the uncached files are read again (the early exit only helps when a legacy file turns up). I think that leaves out the large tables the cache was added for. Could the bound be configurable, or sized so a large scan fits? Relatedly, `readFacts` (line 88) uses `HadoopInputFile.fromPath`, which calls `getFileStatus` for each file even though `selectedPartitions` already listed it. Would `HadoopInputFile.fromStatus`, with a status built from the `ParquetFileInfo` fields, work here? That saves a round trip per footer, which is a HEAD request on S3. The `spark.comet.scan.parquet.checkDatetimeRebase` doc at `spark/src/main/scala/org/apache/comet/CometConf.scala:1005` says that with the check off, results differ from Spark for dates and timestamps before 1582-10-15. For timestamps, Spark's cutoff is `1900-01-01T00:00:00Z` (`RebaseDateTime.lastSwitchJulianTs`, and the `READ_ANCIENT_DATETIME` message says the same). Under the 3.x default `EXCEPTION` mode, Spark raises on an 1850 timestamp in a file with no `org.apache.spark.version`. Reading a LEGACY file with the session time zone set to `America/Los_Angeles` shifts timestamps between 1582 and 1883-11-18 by 422 seconds. Comet returns the stored value in both cases. Could the doc say "dates before 1582-10-15 or timestamps before 1900-01-01T00:00:00Z" so nobody with 19th-century timestamps turns the check off thinking they're safe? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
