dwsmith1983 opened a new issue, #6747: URL: https://github.com/apache/datafusion-comet/issues/6747
## Describe the bug The native CSV scan builds one object store for the whole scan, from the first file of the first partition ([planner.rs#L2003-L2026](https://github.com/apache/datafusion-comet/blob/e038cf05fa665dc61fdecbdde1c07b8b6a1ce3cb/native/core/src/execution/planner.rs#L2003-L2026)). It then reads the current partition's files through that store, keeping only each file's path. A CSV scan over two buckets therefore reads every partition from the first bucket. Unlike the Parquet scan, this also happens when no partition mixes buckets. ## Steps to reproduce MinIO through `CometS3TestBase`, native CSV enabled (`spark.comet.scan.csv.v2.enabled=true` and `spark.sql.sources.useV1SourceList=""`). Two buckets each hold one small CSV file with columns `id` and `bucket` (`'A'` or `'B'`). Set `spark.sql.files.openCostInBytes=128MB` so that each file gets its own partition, then run: ```scala spark.read.schema("id INT, bucket STRING").csv("s3a://repro-a/csv1-same", "s3a://repro-b/csv1-same").collect() ``` The plan is `CometProject <- CometCsvNativeScan`. Rows below are sorted after collecting. - **Same key in both buckets** (`csv1-same/f.csv`): partition 0 is `s3a://repro-a/csv1-same/f.csv` and partition 1 is `s3a://repro-b/csv1-same/f.csv`. Spark returns `[1,A] [1,B] [2,A] [2,B] [3,A] [3,B]`. Comet returns `[1,A] [1,A] [2,A] [2,A] [3,A] [3,A]`, with no error. - **Different keys**: `repro-a/csv1-a/a.csv` and `repro-b/csv1-b/b.csv` fail with `[FAILED_READ_FILE.FILE_NOT_EXIST] ... reading file csv1-b/b.csv`. Listing the bucket-b path first flips it, and the scan fails on `csv1-a/a.csv`. Every partition uses the store of whichever file comes first. - **Both buckets in one partition**: this fails the same way as the Parquet scan. ## Expected behavior The same rows as Spark, or a fallback to Spark. ## Additional context Reproduced on `main` at e038cf05f with the default Spark 4.1 profile. Building the store from the current partition's first file, as the Parquet scan does, fixes the separate-partition case. A partition that mixes buckets then has the Parquet scan's problem (#6746). #6059 declines CSV scans whose ABFS files span more than one container or account. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
