Tim-53 commented on issue #4918: URL: https://github.com/apache/datafusion/issues/4918#issuecomment-6452952527
I tried reproducing this on current main (v55.1.0), and the issue still exists. With an explicit schema, both reordered headers (`a,b` vs `b,a`) and different column names (`a,b` vs `a,c`) silently produce incorrect results. I also tested whether schema inference behaves correctly here. Reordered headers still produce incorrect results, while different column names result in a merged schema (`a,c,b`) but fail during execution with `expected 3 got 2`. arrow-rs recently added `with_header_validation()` in https://github.com/apache/arrow-rs/pull/10144. Enabling it in CsvSource::builder() catches both cases that silently produce incorrect results, including the reordered-header case with schema inference, and produces explicit header mismatch errors instead: ```text CSV header does not match schema at column 0: expected "a" but found "b" CSV header does not match schema at column 1: expected "b" but found "c" ``` The inferred-schema case with different column names still fails with the field-count error, as before. This seems like a fairly simple way to prevent silent incorrect results. The question is whether validation should be enabled by default (possibly with an opt-out for positional mapping), or exposed as an opt-in option. Supporting reordered or missing columns by name would be another possibility, but that seems like a larger change. Any thoughts on which direction makes more sense? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
