raghavyadav01 commented on issue #26514: URL: https://github.com/apache/datafusion/issues/26514#issuecomment-6330350990
I tried the first option I proposed above — defaulting `double_quote` to `false` on the write path when `escape` is set — and it does **not** fix the round trip. Posting the evidence before anyone spends time on that approach. With `'format.escape' '\'` and `double_quote` defaulted to `false`, `COPY` writes: ``` column1 "a\b, c" "d\e" plain ``` The quotes are there, but **the backslashes are written raw**. `arrow-csv` uses the escape character for the quote character only; it never escapes the escape character itself. Reading that file back with the same `format.escape` therefore still consumes them: | value written | read back, before | read back, with double_quote=false | |---|---|---| | `a\b, c` | `ab, c` | `ab, c` | | `d\e` | `d\e` | `de` | It is actually worse than `main`: `d\e` has no delimiter in it so today it is written unquoted and survives, but with `double_quote=false` the writer quotes it and the backslash is then eaten on read. So this cannot be fixed by choosing different writer options from DataFusion. For escape-style CSV to round trip, the writer has to escape the escape character as well, which is `arrow-csv` behaviour rather than ours. That leaves the second option: treat `format.escape` on the write path as unsupported and say so, rather than accepting it and silently producing a file that cannot be read back with the same options. Something like rejecting `escape` for writes (optionally unless `double_quote` is explicitly `false`, which is the existing tested combination), so the failure is loud. Worth noting the existing coverage passes only by luck of its data: `csv_files.slt` tests `double_quote=false` + `escape='#'` over values containing quotes, and quotes *are* escaped. Values containing a literal `#` would fail the same way. Happy to open a PR for the validation route, or for an `arrow-rs` issue on the writer, whichever you prefer. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
