kz930 opened a new issue, #8596:
URL: https://github.com/apache/texera/issues/8596
### What happened?
File Scan offers an Encoding field. `FileScanSourceOpDesc` stores it in its
own `encoding` property, but `FileScanSourceOpExec` decodes with
`fileEncoding`, the property inherited from `ScanSourceOpDesc`. The class
carries `@JsonIgnoreProperties(Array("limit", "offset", "fileEncoding"))`, so
`fileEncoding` never survives serialization into the executor and is always its
default, `UTF_8`.
Choosing any other charset therefore changes nothing. A UTF-16 file comes
back decoded as UTF-8 rather than as its text.
Expected: the executor decodes with the charset the Encoding field names.
### How to reproduce?
Deserialize a File Scan descriptor carrying `"encoding":"UTF_16"`, write it
out the way `getPhysicalOp` does, and read it back the way
`FileScanSourceOpExec` does. `fileEncoding` comes back `UTF_8`, and the
`encoding` the user chose is the only place UTF-16 survives.
In the UI: upload a UTF-16 text file, drop a File Scan on it, set Encoding
to UTF_16 and run. The rows hold the file's bytes read as UTF-8, not its lines.
### Version/Branch
1.4.0-incubating-SNAPSHOT (main)
### Commit Hash (Optional)
2ab8ee0f2
### What browsers are you seeing the problem on?
_No response_
### Relevant log output
```shell
JSON =
{"operatorType":"FileScan","dummyPropertyList":[],"encoding":"UTF_16","extract":false,"outputFileName":false,"attributeType":"string","attributeName":"line",...}
exec reads desc.fileEncoding = UTF_8
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
