Gabriel39 commented on code in PR #68027:
URL: https://github.com/apache/doris/pull/68027#discussion_r4024735881
##########
fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveTextProperties.java:
##########
@@ -160,19 +161,35 @@ private static void extractTextSerDeProps(Map<String,
String> sdParams,
}
private static void extractCsvSerDeProps(Map<String, String> params,
- Map<String, String> result) {
- result.put(ScanNodePropertyKeys.TEXT_COLUMN_SEPARATOR,
- getParamOrDefault(params, SEPARATOR_CHAR, ","));
- result.put(ScanNodePropertyKeys.TEXT_LINE_DELIMITER,
getLineDelimiter(params));
- String quoteChar = getParamOrDefault(params, QUOTE_CHAR, "\"");
+ Map<String, String> tableParams, Map<String, String> result) {
+ // Trino stores CSV settings in table parameters. Honor Hive's
table-over-SerDe precedence
+ // so valid CSV files are not silently split with the default
delimiter and quote characters.
+ String separator = getCsvCharacter(params, tableParams,
SEPARATOR_CHAR, ',');
+ String quoteChar = getCsvCharacter(params, tableParams, QUOTE_CHAR,
'"');
Review Comment:
Addressed in a10147747e. The shared Hive parser implements the complete
embedded-quote transition, including the whole-record UTF-16 position rule and
retention of the embedded quote when appropriate. It also applies OpenCSV
escape semantics to the resulting field. Coverage uses actual Hive-generated
row expectations and both real reader implementations, rather than only
character-tuple acceptance.
##########
fe/fe-connector/fe-connector-hive/src/main/java/org/apache/doris/connector/hive/HiveTextProperties.java:
##########
@@ -160,19 +161,35 @@ private static void extractTextSerDeProps(Map<String,
String> sdParams,
}
private static void extractCsvSerDeProps(Map<String, String> params,
- Map<String, String> result) {
- result.put(ScanNodePropertyKeys.TEXT_COLUMN_SEPARATOR,
- getParamOrDefault(params, SEPARATOR_CHAR, ","));
- result.put(ScanNodePropertyKeys.TEXT_LINE_DELIMITER,
getLineDelimiter(params));
- String quoteChar = getParamOrDefault(params, QUOTE_CHAR, "\"");
+ Map<String, String> tableParams, Map<String, String> result) {
+ // Trino stores CSV settings in table parameters. Honor Hive's
table-over-SerDe precedence
+ // so valid CSV files are not silently split with the default
delimiter and quote characters.
+ String separator = getCsvCharacter(params, tableParams,
SEPARATOR_CHAR, ',');
+ String quoteChar = getCsvCharacter(params, tableParams, QUOTE_CHAR,
'"');
+ String escapeChar = getCsvCharacter(params, tableParams, ESCAPE_CHAR,
'"');
Review Comment:
Addressed in a10147747e. NUL participates in OpenCSV's escape-first branch
even when both quote and escape are disabled. The shared corpus stores binary
inputs and expected fields as hex, distinguishing NULL from empty strings. Both
scanners consume that corpus, and the regression creates raw TEXTFILE fixtures
containing literal NUL bytes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
