github-actions[bot] commented on code in PR #66771:
URL: https://github.com/apache/doris/pull/66771#discussion_r3781986214
##########
fs_brokers/cdc_client/src/main/java/org/apache/doris/cdcclient/service/PipelineCoordinator.java:
##########
@@ -621,7 +621,8 @@ public void writeRecords(WriteRecordRequest
writeRecordRequest) throws Exception
String dorisTable =
targetTableMappings.getOrDefault(table, table);
for (String record : result.getRecords()) {
scannedRows++;
- batchStreamLoad.writeRecord(targetDb, dorisTable,
record.getBytes());
+ batchStreamLoad.writeRecord(
+ targetDb, dorisTable,
record.getBytes(StandardCharsets.UTF_8));
Review Comment:
[P2] Encode the stream-load delimiter as UTF-8 too
This converts each record, but the same entity is still framed with
`DorisBatchStreamLoad.lineDelimiter = "\n".getBytes()`. With
`-Dfile.encoding=UTF-16` and two rows buffered for one table,
`BatchRecordBuffer` sends `UTF8(row1) || FE FF 00 0A || UTF8(row2)`;
`read_json_by_line` splits at `0A` but leaves `FE FF 00` after the first JSON
object, so stream load receives malformed JSON. The cited US-ASCII run uses an
ASCII-compatible one-byte newline and flushes one snapshot row and one binlog
row separately, so it never exercises this delimiter branch. Please make the
delimiter UTF-8 (or literal `0x0A`) as part of this fix and cover at least two
rows in one HTTP body under a non-ASCII-compatible default.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
