github-actions[bot] commented on code in PR #69320:
URL: https://github.com/apache/doris/pull/69320#discussion_r4321532533
##########
be/src/exec/operator/file_scan_operator.cpp:
##########
@@ -145,6 +145,29 @@ bool
FileScanLocalState::TEST_should_use_file_scanner_v2(const TQueryOptions& qu
bool FileScanLocalState::should_use_file_scanner_v2(const TQueryOptions&
query_options,
bool is_load,
const
TFileScanRangeParams& scan_params) {
+ if (is_load) {
+ // Load planners omit query session options. Thrift's default makes an
omitted V2
+ // option indistinguishable from false, so ingestion selects by format
instead.
+ switch (scan_params.format_type) {
+ case TFileFormatType::FORMAT_PARQUET:
+ case TFileFormatType::FORMAT_ORC:
+ case TFileFormatType::FORMAT_JSON:
Review Comment:
[P1] Preserve message-by-message EOF for JSON routine loads
This now sends JSON routine loads to V2, but
`NereidsRoutineLoadTaskInfo.isReadJsonByLine()` is always false and
`KafkaConsumerPipe.append_json()` queues a separate non-chunked message for
each Kafka record. In that mode, `JsonReader::_read_one_document()` sets
`_single_document_read` before reading the first message and reports EOF on the
next call, even when more messages remain in the pipe. A task with two small
JSON messages therefore loads only the first, while `DataConsumerGroup` can
advance and commit offsets for both; larger tasks can instead block when the
unread pipe fills. Keep these loads on V1 until the V2 stream path reads
messages until pipe EOF, and cover multiple Kafka messages in one task.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
