Gabriel39 commented on code in PR #69320:
URL: https://github.com/apache/doris/pull/69320#discussion_r4321568885
##########
be/src/exec/operator/file_scan_operator.cpp:
##########
@@ -145,6 +145,29 @@ bool
FileScanLocalState::TEST_should_use_file_scanner_v2(const TQueryOptions& qu
bool FileScanLocalState::should_use_file_scanner_v2(const TQueryOptions&
query_options,
bool is_load,
const
TFileScanRangeParams& scan_params) {
+ if (is_load) {
+ // Load planners omit query session options. Thrift's default makes an
omitted V2
+ // option indistinguishable from false, so ingestion selects by format
instead.
+ switch (scan_params.format_type) {
+ case TFileFormatType::FORMAT_PARQUET:
+ case TFileFormatType::FORMAT_ORC:
+ case TFileFormatType::FORMAT_JSON:
Review Comment:
Fixed in 439f82b98b. Non-line JSON streams now read successive pipe messages
until pipe EOF, while file ranges remain single-document inputs and chunked
HTTP buffers are still joined into one document. Empty pipe reads also avoid
constructing a string from a null buffer.
The two-message and malformed-middle-message tests reproduced the original
first-message-only behavior before the fix. All 19 targeted tests now pass in
the local ASAN/UBSAN harness, including six real-pipe cases covering Kafka
messages, batch boundaries, malformed input, fixed-length HTTP, chunked HTTP,
and empty EOF. Full BE build and cluster regression execution remain for CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
