github-actions[bot] commented on code in PR #65955:
URL: https://github.com/apache/doris/pull/65955#discussion_r3654377781
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/source/PaimonScanNode.java:
##########
@@ -92,15 +92,13 @@ public class PaimonScanNode extends FileQueryScanNode {
private static final String DORIS_END_TIMESTAMP = "endTimestamp";
private static final String DORIS_INCREMENTAL_BETWEEN_SCAN_MODE =
"incrementalBetweenScanMode";
private static final String PAIMON_PROPERTY_PREFIX = "paimon.";
- private static final String DORIS_ENABLE_FILE_READER_ASYNC =
"jni.enable_file_reader_async";
- private static final String DORIS_ENABLE_JNI_IO_MANAGER =
"doris.enable_jni_io_manager";
- private static final String DORIS_JNI_IO_MANAGER_TMP_DIR =
"doris.jni_io_manager.tmp_dir";
- private static final String DORIS_JNI_IO_MANAGER_IMPL_CLASS =
"doris.jni_io_manager.impl_class";
+ private static final String DORIS_ENABLE_JNI_IO_MANAGER =
"jni.enable_jni_io_manager";
+ private static final String DORIS_JNI_IO_MANAGER_TMP_DIR =
"jni.io_manager.tmp_dir";
+ private static final String DORIS_JNI_IO_MANAGER_IMPL_CLASS =
"jni.io_manager.impl_class";
private static final List<String> BACKEND_PAIMON_OPTIONS = Arrays.asList(
DORIS_ENABLE_JNI_IO_MANAGER,
DORIS_JNI_IO_MANAGER_TMP_DIR,
Review Comment:
[P1] Preserve the async-reader opt-out
This list now drops `jni.enable_file_reader_async`, and
`PaimonJniScanner.initTable()` also no longer maps the persisted
`paimon.jni.enable_file_reader_async=false` property to
`file-reader-async-threshold=Long.MAX_VALUE` (the behavior introduced by
#65365). Because `AbstractPaimonProperties` excludes all `paimon.jni.*` keys
from Paimon options, the old property is now ignored even on a homogeneous
upgraded cluster. Paimon falls back to its 10 MiB threshold, where qualifying
ORC files create `AsyncRecordReader`s backed by a cached thread pool and an
unbounded batch queue, so catalogs that explicitly disabled async reads
silently regain the resource pressure they opted out of. This is separate from
the IO-manager rename thread: this key is deleted rather than renamed. Please
preserve it as a compatibility alias (or explicitly translate it to the new
table option with defined precedence) and retain an old-catalog upgrade test.
##########
fe/be-java-extensions/paimon-scanner/src/main/java/org/apache/doris/paimon/PaimonJniScanner.java:
##########
@@ -571,7 +568,6 @@ static Optional<Long> parseDataSizeBytes(String value) {
private void initTable() {
Preconditions.checkState(params.containsKey("serialized_table"));
table = PaimonUtils.deserialize(params.get("serialized_table"));
Review Comment:
[P2] Preserve V1 physical batching independently of adaptive output
Removing `buildTableOptions()` changes the absent `read.batch-size` case
differently in the two scanner generations. V1 opens Paimon with
`RuntimeState::batch_size()` (8160 by default), so this helper previously fixed
ORC/Parquet physical batches at 8160 even though adaptive output starts at 32
and later grows; the PR now falls back to 1024, increasing physical batch
turnover for narrow V1 scans as their output grows toward the cap. V2 instead
seeds its 32-row probe before opening Java, so generically restoring the
constructor batch would force a 32-row physical batch there. Please preserve
the V1 behavior or introduce a separate stable physical-reader default for both
paths, let an explicit table/catalog `read.batch-size` take precedence, and add
separate V1/V2 effective-batch tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
