github-actions[bot] commented on code in PR #65837:
URL: https://github.com/apache/doris/pull/65837#discussion_r3778184193
##########
be/src/exec/scan/scanner_scheduler.cpp:
##########
@@ -176,14 +176,26 @@ void
ScannerScheduler::_scanner_scan(std::shared_ptr<ScannerContext> ctx,
Status status = Status::OK();
bool eos = false;
+ auto append_late_arrival_runtime_filter = [&] {
+ Status rf_status = scanner->try_append_late_arrival_runtime_filter();
+ if (!rf_status.ok()) {
+ LOG(WARNING) << "Failed to append late arrival runtime filter: "
+ << rf_status.to_string();
+ }
+ };
ASSIGN_STATUS_IF_CATCH_EXCEPTION(
RuntimeState* state = ctx->state(); DCHECK(nullptr != state);
// scanner->open may alloc plenty amount of memory(read blocks of
data),
// so better to also check low memory and clear free blocks here.
if (ctx->low_memory_mode()) { ctx->clear_free_blocks(); }
- if (scanner->check_partition_pruned()) { eos = true; }
+ if (scanner->is_pruned_by_runtime_filter()) {
Review Comment:
**[P2] Poll a READY filter before the first scanner preparation**
This initial check sees only runtime filters already copied into the local
pruner. If scan-local `open()` observes the consumer unready and the producer
signals before the first scanner task runs, the pruner stays stale until
`try_append_late_arrival_runtime_filter()`, which is currently first called
after `prepare()`. Every initially admitted `OlapScanner` can therefore clone
contexts, allocate/configure `BlockReader`, build schema/predicates, and even
perform collection-statistics IO for tablets that were already eliminable. The
current tests either seed the pruner directly or publish only after all
prepares have started, so they miss this window. Poll/apply late RFs before
this initial check/prepare for unopened scanners, retain the post-prepare poll
for arrivals during preparation, and test a real RF made READY before
`_scanner_scan()` with zero prepare/open/read calls on pruned scanners. This is
distinct from the existing cleanup and during-prepare threads.
##########
be/src/exec/scan/olap_scanner.cpp:
##########
@@ -808,11 +810,25 @@ Status OlapScanner::_init_return_columns() {
return Status::OK();
}
-bool OlapScanner::check_partition_pruned() const {
- if (!_local_state) {
- return false;
- }
- return
_local_state->is_partition_pruned(_tablet_reader_params.tablet->partition_id());
+bool OlapScanner::is_pruned_by_runtime_filter() const {
+ DCHECK(_local_state != nullptr);
+ auto* olap_local_state = assert_cast<OlapScanLocalState*>(_local_state);
+ return olap_local_state->_is_tablet_pruned_by_runtime_filter(
+ _tablet_reader_params.tablet->partition_id(), _bucket_seq,
_bucket_num);
+}
+
+void OlapScanner::release_unopened_resources() {
Review Comment:
**[P2] Preserve incurred IO stats when abandoning an unopened scanner**
This hook discards file-cache/IO work that may already have happened before
`open()`. `ParallelScannerBuilder::_load()` moves `get_segment_num_rows()`
preload stats into one split's `_initial_file_cache_stats`, while score-runtime
preparation records collection IO in the `BlockReader` stats. The normal
accounting path is `_open_impl()` plus `update_realtime_counters()`/close-side
profile collection. If pruning reaches this hook first, the reader is reset and
`_has_prepared` is cleared, so neither source is published; `_builder_stats`
does not recover the preload totals. Profiles, query resource IO counters, and
workload-policy evaluation then under-report real work. Please transfer these
stats exactly once before abandonment (or account preload work in the builder)
and add a pre-open-pruned test with nonzero preload/collection stats. This is
distinct from the existing cleanup threads, which cover retained inputs rather
than lost accounting.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
