Gabriel39 commented on code in PR #68649:
URL: https://github.com/apache/doris/pull/68649#discussion_r4145837028
##########
be/src/format_v2/orc/orc_reader.cpp:
##########
@@ -1502,6 +1522,52 @@ Status
OrcReader::_init_search_argument_from_local_filters() {
}
try {
+ // SARG can skip rows before DATE decoding detects an invalid ordinal.
File statistics
+ // must prove the selected DATE domains are representable before
enabling SDK pruning.
+ const auto dates_are_representable = [&](const auto& self, const
::orc::Type& type,
+ const std::set<uint64_t>*
projected_ids) -> bool {
+ if (projected_ids != nullptr &&
!projected_ids->contains(type.getColumnId())) {
+ return true;
+ }
+ if (type.getKind() == ::orc::TypeKind::DATE) {
+ const auto stats =
+
_state->reader->getColumnStatistics(cast_set<uint32_t>(type.getColumnId()));
+ if (stats == nullptr) {
+ return false;
+ }
+ if (stats->getNumberOfValues() == 0) {
+ return true;
+ }
+ segment_v2::ZoneMap zone_map;
+ return set_date_zone_map(*stats, &zone_map);
+ }
+ for (uint64_t child = 0; child < type.getSubtypeCount(); ++child) {
+ if (!self(self, *type.getSubtype(child), projected_ids)) {
+ return false;
+ }
+ }
+ return true;
+ };
+ for (const auto column_id : _state->read_columns) {
+ if (is_virtual_column(column_id)) {
+ continue;
+ }
+ const auto& type =
+
*_state->root_type->getSubtype(static_cast<uint64_t>(column_id.value()));
+ const auto* projection = find_request_projection(*_request,
column_id);
+ DORIS_CHECK(projection != nullptr);
+ std::set<uint64_t> projected_ids;
+ const bool partial = has_pruned_projection(*projection);
+ if (partial) {
+ // Match includeTypes exactly: an unread DATE sibling cannot
raise a conversion
+ // error and must not disable pruning for the projected
columns.
+ RETURN_IF_ERROR(collect_projected_type_ids(type, *projection,
&projected_ids));
+ }
+ if (!dates_are_representable(dates_are_representable, type,
+ partial ? &projected_ids : nullptr)) {
+ return Status::OK();
Review Comment:
Fixed in 7c53e836b0b. Projected DATE ranges are now checked per stripe
within the current split. Safe excluded stripes retain SARG pruning; unsafe
stripes remain readable with SDK row-group pruning disabled. The fix also
handles the SDK cached evaluator and keeps active reader options alive.
The new three-stripe regression fails on the previous implementation (zero
safe stripes pruned instead of two) and passes after the fix, while still
reporting the invalid DATE. All 247 selected ASAN tests pass, including the
complete ORC reader suite. Final local FE/BE integration also passes all three
suites: group6, Flight year zero, and Parquet/ORC outfile year zero. The
integration build and transport substitutions are documented in the PR
description.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
