github-actions[bot] commented on code in PR #68649:
URL: https://github.com/apache/doris/pull/68649#discussion_r4140089889
##########
be/src/format_v2/parquet/parquet_statistics.cpp:
##########
@@ -1661,6 +1661,14 @@ bool set_native_page_scalar_min_max(const
tparquet::ColumnIndex& column_index,
}
const auto min_value =
unaligned_load<ValueType>(column_index.min_values[page_idx].data());
const auto max_value =
unaligned_load<ValueType>(column_index.max_values[page_idx].data());
+ if constexpr (std::is_same_v<ValueType, int32_t>) {
+ if (remove_nullable(column_schema.type)->get_primitive_type() ==
TYPE_DATEV2 &&
+ !epoch_days_range_is_representable(min_value, max_value)) {
Review Comment:
[P2] Keep safe DATE pages eligible for pruning. If one page spans the
unrepresentable -719469 and many other pages have valid ranges with
`null_count=0`, non-strict `WHERE d IS NULL` must retain the unsafe page but
can skip the safe ones. Returning false here makes both page-index evaluators
abandon the entire DATE index on that first unsafe page, so the scan reads
every page in the row group. Treat an unsafe DATE page as a pass-all candidate
while continuing to evaluate safe pages, and add a multi-page regression.
##########
be/src/format_v2/orc/orc_reader.cpp:
##########
@@ -1502,6 +1522,37 @@ Status
OrcReader::_init_search_argument_from_local_filters() {
}
try {
+ // SARG can skip rows before DATE decoding detects an invalid ordinal.
File statistics
+ // must prove the selected DATE domains are representable before
enabling SDK pruning.
+ const auto dates_are_representable = [&](const auto& self,
+ const ::orc::Type& type) ->
bool {
+ if (type.getKind() == ::orc::TypeKind::DATE) {
+ const auto stats =
+
_state->reader->getColumnStatistics(cast_set<uint32_t>(type.getColumnId()));
+ if (stats == nullptr) {
+ return false;
+ }
+ if (stats->getNumberOfValues() == 0) {
+ return true;
+ }
+ segment_v2::ZoneMap zone_map;
+ return set_date_zone_map(*stats, &zone_map);
+ }
+ for (uint64_t child = 0; child < type.getSubtypeCount(); ++child) {
+ if (!self(self, *type.getSubtype(child))) {
+ return false;
+ }
+ }
+ return true;
+ };
+ for (const auto column_id : _state->read_columns) {
Review Comment:
[P2] Limit the DATE SARG guard to projected nested leaves. With
`struct<id:int,s:struct<x:int,d:date>>`, a scan of `s.x` under `WHERE id = 7`
can project only `s.x`, so `s.d` is never decoded. If `s.d` has missing or
unsafe min/max statistics, this loop still visits it and returns before
installing the safe `id` SARG. The scan then loses stripe and row-index
pruning, potentially reading the whole file for a selective query. Check the
same projected type IDs used by `includeTypes`, and cover this partial-STRUCT
case.
##########
be/src/core/data_type_serde/data_type_datev2_serde.cpp:
##########
@@ -217,13 +236,12 @@ Status DataTypeDateV2SerDe::write_column_to_arrow(const
IColumn& column, const N
const auto& col_data = static_cast<const ColumnDateV2&>(column).get_data();
auto& date32_builder = assert_cast<arrow::Date32Builder&>(*array_builder);
for (size_t i = start; i < end; ++i) {
- auto daynr = col_data[i].daynr() - date_threshold;
if (null_map && (*null_map)[i]) {
RETURN_IF_ERROR(checkArrowStatus(date32_builder.AppendNull(),
column, *array_builder));
} else {
- RETURN_IF_ERROR(
- checkArrowStatus(date32_builder.Append(cast_set<int,
int64_t, false>(daynr)),
- column, *array_builder));
+ RETURN_IF_ERROR(checkArrowStatus(
+
date32_builder.Append(daynr_to_epoch_days(col_data[i].daynr())), column,
Review Comment:
[P1] Keep new year-zero DATE files readable by the other scanner and older
BEs. This writes `0000-01-01` as -719528 to Parquet (and the ORC writer now
does likewise), but the v1 Parquet/ORC readers still decode through
`date_day_offset_dict` when `enable_file_scanner_v2=false`: its failed
daynr-zero fallback leaves `1900-01-01` as the result. A pre-upgrade v2 Parquet
reader rejects the same ordinal. A newly committed Iceberg or exported file can
therefore return a wrong DATE in a same-build v1 scan or fail on an older BE
during a rolling upgrade. Update the legacy readers and gate the persisted
encoding until older BEs are gone, with cross-reader and mixed-version coverage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
