This is an automated email from the ASF dual-hosted git repository.
Gabriel39 pushed a commit to branch branch-4.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-4.1 by this push:
new 041a7e2af4f [fix](arrow) Reject out-of-range Flight timestamps (#68604)
041a7e2af4f is described below
commit 041a7e2af4fb92b73b315f41b2f5a8d8a73fc970
Author: Gabriel <[email protected]>
AuthorDate: Tue Sep 29 21:57:29 2026 +0800
[fix](arrow) Reject out-of-range Flight timestamps (#68604)
### What problem does this PR solve?
Flight SQL can publish out-of-range timestamps from legacy INT96 date
materialization. For example, a non-null zero DATETIME becomes
`timestamp[us]` with value `-62169984000000000`: Arrow validation
succeeds, but PyArrow scalar conversion raises `OverflowError`.
Validate timestamps in `ArrowFlightArrowBlockConvertor` before
publishing each batch. Check the actual timestamp unit and both
UTC/local calendar bounds for zoned values. Recursively check arrays,
map keys/values, and structs while skipping values masked by a NULL
parent. Reject unsupported timestamps with an error identifying the
field path and block row. Other Arrow consumers and Parquet scan
compatibility remain unchanged.
Related PR: #68596 adds separate UTF-8 validation. This PR is
independent of that change; both checks must be retained when
integrating the shared Flight conversion entry point.
### Testing
- Added seven BE tests covering zero dates, years 0/10000,
seconds/milliseconds/microseconds, valid calendar boundaries, pre-epoch
fractions, timezone offsets, slices/subsequent batches, nested
timestamps, NULL parents, and other consumers.
- Before the fix: four new test groups failed because malformed
timestamps were published; three valid/NULL groups passed.
- After the fix: all 36 tests in
`ArrowFlightTimestampTest.*:DataTypeSerDeArrowTest.*` passed under ASAN.
Recompiled the changed converter and new tests against existing ASAN BE
libraries, with C++ access control enabled. Test dates are constructed
through the public unchecked setter so invalid calendar values reach the
validator.
- Added a Flight JDBC regression suite and a separate HDFS INT96 suite:
four existing group4 fixtures must be read successfully with microsecond
truncation at the upper calendar boundary; the zero-year fixture must
return a range error. Both suites preserve the configured Flight
credentials and avoid the TLS-dependent `connect` shortcut. The HDFS
suite discovers the live FE Flight endpoint through MySQL because
external clusters can override the configured port. The HDFS suite
requires `enableHiveTest` and the external test environment.
- Fixed the nested NULL assertion that caused the P0/Cloud P0 Groovy
`getAt()` failure by consuming the JDBC array directly.
- Clang-format 16 passed for all affected C++ files. Groovy 4.0.19
executed both scripts against mock JDBC connections in 12 checks
covering TLS on/off, successful assertions, missing expected errors, and
unrelated errors. These are script-level checks; the live external
validation below additionally exercises real SQL and HDFS data.
### Live external regression validation
- Used the CI-built FE/BE binaries for `8d1158d3f5` in an isolated local
cluster and rebuilt the regression framework from this branch. This
follow-up changes only test scripts and expected results.
- Before the update: all four affected suites failed locally,
reproducing the stale Flight port and zero-year timestamp failures.
- After the update: all four suites passed, with zero failures and zero
skips: `test_flight_int96_timestamp_range`,
`test_remote_doris_all_types_select`, `test_remote_doris_statistics`,
and `test_remote_doris_table_stats`.
- The configured Flight port intentionally remained incorrect. The INT96
suite discovered the running FE endpoint, consumed all four valid HDFS
fixtures, and asserted the expected error for the invalid fixture.
- Remote Doris success fixtures now use year 0001 as the minimum
timestamp, with matching golden results. A separate assertion preserves
coverage of zero-year rejection through a Remote Doris catalog.
### Release note
Flight SQL now returns a server-side error for timestamps outside the
supported 0001–9999 calendar range instead of publishing values that
fail client conversion. Valid timestamps and NULLs retain their existing
values.
### Check List (For Author)
- Test
- [x] Regression test
- [x] Unit Test
- Behavior changed:
- [x] Yes: reject out-of-range Flight timestamps before publishing a
batch.
- Does this need documentation?
- [x] No.
---
be/src/format/arrow/arrow_block_convertor.cpp | 137 +++++++++++++
.../format/arrow/arrow_flight_timestamp_test.cpp | 220 +++++++++++++++++++++
.../test_remote_doris_all_types_select.out | 4 +-
.../remote_doris/test_remote_doris_statistics.out | 2 +-
.../test_flight_int96_timestamp_range.groovy | 83 ++++++++
.../test_flight_timestamp_range.groovy | 81 ++++++++
.../test_remote_doris_all_types_select.groovy | 15 +-
.../test_remote_doris_statistics.groovy | 3 +-
.../test_remote_doris_table_stats.groovy | 3 +-
9 files changed, 541 insertions(+), 7 deletions(-)
diff --git a/be/src/format/arrow/arrow_block_convertor.cpp
b/be/src/format/arrow/arrow_block_convertor.cpp
index b62ca98cde9..6b2ed5a71e8 100644
--- a/be/src/format/arrow/arrow_block_convertor.cpp
+++ b/be/src/format/arrow/arrow_block_convertor.cpp
@@ -17,6 +17,8 @@
#include "format/arrow/arrow_block_convertor.h"
+#include <arrow/array/array_nested.h>
+#include <arrow/array/array_primitive.h>
#include <arrow/array/builder_base.h>
#include <arrow/array/builder_binary.h>
#include <arrow/array/builder_decimal.h>
@@ -34,6 +36,7 @@
#include <cctz/time_zone.h>
#include <glog/logging.h>
+#include <algorithm>
#include <array>
#include <cstring>
#include <ctime>
@@ -63,6 +66,134 @@ namespace doris {
namespace {
+class FlightTimestampValidator {
+public:
+ Status init(std::shared_ptr<arrow::Array> array, std::string path,
+ const cctz::time_zone& timezone) {
+ _array = std::move(array);
+ _path = std::move(path);
+ switch (_array->type_id()) {
+ case arrow::Type::TIMESTAMP: {
+ const auto& type = static_cast<const
arrow::TimestampType&>(*_array->type());
+ int64_t units_per_second = 1;
+ switch (type.unit()) {
+ case arrow::TimeUnit::SECOND:
+ units_per_second = 1;
+ break;
+ case arrow::TimeUnit::MILLI:
+ units_per_second = 1000;
+ break;
+ case arrow::TimeUnit::MICRO:
+ units_per_second = 1000000;
+ break;
+ case arrow::TimeUnit::NANO:
+ units_per_second = 1000000000;
+ break;
+ }
+ int64_t min_seconds = -62135596800LL;
+ int64_t end_seconds = 253402300800LL;
+ if (!type.timezone().empty()) {
+ // The ordinary Arrow writer has already checked the
schema/timezone binding.
+ const auto local_start = cctz::convert(cctz::civil_second(1,
1, 1), timezone);
+ const auto local_end = cctz::convert(cctz::civil_second(10000,
1, 1), timezone);
+ // Python first constructs UTC and then applies the Arrow
timezone. Both calendar
+ // representations must fit, including offsets at the first
and last supported day.
+ min_seconds =
+ std::max<int64_t>(min_seconds,
local_start.time_since_epoch().count());
+ end_seconds = std::min<int64_t>(end_seconds,
local_end.time_since_epoch().count());
+ }
+ // Nanosecond bounds do not fit int64_t, even though every encoded
value does.
+ _min = static_cast<__int128>(min_seconds) * units_per_second;
+ _end = static_cast<__int128>(end_seconds) * units_per_second;
+ break;
+ }
+ case arrow::Type::LIST: {
+ const auto& list = static_cast<const arrow::ListArray&>(*_array);
+ RETURN_IF_ERROR(add_child(list.values(), _path + "[]", timezone));
+ break;
+ }
+ case arrow::Type::MAP: {
+ const auto& map = static_cast<const arrow::MapArray&>(*_array);
+ RETURN_IF_ERROR(add_child(map.keys(), _path + ".key", timezone));
+ RETURN_IF_ERROR(add_child(map.items(), _path + ".value",
timezone));
+ break;
+ }
+ case arrow::Type::STRUCT: {
+ const auto& structure = static_cast<const
arrow::StructArray&>(*_array);
+ for (int i = 0; i < structure.num_fields(); ++i) {
+ RETURN_IF_ERROR(add_child(structure.field(i),
+ _path + "." +
_array->type()->field(i)->name(),
+ timezone));
+ }
+ break;
+ }
+ default:
+ break;
+ }
+ return Status::OK();
+ }
+
+ Status validate(int64_t start, int64_t end, size_t block_start, int64_t
parent_row = -1) const {
+ if (!has_timestamps()) {
+ return Status::OK();
+ }
+ for (int64_t row = start; row < end; ++row) {
+ // Children of a NULL struct/list/map are not observable, even if
their physical
+ // buffers contain default zero dates. Never validate those masked
values.
+ if (_array->IsNull(row)) {
+ continue;
+ }
+ const int64_t output_row = parent_row < 0 ? row : parent_row;
+ if (_array->type_id() == arrow::Type::TIMESTAMP) {
+ const int64_t value = static_cast<const
arrow::TimestampArray&>(*_array).Value(row);
+ if (value < _min || value >= _end) {
+ return Status::InvalidArgument(
+ "Arrow Flight timestamp in column '{}' at row {}
is outside the "
+ "supported 0001-9999 range (type: {})",
+ _path, block_start + output_row + 1,
_array->type()->ToString());
+ }
+ continue;
+ }
+ int64_t child_start = row;
+ int64_t child_end = row + 1;
+ if (_array->type_id() == arrow::Type::LIST) {
+ const auto& list = static_cast<const
arrow::ListArray&>(*_array);
+ child_start = list.value_offset(row);
+ child_end = child_start + list.value_length(row);
+ } else if (_array->type_id() == arrow::Type::MAP) {
+ const auto& map = static_cast<const arrow::MapArray&>(*_array);
+ child_start = map.value_offset(row);
+ child_end = child_start + map.value_length(row);
+ }
+ for (const auto& child : _children) {
+ RETURN_IF_ERROR(child.validate(child_start, child_end,
block_start, output_row));
+ }
+ }
+ return Status::OK();
+ }
+
+private:
+ bool has_timestamps() const {
+ return _array->type_id() == arrow::Type::TIMESTAMP ||
!_children.empty();
+ }
+
+ Status add_child(std::shared_ptr<arrow::Array> array, std::string path,
+ const cctz::time_zone& timezone) {
+ FlightTimestampValidator child;
+ RETURN_IF_ERROR(child.init(std::move(array), std::move(path),
timezone));
+ if (child.has_timestamps()) {
+ _children.push_back(std::move(child));
+ }
+ return Status::OK();
+ }
+
+ std::shared_ptr<arrow::Array> _array;
+ std::string _path;
+ std::vector<FlightTimestampValidator> _children;
+ __int128 _min = 0;
+ __int128 _end = 0;
+};
+
int hex_value(char c) {
if (c >= '0' && c <= '9') {
return c - '0';
@@ -355,6 +486,12 @@ Status
ArrowFlightArrowBlockConvertor::convert_to_arrow(const Block& block, arro
i + 1,
batch->schema()->field(i)->name(),
status.ToString());
}
+ // Arrow accepts the full int64 timestamp domain; ValidateFull cannot
enforce the calendar
+ // range required by Flight clients. Check the encoded values before
publishing the batch.
+ FlightTimestampValidator validator;
+ RETURN_IF_ERROR(
+ validator.init(batch->column(i),
batch->schema()->field(i)->name(), _timezone));
+ RETURN_IF_ERROR(validator.validate(0, batch->num_rows(), start_row));
}
*result = std::move(batch);
return Status::OK();
diff --git a/be/test/format/arrow/arrow_flight_timestamp_test.cpp
b/be/test/format/arrow/arrow_flight_timestamp_test.cpp
new file mode 100644
index 00000000000..ebcafc04a91
--- /dev/null
+++ b/be/test/format/arrow/arrow_flight_timestamp_test.cpp
@@ -0,0 +1,220 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+#include <arrow/api.h>
+#include <gtest/gtest.h>
+
+#include "core/column/column_nullable.h"
+#include "core/column/column_vector.h"
+#include "core/data_type/data_type_array.h"
+#include "core/data_type/data_type_date_or_datetime_v2.h"
+#include "core/data_type/data_type_map.h"
+#include "core/data_type/data_type_nullable.h"
+#include "core/data_type/data_type_struct.h"
+#include "format/arrow/arrow_block_convertor.h"
+#include "util/timezone_utils.h"
+
+namespace doris {
+namespace {
+using DateTime = DateV2Value<DateTimeV2ValueType>;
+
+DateTime make_datetime(uint16_t year, uint8_t month, uint8_t day, uint8_t
hour, uint8_t minute,
+ uint16_t second, uint32_t microsecond) {
+ DateTime value;
+ // Invalid calendar values must reach the converter to exercise its range
validation.
+ value.unchecked_set_time(year, month, day, hour, minute, second,
microsecond);
+ return value;
+}
+
+Block timestamp_block(int scale, const std::vector<DateTime>& values) {
+ auto column = ColumnDateTimeV2::create();
+ for (const auto& value : values) {
+ column->insert_value(value);
+ }
+ return Block {{std::move(column),
std::make_shared<DataTypeDateTimeV2>(scale), "event_time"}};
+}
+
+class ArrowFlightTimestampTest : public testing::Test {
+protected:
+ static void SetUpTestSuite() { TimezoneUtils::load_timezones_to_cache(); }
+};
+
+TEST_F(ArrowFlightTimestampTest,
RejectsOutOfRangeInEveryUnitWithoutPublishingBatch) {
+ for (int scale : {0, 3, 6}) {
+ for (const auto& value : {DateTime {}, make_datetime(0, 12, 31, 0, 0,
0, 0),
+ make_datetime(10000, 1, 1, 0, 0, 0, 0)}) {
+ SCOPED_TRACE(scale);
+ auto block = timestamp_block(scale, {value});
+ ArrowFlightArrowBlockConvertor flight(block, "UTC",
cctz::utc_time_zone(), true);
+ ASSERT_TRUE(flight.init().ok());
+ const ArrowBlockConvertor& converter = flight;
+ std::shared_ptr<arrow::RecordBatch> batch;
+ const auto status =
+ converter.convert_to_arrow(block,
arrow::default_memory_pool(), &batch);
+ EXPECT_EQ(ErrorCode::INVALID_ARGUMENT, status.code()) << status;
+ EXPECT_NE(std::string::npos,
status.to_string().find("event_time"));
+ EXPECT_NE(std::string::npos, status.to_string().find("row 1"));
+ EXPECT_NE(std::string::npos, status.to_string().find("0001-9999"));
+ EXPECT_EQ(nullptr, batch);
+
+ // Other Arrow consumers retain their existing date semantics.
+ DorisArrowBlockConvertor ordinary(block, "UTC",
cctz::utc_time_zone(), true);
+ ASSERT_TRUE(ordinary.init().ok());
+ ASSERT_TRUE(
+ ordinary.convert_to_arrow(block,
arrow::default_memory_pool(), &batch).ok());
+ EXPECT_TRUE(batch->ValidateFull().ok());
+ }
+ }
+}
+
+TEST_F(ArrowFlightTimestampTest,
PreservesCalendarBoundariesAndPreEpochFractions) {
+ for (int scale : {0, 3, 6}) {
+ const int64_t factor = scale == 0 ? 1 : scale == 3 ? 1000 : 1000000;
+ const uint32_t fraction = scale == 0 ? 0 : scale == 3 ? 999000 :
999999;
+ auto block = timestamp_block(scale, {make_datetime(1, 1, 1, 0, 0, 0,
0),
+ make_datetime(9999, 12, 31, 23,
59, 59, fraction),
+ make_datetime(1969, 12, 31, 23,
59, 59, fraction)});
+ ArrowFlightArrowBlockConvertor converter(block, "UTC",
cctz::utc_time_zone(), true);
+ ASSERT_TRUE(converter.init().ok());
+ std::shared_ptr<arrow::RecordBatch> batch;
+ ASSERT_TRUE(converter.convert_to_arrow(block,
arrow::default_memory_pool(), &batch).ok());
+ const auto& values = static_cast<const
arrow::TimestampArray&>(*batch->column(0));
+ EXPECT_EQ(-62135596800LL * factor, values.Value(0));
+ EXPECT_EQ(253402300800LL * factor - 1, values.Value(1));
+ EXPECT_EQ(-1, values.Value(2));
+ }
+}
+
+TEST_F(ArrowFlightTimestampTest, ChecksSlicesAndSubsequentBatches) {
+ auto block = timestamp_block(6, {make_datetime(2024, 1, 1, 0, 0, 0, 0),
DateTime {}});
+ ArrowFlightArrowBlockConvertor converter(block, "UTC",
cctz::utc_time_zone(), true);
+ ASSERT_TRUE(converter.init().ok());
+ std::shared_ptr<arrow::RecordBatch> batch;
+ ASSERT_TRUE(converter.convert_to_arrow(block,
arrow::default_memory_pool(), &batch, 0, 1).ok());
+ auto previous = batch;
+ const auto status =
+ converter.convert_to_arrow(block, arrow::default_memory_pool(),
&batch, 1, 2);
+ EXPECT_EQ(ErrorCode::INVALID_ARGUMENT, status.code());
+ EXPECT_NE(std::string::npos, status.to_string().find("row 2"));
+ EXPECT_EQ(previous, batch);
+}
+
+TEST_F(ArrowFlightTimestampTest, RejectsNestedTimestampValuesAndMapKeys) {
+ auto datetime = make_nullable(std::make_shared<DataTypeDateTimeV2>(6));
+ const auto invalid = Field::create_field<TYPE_DATETIMEV2>(DateTime {});
+ const auto valid =
Field::create_field<TYPE_DATETIMEV2>(make_datetime(2024, 1, 1, 0, 0, 0, 0));
+ DataTypes types {std::make_shared<DataTypeArray>(datetime),
+ std::make_shared<DataTypeStruct>(DataTypes {datetime},
Strings {"child"}),
+ std::make_shared<DataTypeMap>(datetime, datetime),
+ std::make_shared<DataTypeMap>(datetime, datetime),
+
std::make_shared<DataTypeArray>(std::make_shared<DataTypeStruct>(
+ DataTypes {datetime}, Strings {"child"}))};
+ FieldVector fields {
+ Field::create_field<TYPE_ARRAY>(Array {valid, invalid}),
+ Field::create_field<TYPE_STRUCT>(Struct {invalid}),
+ Field::create_field<TYPE_MAP>(Map
{Field::create_field<TYPE_ARRAY>(Array {valid}),
+
Field::create_field<TYPE_ARRAY>(Array {invalid})}),
+ Field::create_field<TYPE_MAP>(Map
{Field::create_field<TYPE_ARRAY>(Array {invalid}),
+
Field::create_field<TYPE_ARRAY>(Array {valid})}),
+ Field::create_field<TYPE_ARRAY>(
+ Array {Field::create_field<TYPE_STRUCT>(Struct
{invalid})})};
+ for (size_t i = 0; i < types.size(); ++i) {
+ SCOPED_TRACE(types[i]->get_name());
+ auto column = types[i]->create_column();
+ column->insert_default();
+ column->insert(fields[i]);
+ Block block {{std::move(column), types[i], "nested"}};
+ ArrowFlightArrowBlockConvertor converter(block, "UTC",
cctz::utc_time_zone(), true);
+ ASSERT_TRUE(converter.init().ok());
+ std::shared_ptr<arrow::RecordBatch> batch;
+ const auto status =
+ converter.convert_to_arrow(block,
arrow::default_memory_pool(), &batch, 1, 2);
+ EXPECT_EQ(ErrorCode::INVALID_ARGUMENT, status.code()) << status;
+ EXPECT_NE(std::string::npos, status.to_string().find("nested"));
+ EXPECT_NE(std::string::npos, status.to_string().find("row 2"));
+ EXPECT_EQ(nullptr, batch);
+ }
+}
+
+TEST_F(ArrowFlightTimestampTest, ChecksBothUtcAndZonedCalendarBounds) {
+ const std::vector<std::pair<std::string, DateTime>> cases {
+ {"+08:00", make_datetime(1, 1, 1, 0, 0, 0, 0)},
+ {"-08:00", make_datetime(9999, 12, 31, 23, 0, 0, 0)},
+ {"-08:00", make_datetime(0, 12, 31, 23, 0, 0, 0)},
+ {"+08:00", make_datetime(10000, 1, 1, 1, 0, 0, 0)}};
+ for (const auto& [zone, value] : cases) {
+ SCOPED_TRACE(zone);
+ cctz::time_zone timezone;
+ ASSERT_TRUE(TimezoneUtils::find_cctz_time_zone(zone, timezone));
+ auto block = timestamp_block(6, {value});
+ ArrowFlightArrowBlockConvertor converter(block, zone, timezone);
+ ASSERT_TRUE(converter.init().ok());
+ std::shared_ptr<arrow::RecordBatch> batch;
+ const auto status = converter.convert_to_arrow(block,
arrow::default_memory_pool(), &batch);
+ EXPECT_EQ(ErrorCode::INVALID_ARGUMENT, status.code()) << status;
+ EXPECT_EQ(nullptr, batch);
+
+ auto valid = timestamp_block(
+ 6, {make_datetime(1, 1, 2, 0, 0, 0, 0), make_datetime(9999,
12, 30, 23, 0, 0, 0)});
+ ASSERT_TRUE(converter.convert_to_arrow(valid,
arrow::default_memory_pool(), &batch).ok());
+ }
+}
+
+TEST_F(ArrowFlightTimestampTest, NaiveBoundsDoNotDependOnSessionTimezone) {
+ cctz::time_zone timezone;
+ ASSERT_TRUE(TimezoneUtils::find_cctz_time_zone("+08:00", timezone));
+ auto block = timestamp_block(6, {make_datetime(1, 1, 1, 0, 0, 0, 0),
+ make_datetime(9999, 12, 31, 23, 59, 59,
999999)});
+ ArrowFlightArrowBlockConvertor converter(block, "+08:00", timezone, true);
+ ASSERT_TRUE(converter.init().ok());
+ std::shared_ptr<arrow::RecordBatch> batch;
+ ASSERT_TRUE(converter.convert_to_arrow(block,
arrow::default_memory_pool(), &batch).ok());
+ const auto& values = static_cast<const
arrow::TimestampArray&>(*batch->column(0));
+ EXPECT_EQ(-62135596800000000LL, values.Value(0));
+ EXPECT_EQ(253402300799999999LL, values.Value(1));
+}
+
+TEST_F(ArrowFlightTimestampTest, IgnoresTimestampsMaskedByNullParents) {
+ auto datetime = std::make_shared<DataTypeDateTimeV2>(6);
+ const auto invalid = Field::create_field<TYPE_DATETIMEV2>(DateTime {});
+ DataTypes types {
+ datetime, std::make_shared<DataTypeStruct>(DataTypes {datetime},
Strings {"child"}),
+ std::make_shared<DataTypeArray>(datetime),
+ std::make_shared<DataTypeMap>(make_nullable(datetime),
make_nullable(datetime))};
+ FieldVector fields {
+ invalid, Field::create_field<TYPE_STRUCT>(Struct {invalid}),
+ Field::create_field<TYPE_ARRAY>(Array {invalid}),
+ Field::create_field<TYPE_MAP>(Map
{Field::create_field<TYPE_ARRAY>(Array {invalid}),
+
Field::create_field<TYPE_ARRAY>(Array {invalid})})};
+ for (size_t i = 0; i < types.size(); ++i) {
+ auto data = types[i]->create_column();
+ data->insert(fields[i]);
+ auto nulls = ColumnUInt8::create();
+ nulls->insert_value(1);
+ Block block {{ColumnNullable::create(std::move(data),
std::move(nulls)),
+ make_nullable(types[i]), "masked"}};
+ ArrowFlightArrowBlockConvertor converter(block, "UTC",
cctz::utc_time_zone(), true);
+ ASSERT_TRUE(converter.init().ok());
+ std::shared_ptr<arrow::RecordBatch> batch;
+ const auto status = converter.convert_to_arrow(block,
arrow::default_memory_pool(), &batch);
+ ASSERT_TRUE(status.ok()) << status;
+ EXPECT_TRUE(batch->column(0)->IsNull(0));
+ }
+}
+
+} // namespace
+} // namespace doris
diff --git
a/regression-test/data/external_table_p0/remote_doris/test_remote_doris_all_types_select.out
b/regression-test/data/external_table_p0/remote_doris/test_remote_doris_all_types_select.out
index a8823bb027e..70fb18a6da0 100644
---
a/regression-test/data/external_table_p0/remote_doris/test_remote_doris_all_types_select.out
+++
b/regression-test/data/external_table_p0/remote_doris/test_remote_doris_all_types_select.out
@@ -1,12 +1,12 @@
-- This file is automatically generated. You should know what you did if you
want to edit this
-- !sql --
-2025-05-18T01:00 true -128 -32768 -2147483648
-9223372036854775808 -1234567890123456790 -123.456 -123456.789
-123457 -123456789012346 -1234567890123456789012345678 1970-01-01
0000-01-01T00:00 A Hello Hello, Doris! ["apple", "banana",
"orange"] {"Emily":101, "age":25} {"f1":11, "f2":3.14, "f3":"Emily"}
{"k1":"v31","k2":300,"k3":[123,456],"k4":[],"k5":{"i1":"iv1"}}
+2025-05-18T01:00 true -128 -32768 -2147483648
-9223372036854775808 -1234567890123456790 -123.456 -123456.789
-123457 -123456789012346 -1234567890123456789012345678 1970-01-01
0001-01-01T00:00 A Hello Hello, Doris! ["apple", "banana",
"orange"] {"Emily":101, "age":25} {"f1":11, "f2":3.14, "f3":"Emily"}
{"k1":"v31","k2":300,"k3":[123,456],"k4":[],"k5":{"i1":"iv1"}}
2025-05-18T02:00 \N \N \N \N \N \N \N
\N \N \N \N \N \N \N \N \N \N
\N \N \N
2025-05-18T03:00 false 127 32767 2147483647
9223372036854775807 1234567890123456789 123.456 123456.789 123457
123456789012346 1234567890123456789012345678 9999-12-31
9999-12-31T23:59:59 [] {} {"f1":11,
"f2":3.14, "f3":"Emily"} {}
2025-05-18T04:00 true 0 0 0 0 0 0.0
0 0 0 0 2023-10-01 2023-10-01T12:34:56 A
Hello Hello, Doris! ["apple", "banana", "orange"] {"Emily":101, "age":25}
{"f1":11, "f2":3.14, "f3":"Emily"} []
-- !sql --
-2025-05-18T01:00 [1] [-128] [-32768] [-2147483648]
[-9223372036854775808] [-1234567890123456790] [-123.456] [-123456.789]
[-123457] [-123456789012346] [-1234567890123456789012345678]
["0000-01-01"] ["0000-01-01 00:00:00"] ["A"] ["Hello"] ["Hello,
Doris!"]
+2025-05-18T01:00 [1] [-128] [-32768] [-2147483648]
[-9223372036854775808] [-1234567890123456790] [-123.456] [-123456.789]
[-123457] [-123456789012346] [-1234567890123456789012345678]
["0000-01-01"] ["0001-01-01 00:00:00"] ["A"] ["Hello"] ["Hello,
Doris!"]
2025-05-18T02:00 [null] [null] [null] [null] [null] [null] [null]
[null] [null] [null] [null] [null] [null] [null] [null] [null]
2025-05-18T03:00 [0] [127] [32767] [2147483647]
[9223372036854775807] [1234567890123456789] [123.456] [123456.789]
[123457] [123456789012346] [1234567890123456789012345678]
["9999-12-31"] ["9999-12-31 23:59:59"] [""] [""] [""]
2025-05-18T04:00 [1] [0] [0] [0] [0] [0] [0]
[0] [0] [0] [0] ["2023-10-01"] ["2023-10-01 12:34:56"] ["A"]
["Hello"] ["Hello, Doris!"]
diff --git
a/regression-test/data/external_table_p0/remote_doris/test_remote_doris_statistics.out
b/regression-test/data/external_table_p0/remote_doris/test_remote_doris_statistics.out
index eecba5cbcd6..aebd716491e 100644
---
a/regression-test/data/external_table_p0/remote_doris/test_remote_doris_statistics.out
+++
b/regression-test/data/external_table_p0/remote_doris/test_remote_doris_statistics.out
@@ -4,7 +4,7 @@ c_bigint 4 3 1 -9223372036854775808
9223372036854775807 32
c_boolean 4 2 1 0 1 4
c_char 4 2 1 A 2
c_date 4 3 1 1970-01-01 9999-12-31 16
-c_datetime 4 3 1 0000-01-01 00:00:00 9999-12-31
23:59:59 32
+c_datetime 4 3 1 0001-01-01 00:00:00 9999-12-31
23:59:59 32
c_decimal18 4 3 1 -123456789012346 123456789012346
32
c_decimal32 4 3 1 -1234567890123456789012345678
1234567890123456789012345678 64
c_decimal9 4 3 1 -123457 123457 16
diff --git
a/regression-test/suites/arrow_flight_sql_p0/test_flight_int96_timestamp_range.groovy
b/regression-test/suites/arrow_flight_sql_p0/test_flight_int96_timestamp_range.groovy
new file mode 100644
index 00000000000..7cdc5f1bbfe
--- /dev/null
+++
b/regression-test/suites/arrow_flight_sql_p0/test_flight_int96_timestamp_range.groovy
@@ -0,0 +1,83 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+import java.sql.DriverManager
+import java.sql.Types
+import java.time.LocalDateTime
+
+suite("test_flight_int96_timestamp_range",
"arrow_flight_sql,external,hive,tvf,external_docker") {
+ if
(!"true".equalsIgnoreCase(context.config.otherConfigs.get("enableHiveTest"))) {
+ return
+ }
+ def host = context.config.otherConfigs.get("externalEnvIp")
+ def port = context.config.otherConfigs.get("hive2HdfsPort")
+ // External clusters can override the configured Flight port; use the
running FE's endpoint.
+ def frontend = jdbc_sql_return_maparray("SHOW FRONTENDS").find {
+ it.IsMaster.toString().toBoolean() && it.Alive.toString().toBoolean()
+ }
+ assertNotNull(frontend)
+ assertTrue(frontend.ArrowFlightSqlPort.toString().toInteger() > 0)
+ def flightUrl =
"jdbc:arrow-flight-sql://${frontend.Host}:${frontend.ArrowFlightSqlPort}" +
+
"/catalog=${context.dbName}?useServerPrepStmts=false&useSSL=false&useEncryption=false"
+ Class.forName("org.apache.arrow.driver.jdbc.ArrowFlightJdbcDriver")
+ def query = { String file ->
+ """SELECT * FROM HDFS(
+ "uri" =
"hdfs://${host}:${port}/user/doris/tvf_data/test_hdfs_parquet/group4/${file}",
+ "hadoop.username" = "doris", "format" = "parquet") LIMIT 10"""
+ }
+ // Keep the configured Flight identity without the TLS-dependent connect()
wrapper.
+ DriverManager.getConnection(flightUrl,
context.config.otherConfigs.get("extArrowFlightSqlUser"),
+
context.config.otherConfigs.get("extArrowFlightSqlPassword")).withCloseable {
flight ->
+ flight.createStatement().withCloseable { statement ->
+ // Nanosecond fractions in these fixtures truncate to valid
DATETIME(6) values.
+ for (def file :
["part-00000-570d8e52-652d-4892-8bdc-7fa5466ffa69.c000.snappy.parquet",
+
"part-00000-b945dfb5-9982-4f86-b903-dabef99caba1.c000.snappy.parquet",
+
"part-00000-721700d2-26d7-42a3-a8f9-b6601628ccd4.c000.snappy.parquet",
+
"part-00000-afeef968-a917-4d51-a652-e5a4214df453.c000.snappy.parquet"]) {
+ logger.info("Read INT96 Flight fixture: ${file}")
+ statement.executeQuery(query(file)).withCloseable { rows ->
+ assertTrue(rows.next())
+ def metadata = rows.getMetaData()
+ for (int column = 1; column <= metadata.getColumnCount();
++column) {
+ if (metadata.getColumnType(column) == Types.TIMESTAMP)
{
+ assertNotNull(rows.getObject(column,
LocalDateTime.class))
+ }
+ }
+
assertEquals(LocalDateTime.parse("9999-12-31T23:59:59.999999"),
+ rows.getObject(metadata.getColumnCount(),
LocalDateTime.class))
+ assertFalse(rows.next())
+ }
+ }
+ }
+ // Consume the result on the same discovered endpoint so zero-year
values must fail over Flight.
+ try {
+ flight.createStatement().withCloseable { statement ->
+
statement.executeQuery(query("int96_timestamps_nanos_outside_day_range.parquet"))
+ .withCloseable { rows ->
+ while (rows.next()) {
+ for (int column = 1; column <=
rows.getMetaData().getColumnCount(); ++column) {
+ rows.getObject(column)
+ }
+ }
+ }
+ }
+ assertTrue(false, "Expected an out-of-range Flight timestamp
error")
+ } catch (Exception error) {
+ assertTrue(error.toString().contains("outside the supported
0001-9999 range"), error.toString())
+ }
+ }
+}
diff --git
a/regression-test/suites/arrow_flight_sql_p0/test_flight_timestamp_range.groovy
b/regression-test/suites/arrow_flight_sql_p0/test_flight_timestamp_range.groovy
new file mode 100644
index 00000000000..c494bd84684
--- /dev/null
+++
b/regression-test/suites/arrow_flight_sql_p0/test_flight_timestamp_range.groovy
@@ -0,0 +1,81 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+import java.time.LocalDateTime
+
+suite("test_flight_timestamp_range", "arrow_flight_sql") {
+ // Reuse the configured Flight connection so TLS settings cannot skip the
assertions or change credentials.
+ def flight = context.getArrowFlightSqlConnection()
+ def input = "${context.dbName}.flight_timestamp_input"
+ def expectRangeError = { String query ->
+ try {
+ arrow_flight_sql(query)
+ assertTrue(false, "Expected an out-of-range Flight timestamp
error")
+ } catch (Exception error) {
+ assertTrue(error.toString().contains("outside the supported
0001-9999 range"), error.toString())
+ }
+ }
+ arrow_flight_sql "DROP TABLE IF EXISTS ${input}"
+ arrow_flight_sql """CREATE TABLE ${input} (id INT, value STRING)
+ DUPLICATE KEY(id) DISTRIBUTED BY HASH(id) BUCKETS 1
+ PROPERTIES ("replication_num" = "1")"""
+ try {
+ arrow_flight_sql """INSERT INTO ${input} VALUES
+ (1, '0001-01-01 00:00:00.000000'),
+ (2, '9999-12-31 23:59:59.999999'),
+ (3, '1969-12-31 23:59:59.999999'),
+ (4, NULL), (5, '0000-12-31 00:00:00.000000')"""
+ // Check that the source value reaches the date conversion instead of
becoming SQL NULL.
+ assertEquals([['0000-12-31 00:00:00.000000']],
+ arrow_flight_sql("SELECT CAST(CAST(value AS DATETIME(6)) AS
STRING) FROM ${input} WHERE id = 5"))
+ for (def scale : [0, 3, 6]) {
+ expectRangeError("SELECT CAST(value AS DATETIME(${scale})) AS
event_time FROM ${input} WHERE id = 5")
+ }
+ expectRangeError("SELECT CAST('0000-12-31 00:00:00' AS DATETIME(6)) AS
event_time")
+ for (def expression : ["array(CAST(value AS DATETIME(6)))",
+ "named_struct('child', CAST(value AS
DATETIME(6)))",
+ "map('key', CAST(value AS DATETIME(6)))",
+ "map(CAST(value AS DATETIME(6)), 'value')"]) {
+ expectRangeError("SELECT ${expression} AS nested FROM ${input}
WHERE id = 5")
+ }
+ expectRangeError("SELECT CAST(value AS DATETIME(6)) AS event_time FROM
${input} ORDER BY id")
+ flight.createStatement().withCloseable { statement ->
+ statement.executeQuery("SELECT CAST(value AS DATETIME(6)) FROM
${input} WHERE id <= 4 ORDER BY id")
+ .withCloseable { rows ->
+ // Typed access preserves proleptic calendar dates and
microseconds.
+ for (def expected : ["0001-01-01T00:00:00",
"9999-12-31T23:59:59.999999",
+ "1969-12-31T23:59:59.999999"]) {
+ assertTrue(rows.next())
+ assertEquals(LocalDateTime.parse(expected),
rows.getObject(1, LocalDateTime.class))
+ }
+ assertTrue(rows.next())
+ assertNull(rows.getObject(1, LocalDateTime.class))
+ assertFalse(rows.next())
+ }
+ statement.executeQuery("SELECT array(CAST(value AS DATETIME(6)))
FROM ${input} WHERE id = 4")
+ .withCloseable { rows ->
+ assertTrue(rows.next())
+ def values = rows.getArray(1).getArray()
+ assertEquals(1, values.length)
+ assertNull(values[0])
+ assertFalse(rows.next())
+ }
+ }
+ } finally {
+ arrow_flight_sql "DROP TABLE IF EXISTS ${input}"
+ }
+}
diff --git
a/regression-test/suites/external_table_p0/remote_doris/test_remote_doris_all_types_select.groovy
b/regression-test/suites/external_table_p0/remote_doris/test_remote_doris_all_types_select.groovy
index 31530d7bfc1..c6919d62b57 100644
---
a/regression-test/suites/external_table_p0/remote_doris/test_remote_doris_all_types_select.groovy
+++
b/regression-test/suites/external_table_p0/remote_doris/test_remote_doris_all_types_select.groovy
@@ -67,8 +67,9 @@ suite("test_remote_doris_all_types_select",
"p0,external,doris,external_docker,e
);
"""
+ // Successful Flight reads require timestamps within the supported
0001-9999 range.
sql """
- INSERT INTO
`test_remote_doris_all_types_select_db`.`test_remote_doris_all_types_select_t`
values('2025-05-18 01:00:00.000', true, -128, -32768, -2147483648,
-9223372036854775808, -1234567890123456790, -123.456, -123456.789, -123457,
-123456789012346, -1234567890123456789012345678, '1970-01-01', '0000-01-01
00:00:00', 'A', 'Hello', 'Hello, Doris!', '["apple", "banana", "orange"]',
{"Emily":101,"age":25} , {11, 3.14, "Emily"}, '{"k1":"v31", "k2": 300, "k3":
[123, 456], "k4": [], " [...]
+ INSERT INTO
`test_remote_doris_all_types_select_db`.`test_remote_doris_all_types_select_t`
values('2025-05-18 01:00:00.000', true, -128, -32768, -2147483648,
-9223372036854775808, -1234567890123456790, -123.456, -123456.789, -123457,
-123456789012346, -1234567890123456789012345678, '1970-01-01', '0001-01-01
00:00:00', 'A', 'Hello', 'Hello, Doris!', '["apple", "banana", "orange"]',
{"Emily":101,"age":25} , {11, 3.14, "Emily"}, '{"k1":"v31", "k2": 300, "k3":
[123, 456], "k4": [], " [...]
"""
sql """
INSERT INTO
`test_remote_doris_all_types_select_db`.`test_remote_doris_all_types_select_t`
values('2025-05-18 02:00:00.000', NULL, NULL, NULL, NULL, NULL, NULL, NULL,
NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL)
@@ -108,7 +109,7 @@ suite("test_remote_doris_all_types_select",
"p0,external,doris,external_docker,e
"""
sql """
- INSERT INTO
`test_remote_doris_all_types_select_db`.`test_remote_doris_all_types_select_t2`
values('2025-05-18 01:00:00.000', [true], [-128], [-32768], [-2147483648],
[-9223372036854775808], [-1234567890123456790], [-123.456], [-123456.789],
[-123457], [-123456789012346], [-1234567890123456789012345678], ['0000-01-01'],
['0000-01-01 00:00:00'], ['A'], ['Hello'], ['Hello, Doris!'])
+ INSERT INTO
`test_remote_doris_all_types_select_db`.`test_remote_doris_all_types_select_t2`
values('2025-05-18 01:00:00.000', [true], [-128], [-32768], [-2147483648],
[-9223372036854775808], [-1234567890123456790], [-123.456], [-123456.789],
[-123457], [-123456789012346], [-1234567890123456789012345678], ['0000-01-01'],
['0001-01-01 00:00:00'], ['A'], ['Hello'], ['Hello, Doris!'])
"""
sql """
INSERT INTO
`test_remote_doris_all_types_select_db`.`test_remote_doris_all_types_select_t2`
values('2025-05-18 02:00:00.000', [NULL], [NULL], [NULL], [NULL], [NULL],
[NULL], [NULL], [NULL], [NULL], [NULL], [NULL], [NULL], [NULL], [NULL], [NULL],
[NULL])
@@ -172,6 +173,16 @@ suite("test_remote_doris_all_types_select",
"p0,external,doris,external_docker,e
select * from
`test_remote_doris_all_types_select_catalog`.`test_remote_doris_all_types_select_db`.`test_remote_doris_all_types_select_t3`
order by id
"""
+ // Keep zero-year rejection covered separately from successful all-type
round trips.
+ sql """INSERT INTO
test_remote_doris_all_types_select_db.test_remote_doris_all_types_select_t3
+ (id, datetime_0) VALUES ('2025-05-18 02:00:00', '0000-01-01
00:00:00')"""
+ test {
+ sql """SELECT datetime_0 FROM
test_remote_doris_all_types_select_catalog
+
.test_remote_doris_all_types_select_db.test_remote_doris_all_types_select_t3
+ WHERE id = '2025-05-18 02:00:00'"""
+ exception "outside the supported 0001-9999 range"
+ }
+
sql """ DROP DATABASE IF EXISTS test_remote_doris_all_types_select_db """
sql """ DROP CATALOG IF EXISTS
`test_remote_doris_all_types_select_catalog` """
}
diff --git
a/regression-test/suites/external_table_p0/remote_doris/test_remote_doris_statistics.groovy
b/regression-test/suites/external_table_p0/remote_doris/test_remote_doris_statistics.groovy
index 223c294d811..210946f397d 100644
---
a/regression-test/suites/external_table_p0/remote_doris/test_remote_doris_statistics.groovy
+++
b/regression-test/suites/external_table_p0/remote_doris/test_remote_doris_statistics.groovy
@@ -82,8 +82,9 @@ suite("test_remote_doris_statistics",
"p0,external,doris,external_docker,externa
);
"""
+ // Successful Flight reads require timestamps within the supported
0001-9999 range.
sql """
- INSERT INTO
`test_remote_doris_statistics_db`.`test_remote_doris_statistics_t1`
values('2025-05-18 01:00:00.000', true, -128, -32768, -2147483648,
-9223372036854775808, -1234567890123456790, -123.456, -123456.789, -123457,
-123456789012346, -1234567890123456789012345678, '1970-01-01', '0000-01-01
00:00:00', 'A', 'Hello', 'Hello, Doris!', '["apple", "banana", "orange"]',
{"Emily":101,"age":25} , {11, 3.14, "Emily"})
+ INSERT INTO
`test_remote_doris_statistics_db`.`test_remote_doris_statistics_t1`
values('2025-05-18 01:00:00.000', true, -128, -32768, -2147483648,
-9223372036854775808, -1234567890123456790, -123.456, -123456.789, -123457,
-123456789012346, -1234567890123456789012345678, '1970-01-01', '0001-01-01
00:00:00', 'A', 'Hello', 'Hello, Doris!', '["apple", "banana", "orange"]',
{"Emily":101,"age":25} , {11, 3.14, "Emily"})
"""
sql """
INSERT INTO
`test_remote_doris_statistics_db`.`test_remote_doris_statistics_t1`
values('2025-05-18 02:00:00.000', NULL, NULL, NULL, NULL, NULL, NULL, NULL,
NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL)
diff --git
a/regression-test/suites/external_table_p0/remote_doris/test_remote_doris_table_stats.groovy
b/regression-test/suites/external_table_p0/remote_doris/test_remote_doris_table_stats.groovy
index e6de44e005a..9486e888a6d 100644
---
a/regression-test/suites/external_table_p0/remote_doris/test_remote_doris_table_stats.groovy
+++
b/regression-test/suites/external_table_p0/remote_doris/test_remote_doris_table_stats.groovy
@@ -82,8 +82,9 @@ suite("test_remote_doris_table_stats",
"p0,external,doris,external_docker,extern
);
"""
+ // Successful Flight reads require timestamps within the supported
0001-9999 range.
sql """
- INSERT INTO
`test_remote_doris_table_stats_db`.`test_remote_doris_table_stats_t1`
values('2025-05-18 01:00:00.000', true, -128, -32768, -2147483648,
-9223372036854775808, -1234567890123456790, -123.456, -123456.789, -123457,
-123456789012346, -1234567890123456789012345678, '1970-01-01', '0000-01-01
00:00:00', 'A', 'Hello', 'Hello, Doris!', '["apple", "banana", "orange"]',
{"Emily":101,"age":25} , {11, 3.14, "Emily"})
+ INSERT INTO
`test_remote_doris_table_stats_db`.`test_remote_doris_table_stats_t1`
values('2025-05-18 01:00:00.000', true, -128, -32768, -2147483648,
-9223372036854775808, -1234567890123456790, -123.456, -123456.789, -123457,
-123456789012346, -1234567890123456789012345678, '1970-01-01', '0001-01-01
00:00:00', 'A', 'Hello', 'Hello, Doris!', '["apple", "banana", "orange"]',
{"Emily":101,"age":25} , {11, 3.14, "Emily"})
"""
sql """
INSERT INTO
`test_remote_doris_table_stats_db`.`test_remote_doris_table_stats_t1`
values('2025-05-18 02:00:00.000', NULL, NULL, NULL, NULL, NULL, NULL, NULL,
NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
