wenzhenghu opened a new issue, #69326:
URL: https://github.com/apache/doris/issues/69326
### Search before asking
Searched existing issues and PRs for `Zulu`, timezone parsing, and VARIANT.
No duplicate of the BE parsing/file-dependency problem was found. Related PR
#68979 addresses offset validation and explicitly excludes system timezone
database differences; its current diff does not add a `Z` normalization
fallback.
### Version
- Apache Doris master: source analysis at
`349cd15471d7531a687f71db19d5742b3759d4cc`. The relevant `TimezoneUtils`
implementation was rechecked at `cc8a24a13edb46e62359678bb79f79a0a45802ce` and
still contains this gap.
- The regression failure described below was observed on a downstream Doris
branch-3.1 build, not on a freshly built Apache master. **Master applicability
is based on static source analysis; no master runtime reproduction has been
performed yet.**
- The investigated BE environment is Ubuntu 24.04.3 with `tzdata` version
`2025b-0ubuntu0.24.04.1`. `/usr/share/zoneinfo/Zulu` is absent, while
`/usr/share/zoneinfo/Etc/Zulu -> UTC`, `UTC`, and `Etc/UTC` exist.
### What's Wrong?
BE resolves the ISO 8601 / RFC 3339 UTC designator `Z` through a cache alias
that is installed only if the system timezone scan finds the top-level `Zulu`
entry. If that compatibility file is absent, a valid timestamp such as
`2015-01-01T03:47:41Z` can fail datetime conversion even though the system has
UTC timezone data.
This can cause a data-correctness problem for a VARIANT path predefined as
DATETIME: a failed non-strict conversion produces NULL, and the typed-path
conversion can filter that NULL out. A subsequent query of the complete VARIANT
value then omits the field, while the containing row and the load itself can
still succeed.
Observed downstream regression:
`variant_gharchive_schema_nonConcurrent_p2.load.test_variant_index_type_p2`.
- The top-level `created_at` path is predefined as DATETIME.
- Source value: `2015-01-01T03:47:41Z`.
- Expected field in the test output: `"created_at":"2015-01-01 11:47:41"`
(UTC+08:00 representation).
- Actual full VARIANT JSON: the top-level `created_at` field is absent;
other fields remain present.
- Eight loads reported 67,843 rows in total and zero filtered rows. The
failing run used `enable_typed_paths_to_sparse=true`.
- The mismatch was checked after cumulative compaction. There is no
before-compaction comparison, so this report does **not** attribute the defect
specifically to compaction.
The timezone-file state was inspected after the failure; the failing
process's timezone cache was not directly probed. The causal explanation below
is strongly supported by source analysis and the observed environment, but
should be closed out with a controlled reproduction.
### What You Expected?
`Z` denotes UTC and should resolve to zero offset without requiring a legacy
timezone-file alias. These inputs should represent the same instant:
```text
2015-01-01T03:47:41Z
2015-01-01T03:47:41+00:00
```
Converting either input to DATETIME in a UTC+08:00 context should yield
`2015-01-01 11:47:41`. A predefined VARIANT DATETIME path containing this valid
input should remain present and non-NULL.
### How to Reproduce?
The following is a proposed deterministic unit-level reproducer, derived
from the source and **not yet executed**. It can be added temporarily to
`be/test/util/timezone_utils_test.cpp`:
```cpp
TEST(TimezoneUtilsTest, UtcZWithoutSystemTimezoneAliases) {
cctz::time_zone zone;
// Exercise normalization directly, independent of the host's Zulu file.
EXPECT_TRUE(TimezoneUtils::parse_tz_offset_string("Z", zone));
EXPECT_TRUE(TimezoneUtils::parse_tz_offset_string("z", zone));
// Exercise the resolver with no named timezone entries in its cache.
TimezoneUtils::clear_timezone_caches();
TimezoneUtils::load_offsets_to_cache();
EXPECT_TRUE(TimezoneUtils::find_cctz_time_zone("Z", zone));
EXPECT_TRUE(TimezoneUtils::find_cctz_time_zone("z", zone));
// Restore the normal cache for subsequent tests.
TimezoneUtils::load_timezones_to_cache();
}
```
The current implementation is expected to fail these assertions.
Cache-mutating tests must run without concurrent cache users and use the
existing fixture/cleanup conventions; a final regression should also assert
that the returned offset is zero and restore state on early exit.
For end-to-end validation, use an isolated BE test environment whose
timezone directory has UTC data but no top-level `Zulu` entry. Create a VARIANT
column with `created_at` predefined as DATETIME using current master syntax,
load the two timestamp forms above, and check the typed path and full JSON
before and after compaction. Cover ordinary and sparse typed-path storage. Use
a table column to exercise BE conversion rather than relying solely on
FE-folded constant expressions. Do not remove timezone files from a shared or
production host.
### Anything Else?
#### Source-level cause
1.
[`load_timezones_to_cache()`](https://github.com/apache/doris/blob/cc8a24a13edb46e62359678bb79f79a0a45802ce/be/src/util/timezone_utils.cpp#L114)
adds `z` only when `lower_zone_cache_` contains `zulu`. A scanned `Etc/Zulu`
entry has the key `etc/zulu` and does not satisfy this condition.
2. [`find_cctz_time_zone()` and
`normalize_timezone_name()`](https://github.com/apache/doris/blob/cc8a24a13edb46e62359678bb79f79a0a45802ce/be/src/util/timezone_utils.cpp#L162)
cannot recover from that cache miss: normalization recognizes `utc`,
`etc/utc`, and `zulu`, but omits `z`.
3. The strict and permissive datetime parsers resolve a named suffix through
`find_cctz_time_zone()`: see
[`cast_to_datetimev2_impl.hpp`](https://github.com/apache/doris/blob/349cd15471d7531a687f71db19d5742b3759d4cc/be/src/exprs/function/cast/cast_to_datetimev2_impl.hpp#L750).
4. The non-strict datetime serde can return a NULL conversion result without
an error status:
[`data_type_datetimev2_serde.cpp`](https://github.com/apache/doris/blob/349cd15471d7531a687f71db19d5742b3759d4cc/be/src/core/data_type_serde/data_type_datetimev2_serde.cpp#L283).
The VARIANT typed-path promotion can then filter cast NULLs:
[`variant_path_builder.cpp`](https://github.com/apache/doris/blob/349cd15471d7531a687f71db19d5742b3759d4cc/be/src/storage/segment/variant/v2/variant_path_builder.cpp#L1081).
Error behavior of other CAST entry points depends on strictness and should be
tested separately.
#### Suggested fix
Include `lower == "z"` in the existing UTC branch of
`TimezoneUtils::normalize_timezone_name()`. The current fallback already maps
normalized `UTC` to `cctz::utc_time_zone()`, so no system `Zulu` file is
necessary. Keep existing timezone conversion and VARIANT NULL-handling
semantics unchanged.
Add environment-independent BE coverage that fails on the old implementation
even on a host with `Zulu`, plus focused datetime and VARIANT regression
coverage. Existing data already converted to NULL or discarded would not be
recovered automatically by fixing future parsing.
Reference for `Z` semantics: [RFC 3339, section
5.6](https://www.rfc-editor.org/rfc/rfc3339.html#section-5.6).
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://www.apache.org/foundation/policies/conduct).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
