This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a change to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/doris.git
from f6702049866 branch-4.2: [fix](nereids) Match nested access path roots
case-insensitively (#68360)
new 388a540d005 [fix](types) Fix binary value ownership and timestamp
primitives (#68297)
new fb49ac4e52c branch-4.1: [fix](compaction) Avoid repeatedly compacting
large cumulative rowset #64954 (#65601)
new 35209246c04 branch-4.1: [fix](compaction) Count empty rowsets in
approximate compaction score #66610 (#66656)
new 5a9a78fb608 branch-4.1: [fix](metacache) release Iceberg FileIO and
Hudi fsView on cache eviction #66914 (#66913)
new 1231c9ad942 branch-4.1: [fix](be) Avoid rowset map race in backend
tablets scanner (#68306) (#68316)
new 2e954921b0f branch-4.1: [fix](be) Preserve tablet ID in synchronous
file cache writes #68310 (#68347)
new cbf56d29da4 branch-4.1: [fix](be) Reject oversized Unix timestamps
before year narrowing #68115 (#68364)
new 4bfeb55c882 branch-4.1: [fix](case) Improve
`test_tablet_size_semantics` robustness #68337 (#68373)
new 5ab9d43d2cb branch-4.1: [Fix](be) Prevent BE crash on malformed
adapter responses #68247 (#68333)
new be650e11667 branch-4.1: [chore](build) Add simdutf third-party
dependency #68369 (#68391)
new f051ca12cfe branch-4.1: [fix](iceberg) Promote equality-delete keys to
current schema type #68267 (#68383)
new 97976581cc5 branch-4.1: [Enhance](ai_func) Support dedicate embed
properties in AI RESOURCE (#67673) (#68384)
The 12 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
be/cmake/thirdparty.cmake | 1 +
.../cloud/cloud_cumulative_compaction_policy.cpp | 17 +-
be/src/cloud/cloud_delete_task.cpp | 1 +
be/src/cloud/cloud_rowset_builder.cpp | 4 +-
be/src/core/column/column_varbinary.cpp | 20 +
be/src/core/column/column_varbinary.h | 6 +
be/src/core/data_type/data_type_factory.cpp | 4 +
.../data_type_serde/data_type_varbinary_serde.cpp | 80 ++
.../data_type_serde/data_type_varbinary_serde.h | 16 +
be/src/core/field.cpp | 73 +-
be/src/core/field.h | 7 +-
be/src/core/string_buffer.hpp | 7 +-
be/src/core/value/timestamptz_value.cpp | 16 +
be/src/core/value/vdatetime_value.h | 4 +
be/src/exec/common/hash_table/hash_key_type.h | 7 +
.../sink/writer/iceberg/partition_transformers.cpp | 6 +
.../aggregate/aggregate_function_min_max_impl.h | 4 +
be/src/exprs/create_predicate_function.h | 3 +
be/src/exprs/function/ai/ai_adapter.h | 179 +++-
be/src/exprs/function/ai/ai_functions.h | 25 +-
be/src/exprs/function/ai/embed.h | 34 +-
be/src/exprs/function/cast/cast_to_date.h | 6 +-
.../function/cast/cast_to_datetimev2_impl.hpp | 89 +-
be/src/exprs/function/cast/cast_to_string.h | 19 +-
be/src/exprs/function/cast/cast_to_timestamptz.h | 8 +-
be/src/exprs/function/date_time_transforms.h | 2 +-
.../function_date_or_datetime_computation.h | 5 +-
be/src/exprs/function/in.h | 4 +
be/src/exprs/vexpr.cpp | 5 +-
be/src/exprs/vin_predicate.cpp | 4 +-
be/src/format_v2/table/iceberg_reader.cpp | 29 +-
.../information_schema/schema_tablets_scanner.cpp | 161 ++-
be/src/io/cache/block_file_cache.cpp | 2 +-
be/src/io/cache/cached_remote_file_reader.cpp | 3 +-
.../compaction/cumulative_compaction_policy.cpp | 16 +-
be/src/util/raw_value.h | 6 +
be/src/util/timezone_utils.cpp | 138 ++-
be/src/util/timezone_utils.h | 6 +
be/test/ai/ai_adapter_test.cpp | 71 ++
be/test/ai/ai_function_test.cpp | 12 +-
be/test/ai/embed_test.cpp | 244 +++++
be/test/cloud/cloud_compaction_test.cpp | 45 +
.../cloud_cumulative_compaction_policy_test.cpp | 118 ++-
be/test/core/column/column_varbinary_test.cpp | 154 +++
.../core/data_type/data_type_varbinary_test.cpp | 18 +-
.../data_type_serde_varbinary_test.cpp | 69 ++
be/test/core/string_buffer_test.cpp | 9 +
.../exec/common/hash_table/hash_key_type_test.cpp | 10 +
.../writer/iceberg/partition_transformers_test.cpp | 14 +
be/test/exprs/aggregate/agg_min_max_test.cpp | 12 +
be/test/exprs/expr_zonemap_filter_test.cpp | 17 +
.../function/cast/cast_to_string_api_test.cpp | 46 +
.../function/cast/cast_to_timestamptz_test.cpp | 45 +
be/test/exprs/function/function_time_test.cpp | 37 +
be/test/exprs/function/function_varbinary_test.cpp | 15 +
be/test/exprs/vexpr_test.cpp | 24 +
be/test/format_v2/table/iceberg_reader_test.cpp | 102 ++
.../io/cache/cached_remote_file_reader_test.cpp | 50 +
be/test/runtime/timestamptz_value_test.cpp | 133 +++
.../cumulative_compaction_policy_test.cpp | 230 +++-
be/test/util/timezone_utils_test.cpp | 66 +-
dist/LICENSE-dist.txt | 1 +
.../java/org/apache/doris/catalog/AIResource.java | 123 ++-
.../org/apache/doris/common/ThreadPoolManager.java | 26 +-
.../org/apache/doris/common/util/PrintableMap.java | 3 +
.../org/apache/doris/datasource/CatalogMgr.java | 2 -
.../doris/datasource/ExternalMetaCacheMgr.java | 8 +-
.../apache/doris/datasource/SplitAssignment.java | 34 +-
.../doris/datasource/hive/HMSExternalCatalog.java | 273 ++++-
.../doris/datasource/hive/HMSExternalTable.java | 18 +-
.../doris/datasource/hive/IcebergDlaTable.java | 30 +-
.../datasource/hudi/HudiExternalMetaCache.java | 124 ++-
.../datasource/hudi/HudiFsViewCacheValue.java | 112 ++
.../doris/datasource/hudi/source/HudiScanNode.java | 642 ++++++++++--
.../datasource/iceberg/DorisHadoopCatalog.java | 96 ++
.../doris/datasource/iceberg/DorisHiveCatalog.java | 96 ++
.../iceberg/IcebergCatalogResourceTracker.java | 121 +++
.../datasource/iceberg/IcebergExternalCatalog.java | 205 +++-
.../iceberg/IcebergExternalDatabase.java | 11 +-
.../iceberg/IcebergExternalMetaCache.java | 762 ++++++++++++--
.../datasource/iceberg/IcebergExternalTable.java | 88 +-
.../datasource/iceberg/IcebergMetadataOps.java | 1100 +++++++++++---------
.../datasource/iceberg/IcebergRuntimeContext.java | 77 ++
.../datasource/iceberg/IcebergSchemaCacheKey.java | 59 +-
.../iceberg/IcebergSnapshotCacheValue.java | 70 ++
.../iceberg/IcebergSysExternalTable.java | 27 +-
.../datasource/iceberg/IcebergTableCacheValue.java | 210 +++-
.../datasource/iceberg/IcebergTransaction.java | 131 ++-
.../doris/datasource/iceberg/IcebergUtils.java | 237 ++++-
.../iceberg/IcebergWriteSchemaContext.java | 47 +-
.../iceberg/action/BaseIcebergAction.java | 32 +-
.../action/IcebergCherrypickSnapshotAction.java | 3 +-
.../action/IcebergExpireSnapshotsAction.java | 4 +-
.../iceberg/action/IcebergFastForwardAction.java | 4 +-
.../action/IcebergPublishChangesAction.java | 3 +-
.../action/IcebergRewriteDataFilesAction.java | 10 +-
.../action/IcebergRewriteManifestsAction.java | 3 +-
.../action/IcebergRollbackToSnapshotAction.java | 3 +-
.../action/IcebergRollbackToTimestampAction.java | 4 +-
.../action/IcebergSetCurrentSnapshotAction.java | 4 +-
.../iceberg/cache/IcebergManifestCacheLoader.java | 15 +-
.../iceberg/rewrite/RewriteDataFileExecutor.java | 182 ++--
.../iceberg/rewrite/RewriteGroupTask.java | 56 +-
.../datasource/iceberg/source/IcebergScanNode.java | 205 +++-
.../metacache/AbstractExternalMetaCache.java | 3 +-
.../doris/datasource/metacache/MetaCacheEntry.java | 131 ++-
.../datasource/metacache/MetaCacheEntryDef.java | 41 +-
.../property/constants/AIProperties.java | 88 +-
.../IcebergFileSystemMetaStoreProperties.java | 4 +-
.../metastore/IcebergHMSMetaStoreProperties.java | 6 +-
.../org/apache/doris/nereids/StatementContext.java | 60 +-
.../trees/expressions/functions/agg/AIAgg.java | 4 +
.../expressions/functions/agg/CollectSet.java | 11 +
.../trees/expressions/functions/ai/AIFunction.java | 4 +
.../trees/expressions/functions/ai/Embed.java | 17 +-
.../functions/scalar/ArrayContains.java | 5 +
.../functions/scalar/ArrayContainsAll.java | 5 +
.../functions/scalar/ArrayDistinct.java | 1 +
.../functions/scalar/ArrayEnumerateUniq.java | 1 +
.../expressions/functions/scalar/ArrayExcept.java | 5 +
.../{EncodeString.java => ArrayFunctionUtils.java} | 30 +-
.../functions/scalar/ArrayIntersect.java | 1 +
.../functions/scalar/ArrayPosition.java | 1 +
.../expressions/functions/scalar/ArrayRemove.java | 1 +
.../expressions/functions/scalar/ArrayUnion.java | 1 +
.../functions/scalar/ArraysOverlap.java | 1 +
.../expressions/functions/scalar/CountEqual.java | 1 +
.../expressions/literal/TimestampTzLiteral.java | 4 +
.../commands/insert/AbstractInsertExecutor.java | 4 +-
.../insert/BaseExternalTableInsertExecutor.java | 18 +-
.../org/apache/doris/catalog/AIResourceTest.java | 515 +++++++++
.../apache/doris/common/ThreadPoolManagerTest.java | 39 +
.../apache/doris/common/util/PrintableMapTest.java | 7 +
.../apache/doris/datasource/CatalogMgrTest.java | 31 +
.../datasource/ExternalCatalogDeadlockTest.java | 8 +
.../ExternalCatalogRuntimeStateTest.java | 118 +++
.../doris/datasource/FileQueryScanNodeTest.java | 29 +
.../doris/datasource/SplitAssignmentTest.java | 26 +
.../hive/HMSExternalCatalogLifecycleTest.java | 98 ++
.../datasource/hive/HMSExternalTableTest.java | 36 +
.../datasource/hudi/HudiExternalMetaCacheTest.java | 213 ++++
.../datasource/hudi/HudiFsViewCacheValueTest.java | 77 ++
.../hudi/source/HudiBatchFsViewOwnerTest.java | 487 +++++++++
.../datasource/hudi/source/HudiScanNodeTest.java | 119 ++-
.../datasource/iceberg/CreateIcebergTableTest.java | 2 +-
.../datasource/iceberg/DorisHadoopCatalogTest.java | 84 ++
.../datasource/iceberg/DorisHiveCatalogTest.java | 94 ++
.../iceberg/IcebergCatalogResourceTrackerTest.java | 79 ++
.../iceberg/IcebergDDLAndDMLPlanTest.java | 7 +
.../iceberg/IcebergExternalMetaCacheTest.java | 718 ++++++++++++-
.../IcebergExternalTableBranchAndTagTest.java | 7 +
.../iceberg/IcebergExternalTableTest.java | 48 +-
.../datasource/iceberg/IcebergMetadataOpTest.java | 255 ++++-
.../iceberg/IcebergMetadataOpsValidationTest.java | 146 ++-
.../iceberg/IcebergSchemaDisplayTest.java | 45 +
.../iceberg/IcebergSysExternalTableTest.java | 83 +-
.../iceberg/IcebergTableCacheValueTest.java | 380 +++++++
.../datasource/iceberg/IcebergTransactionTest.java | 187 ++--
.../doris/datasource/iceberg/IcebergUtilsTest.java | 83 +-
.../iceberg/IcebergWriteSchemaContextTest.java | 68 +-
.../iceberg/action/BaseIcebergActionTest.java | 116 +++
.../rewrite/RewriteDataFileExecutorTest.java | 169 ++-
.../iceberg/rewrite/RewriteGroupTaskTest.java | 17 +
.../iceberg/source/IcebergScanNodeTest.java | 322 ++++++
.../datasource/metacache/MetaCacheEntryTest.java | 198 +++-
.../apache/doris/nereids/StatementContextTest.java | 24 +
.../functions/VarBinaryCollectionLegalityTest.java | 106 ++
.../literal/TimestampTzLiteralTest.java | 39 +
.../plans/commands/ExecuteActionCommandTest.java | 24 +-
.../insert/IcebergInsertFailureLifecycleTest.java | 149 +++
.../plans/commands/insert/InsertUtilsTest.java | 14 +-
.../types/VarBinaryUnsupportedCollectionTest.java | 62 ++
gensrc/thrift/PaloInternalService.thrift | 15 +-
.../datatype_p0/date/test_unix_timestamp_range.out | 28 +
.../stream_load/test_timestamptz_stream_load.out | 8 +-
.../timestamptz/test_cast_timestamptz.out | 6 +
.../timestamptz/test_timestamptz_binary_output.out | 16 +-
.../tablets/test_tablet_size_semantics.groovy | 6 +
.../date/test_unix_timestamp_range.groovy | 102 ++
.../test_timestamptz_stream_load.groovy | 2 +
.../timestamptz/test_cast_timestamptz.groovy | 14 +-
.../test_timestamptz_binary_output.groovy | 4 +
.../test_timestamptz_historical_offset.groovy | 53 +
.../test_timestamptz_null_string.groovy | 48 +
.../test_timestamptz_output_boundary.groovy | 71 ++
thirdparty/build-thirdparty.sh | 22 +
thirdparty/vars.sh | 7 +
187 files changed, 12385 insertions(+), 1613 deletions(-)
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/datasource/hudi/HudiFsViewCacheValue.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/DorisHadoopCatalog.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/DorisHiveCatalog.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergCatalogResourceTracker.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/datasource/iceberg/IcebergRuntimeContext.java
copy
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/scalar/{EncodeString.java
=> ArrayFunctionUtils.java} (52%)
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/datasource/ExternalCatalogRuntimeStateTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/datasource/hive/HMSExternalCatalogLifecycleTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/datasource/hudi/HudiFsViewCacheValueTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/datasource/hudi/source/HudiBatchFsViewOwnerTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/datasource/iceberg/DorisHadoopCatalogTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/datasource/iceberg/DorisHiveCatalogTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/datasource/iceberg/IcebergCatalogResourceTrackerTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/datasource/iceberg/IcebergTableCacheValueTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/datasource/iceberg/action/BaseIcebergActionTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/nereids/trees/expressions/functions/VarBinaryCollectionLegalityTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/nereids/trees/plans/commands/insert/IcebergInsertFailureLifecycleTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/nereids/types/VarBinaryUnsupportedCollectionTest.java
create mode 100644
regression-test/data/datatype_p0/date/test_unix_timestamp_range.out
create mode 100644
regression-test/suites/datatype_p0/date/test_unix_timestamp_range.groovy
create mode 100644
regression-test/suites/datatype_p0/timestamptz/test_timestamptz_historical_offset.groovy
create mode 100644
regression-test/suites/datatype_p0/timestamptz/test_timestamptz_null_string.groovy
create mode 100644
regression-test/suites/datatype_p0/timestamptz/test_timestamptz_output_boundary.groovy
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
