This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a change to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/doris.git
omit 57998841095 [fix](ci) Update github host from macos-13 to macos-15
(#67747)
add 8d2018bdc72 [Feature](lance) Support multi-vector columns and queries
(#68028)
add 094b27b7c4d [fix](paimon) Read row count estimates from snapshot
metadata (#68155)
add 6b5a6a0de91 [opt](lance) add session cache in fe lance catalog (#68078)
add 8c0b4782979 branch-4.1: [fix](lazy topn) Preserve required slots in
TopN lazy materialization #65530 (#68072)
add ce680bba998 branch-4.1: [fix](fe) Quote unsafe view comments in
exported DDL #67708 (#68076)
add d9ef9e9bf9d branch-4.1: [fix](variable) Apply sql_select_limit to
materialized CTE results #67656 (#68077)
add 41fd07f8215 branch-4.1: [refactor](be) Remove Doris Native file format
(#68158)
add f5a3f3fd20b branch-4.1: [fix](like) Fix LIKE matching values with
extra characters at the end #68133 (#68159)
add 97aa9f48af2 branch-4.1: [fix](cloud) Release warm-up destination on
initialization failure #67924 (#68107)
add b578847894e branch-4.1: [fix](catalog) Unify the jdbc driver_url
security check across jdbc/iceberg/paimon and ALTER CATALOG (#68164)
add 072bed98cc2 branch-4.1: [fix](rbo) Preserve union constant rows across
rewrites #67790 (#68172)
add 73cc2da553b branch-4.1: [fix](test) sync Paimon external paths output
(#68167)
add 8c2803ed866 branch-4.1: [fix](subquery) Preserve generate child
outputs for subqueries #67807 (#68173)
add b98482ddafd branch-4.1: [fix](function) Align interval folding with
runtime #67840 (#68174)
add 80f3a817c3c branch-4.1: [fix](fd) Guard uniform aggregate inference by
participation #67881 (#68175)
add 851858d5702 branch-4.1: [fix](inverted index) Add a norms index
property and a BE config to skip norms on variant paths (#68140)
add 5bcabaa679a branch-4.1: [Fix](regexp) Fix empty constant pattern
matching in regexp (#68179)
add 1aa13de013a branch-4.1: [fix](analysis) Hide internal HAVING helper
outputs #67812 (#68075)
add 8a389b3f546 branch-4.1: [fix](build) add -dead_strip on macOS to fix
dyld cache crash for large BE binary #61742 (#68218)
add 2c8d7c1f347 branch-4.1: [fix](inverted index) Persist empty index
files and stop writing orphan ones #67859 (#68082)
add 47200573335 branch-4.1: [fix](file cache) count evicted bytes only for
blocks that were downloaded #67317 (#68157)
add aeb7e1a0107 branch-4.1: [fix](iceberg) Reject invalid name mapping
instead of reading NULL (#68004) (#68003)
new e3ad599bbfe [fix](ci) Update github host from macos-13 to macos-15
(#67747)
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (57998841095)
\
N -- N -- N refs/heads/branch-4.2 (e3ad599bbfe)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
be/CMakeLists.txt | 5 +
be/src/common/config.cpp | 5 +
be/src/common/config.h | 5 +
be/src/exec/scan/file_scanner.cpp | 8 -
be/src/exec/scan/file_scanner_v2.cpp | 16 +-
be/src/exec/sink/writer/vfile_result_writer.cpp | 9 -
be/src/exprs/function/like.cpp | 168 +-
be/src/exprs/function/like.h | 11 +-
be/src/exprs/function/match.cpp | 5 +
be/src/exprs/function/regexps.h | 4 +
be/src/format/native/native_format.h | 57 -
be/src/format/native/native_reader.cpp | 387 ----
be/src/format/native/native_reader.h | 114 --
be/src/format/transformer/vnative_transformer.cpp | 133 --
be/src/format/transformer/vnative_transformer.h | 65 -
be/src/format_v2/file_reader.h | 1 -
be/src/format_v2/native/native_reader.cpp | 369 ----
be/src/format_v2/native/native_reader.h | 76 -
be/src/format_v2/table/lance_reader.cpp | 85 +-
be/src/format_v2/table_reader.cpp | 8 -
be/src/io/cache/block_file_cache.cpp | 23 +-
be/src/io/cache/block_file_cache.h | 2 +
be/src/service/internal_service.cpp | 6 -
.../storage/compaction/collection_statistics.cpp | 20 +-
be/src/storage/index/index_file_writer.cpp | 24 +-
.../index/inverted/inverted_index_parser.cpp | 7 +
.../storage/index/inverted/inverted_index_parser.h | 7 +
.../index/inverted/inverted_index_writer.cpp | 17 +-
.../storage/index/inverted/query/regexp_query.cpp | 4 +
.../query_v2/regexp_query/regexp_weight.cpp | 4 +
.../index/inverted/similarity/bm25_similarity.cpp | 2 +-
be/src/storage/task/index_builder.cpp | 33 +-
be/src/util/hyperscan_util.cpp | 93 +
.../src/util/hyperscan_util.h | 24 +-
.../data/vec/native/all_types_single_row.native | Bin 1140 -> 0 bytes
be/test/exec/scan/file_scanner_v2_test.cpp | 2 -
be/test/exprs/function/function_like_test.cpp | 248 ++-
be/test/exprs/function/function_match_test.cpp | 21 +
.../exprs/function/function_multi_match_test.cpp | 10 +
.../format/native/native_reader_writer_test.cpp | 1350 --------------
be/test/format_v2/native/native_reader_test.cpp | 450 -----
.../auxiliary.idx | Bin 0 -> 833 bytes
.../28f6d79f-9c00-4135-ba4b-fc8de0b635f3/index.idx | Bin 0 -> 333 bytes
.../auxiliary.idx | Bin 0 -> 829 bytes
.../3b5436e9-4e0b-4db4-abaa-6b34a4ec78ea/index.idx | Bin 0 -> 333 bytes
.../auxiliary.idx | Bin 0 -> 894 bytes
.../54a5446e-5740-4741-9a84-2c3661ca9dac/index.idx | Bin 0 -> 333 bytes
.../auxiliary.idx | Bin 0 -> 819 bytes
.../a8529803-c50d-4ea3-9fcc-302b385b4dc5/index.idx | Bin 0 -> 333 bytes
.../0-98248c30-74fc-4328-955b-93ad3a6572c5.txn | Bin 0 -> 546 bytes
.../1-2bdf0c85-4f23-47ac-b2c4-ae0bc3ba00e2.txn | Bin 0 -> 378 bytes
.../2-2f0814cd-23f1-409f-97a7-f11802ae0000.txn | Bin 0 -> 378 bytes
.../3-85968aa2-1806-42ac-8528-a49e4ac9f753.txn | Bin 0 -> 378 bytes
.../4-f91c85f5-89bd-4744-9cee-4386e21a8734.txn | Bin 0 -> 382 bytes
.../5-8f686607-2ffa-4f37-b5ba-e9afa8748b3b.txn | Bin 0 -> 130 bytes
.../_versions/18446744073709551609.manifest | Bin 0 -> 2188 bytes
.../_versions/18446744073709551610.manifest | Bin 0 -> 2351 bytes
.../_versions/18446744073709551611.manifest | Bin 0 -> 2009 bytes
.../_versions/18446744073709551612.manifest | Bin 0 -> 1675 bytes
.../_versions/18446744073709551613.manifest | Bin 0 -> 1341 bytes
.../_versions/18446744073709551614.manifest | Bin 0 -> 1168 bytes
.../_versions/latest_version_hint.json | 1 +
...000011101010111d600654aa4bf152143f353ff5a.lance | Bin 0 -> 2508 bytes
...11001101011010140ce1040debed650850b651125.lance | Bin 0 -> 2512 bytes
.../0-0d81521a-dd03-451f-96a6-81f50ff5edec.txn | Bin 0 -> 1238 bytes
.../_versions/18446744073709551614.manifest | Bin 0 -> 2553 bytes
.../_versions/latest_version_hint.json | 0
...00100010111001031383243dc838910cb0b1e7016.lance | Bin 0 -> 25966 bytes
.../0-efbaaf9c-8337-4ad8-af21-1b1274f3067e.txn | Bin 0 -> 354 bytes
.../_versions/18446744073709551614.manifest | Bin 0 -> 784 bytes
.../_versions/latest_version_hint.json | 0
...00111101001000111cfaa495a9857e3b9afc0daee.lance | Bin 0 -> 1309 bytes
.../0-d5ed3155-d19a-4ab9-bb2a-9f1aa804ebd9.txn | Bin 0 -> 788 bytes
.../_versions/18446744073709551614.manifest | Bin 0 -> 1652 bytes
.../_versions/latest_version_hint.json | 0
...0011010001111001c07dd49c690bdc468eac853df.lance | Bin 0 -> 3769 bytes
.../auxiliary.idx | Bin 0 -> 329407 bytes
.../7fd93781-4cc1-49fb-ac49-292848aca6fd/index.idx | Bin 0 -> 2390 bytes
.../auxiliary.idx | Bin 0 -> 332361 bytes
.../e6359049-3830-4e8d-93ac-2993bad0837f/index.idx | Bin 0 -> 2390 bytes
.../0-b0e5a832-c817-4b1a-b7fa-b1bb52c0eea9.txn | Bin 0 -> 299 bytes
.../1-5f2b5ed1-20aa-4902-9e98-08608d0f16a9.txn | Bin 0 -> 377 bytes
.../2-b1eca4c0-481a-43d9-b1cd-5d9b69e69c94.txn | Bin 0 -> 128 bytes
.../3-4c0638b9-b006-43ee-b556-81b6d0172760.txn | Bin 0 -> 377 bytes
.../4-626d52d7-f751-41ec-aba6-4c26d7fc984b.txn | Bin 0 -> 128 bytes
.../_versions/18446744073709551610.manifest | Bin 0 -> 1351 bytes
.../_versions/18446744073709551611.manifest | Bin 0 -> 1513 bytes
.../_versions/18446744073709551612.manifest | Bin 0 -> 931 bytes
.../_versions/18446744073709551613.manifest | Bin 0 -> 1093 bytes
.../_versions/18446744073709551614.manifest | Bin 0 -> 675 bytes
.../_versions/latest_version_hint.json | 0
...01010101010111193f4f74d719dca683e57a6fe9f.lance | Bin 0 -> 338911 bytes
...0011001110101011d7b8c41c18a187301e7e13396.lance | Bin 0 -> 338911 bytes
...000010001101010db50ff47508cb03308b653c236.lance | Bin 0 -> 338911 bytes
.../auxiliary.idx | Bin 0 -> 15754 bytes
.../a4de8fca-5bd2-4640-ae9f-e03abcc0b552/index.idx | Bin 0 -> 2388 bytes
.../auxiliary.idx | Bin 0 -> 12800 bytes
.../f243db3b-18a5-4182-8c37-7481361bf628/index.idx | Bin 0 -> 2388 bytes
.../0-c28d1916-321e-48b7-a496-97da54424c30.txn | Bin 0 -> 299 bytes
.../1-cb87fcf7-76e1-4c7e-865c-a191e2f9180a.txn | Bin 0 -> 463 bytes
.../2-b0ac79bc-4068-4cb9-97de-7c27c89e8534.txn | Bin 0 -> 128 bytes
.../3-b7e0ec94-3dc6-4fa7-bc0d-ff28f50e359d.txn | Bin 0 -> 463 bytes
.../4-f99159cc-0549-4cc1-9919-fdecb1aa7f3b.txn | Bin 0 -> 128 bytes
.../_versions/18446744073709551610.manifest | Bin 0 -> 1522 bytes
.../_versions/18446744073709551611.manifest | Bin 0 -> 1770 bytes
.../_versions/18446744073709551612.manifest | Bin 0 -> 1016 bytes
.../_versions/18446744073709551613.manifest | Bin 0 -> 1264 bytes
.../_versions/18446744073709551614.manifest | Bin 0 -> 675 bytes
.../_versions/latest_version_hint.json | 0
...00001011000001059c95649db97fb49f2a95dffb0.lance | Bin 0 -> 338911 bytes
...001001011111111f440e6407e88010674ae90f980.lance | Bin 0 -> 338911 bytes
...1001100011011014b3ee846deabcc7ffc48955629.lance | Bin 0 -> 338911 bytes
.../auxiliary.idx | Bin 0 -> 819 bytes
.../b623b1e5-7461-4a30-880f-394fd4b16617/index.idx | Bin 0 -> 333 bytes
.../0-8e3d8793-6052-45e0-b1ab-43cc9438e25b.txn | Bin 0 -> 224 bytes
.../1-5ddfb769-46bb-4370-91f7-34cc271f6ff7.txn | Bin 0 -> 376 bytes
.../2-5512f591-b4c5-4159-88b5-8973be1a07ca.txn | Bin 0 -> 122 bytes
.../_versions/18446744073709551612.manifest | Bin 0 -> 843 bytes
.../_versions/18446744073709551613.manifest | Bin 0 -> 1016 bytes
.../_versions/18446744073709551614.manifest | Bin 0 -> 525 bytes
.../_versions/latest_version_hint.json | 0
...101000001100111eeb02d4d5c92691235fbf98a81.lance | Bin 0 -> 757 bytes
...00110111101001099e8aa4d8dbb19533ad5d162fe.lance | Bin 0 -> 617 bytes
be/test/format_v2/table/lance_reader_test.cpp | 432 +++++
be/test/format_v2/table_reader_test.cpp | 6 +-
be/test/io/cache/block_file_cache_test.cpp | 89 +
be/test/io/fs/s3_file_writer_test.cpp | 24 +-
be/test/storage/index/index_builder_test.cpp | 153 +-
.../index/inverted/empty_index_file_test.cpp | 253 ++-
.../index/inverted/query/regexp_query_test.cpp | 26 +-
.../index/inverted/query_v2/regexp_query_test.cpp | 17 +-
.../storage/segment/inverted_index_writer_test.cpp | 106 +-
.../index_storage_variant_debug_point_test.cpp | 1 +
.../iceberg/scripts/lance_build_multivector.py | 237 +++
.../scripts/lance_build_preinstalled_catalog.py | 3 +
.../auxiliary.idx | Bin 0 -> 833 bytes
.../28f6d79f-9c00-4135-ba4b-fc8de0b635f3/index.idx | Bin 0 -> 333 bytes
.../auxiliary.idx | Bin 0 -> 829 bytes
.../3b5436e9-4e0b-4db4-abaa-6b34a4ec78ea/index.idx | Bin 0 -> 333 bytes
.../auxiliary.idx | Bin 0 -> 894 bytes
.../54a5446e-5740-4741-9a84-2c3661ca9dac/index.idx | Bin 0 -> 333 bytes
.../auxiliary.idx | Bin 0 -> 819 bytes
.../a8529803-c50d-4ea3-9fcc-302b385b4dc5/index.idx | Bin 0 -> 333 bytes
.../0-98248c30-74fc-4328-955b-93ad3a6572c5.txn | Bin 0 -> 546 bytes
.../1-2bdf0c85-4f23-47ac-b2c4-ae0bc3ba00e2.txn | Bin 0 -> 378 bytes
.../2-2f0814cd-23f1-409f-97a7-f11802ae0000.txn | Bin 0 -> 378 bytes
.../3-85968aa2-1806-42ac-8528-a49e4ac9f753.txn | Bin 0 -> 378 bytes
.../4-f91c85f5-89bd-4744-9cee-4386e21a8734.txn | Bin 0 -> 382 bytes
.../5-8f686607-2ffa-4f37-b5ba-e9afa8748b3b.txn | Bin 0 -> 130 bytes
.../_versions/18446744073709551609.manifest | Bin 0 -> 2188 bytes
.../_versions/18446744073709551610.manifest | Bin 0 -> 2351 bytes
.../_versions/18446744073709551611.manifest | Bin 0 -> 2009 bytes
.../_versions/18446744073709551612.manifest | Bin 0 -> 1675 bytes
.../_versions/18446744073709551613.manifest | Bin 0 -> 1341 bytes
.../_versions/18446744073709551614.manifest | Bin 0 -> 1168 bytes
.../_versions/latest_version_hint.json | 1 +
...000011101010111d600654aa4bf152143f353ff5a.lance | Bin 0 -> 2508 bytes
...11001101011010140ce1040debed650850b651125.lance | Bin 0 -> 2512 bytes
.../0-0d81521a-dd03-451f-96a6-81f50ff5edec.txn | Bin 0 -> 1238 bytes
.../_versions/18446744073709551614.manifest | Bin 0 -> 2553 bytes
.../_versions/latest_version_hint.json | 0
...00100010111001031383243dc838910cb0b1e7016.lance | Bin 0 -> 25966 bytes
.../0-efbaaf9c-8337-4ad8-af21-1b1274f3067e.txn | Bin 0 -> 354 bytes
.../_versions/18446744073709551614.manifest | Bin 0 -> 784 bytes
.../_versions/latest_version_hint.json | 0
...00111101001000111cfaa495a9857e3b9afc0daee.lance | Bin 0 -> 1309 bytes
.../0-d5ed3155-d19a-4ab9-bb2a-9f1aa804ebd9.txn | Bin 0 -> 788 bytes
.../_versions/18446744073709551614.manifest | Bin 0 -> 1652 bytes
.../_versions/latest_version_hint.json | 0
...0011010001111001c07dd49c690bdc468eac853df.lance | Bin 0 -> 3769 bytes
.../auxiliary.idx | Bin 0 -> 329407 bytes
.../7fd93781-4cc1-49fb-ac49-292848aca6fd/index.idx | Bin 0 -> 2390 bytes
.../auxiliary.idx | Bin 0 -> 332361 bytes
.../e6359049-3830-4e8d-93ac-2993bad0837f/index.idx | Bin 0 -> 2390 bytes
.../0-b0e5a832-c817-4b1a-b7fa-b1bb52c0eea9.txn | Bin 0 -> 299 bytes
.../1-5f2b5ed1-20aa-4902-9e98-08608d0f16a9.txn | Bin 0 -> 377 bytes
.../2-b1eca4c0-481a-43d9-b1cd-5d9b69e69c94.txn | Bin 0 -> 128 bytes
.../3-4c0638b9-b006-43ee-b556-81b6d0172760.txn | Bin 0 -> 377 bytes
.../4-626d52d7-f751-41ec-aba6-4c26d7fc984b.txn | Bin 0 -> 128 bytes
.../_versions/18446744073709551610.manifest | Bin 0 -> 1351 bytes
.../_versions/18446744073709551611.manifest | Bin 0 -> 1513 bytes
.../_versions/18446744073709551612.manifest | Bin 0 -> 931 bytes
.../_versions/18446744073709551613.manifest | Bin 0 -> 1093 bytes
.../_versions/18446744073709551614.manifest | Bin 0 -> 675 bytes
.../_versions/latest_version_hint.json | 0
...01010101010111193f4f74d719dca683e57a6fe9f.lance | Bin 0 -> 338911 bytes
...0011001110101011d7b8c41c18a187301e7e13396.lance | Bin 0 -> 338911 bytes
...000010001101010db50ff47508cb03308b653c236.lance | Bin 0 -> 338911 bytes
.../auxiliary.idx | Bin 0 -> 15754 bytes
.../a4de8fca-5bd2-4640-ae9f-e03abcc0b552/index.idx | Bin 0 -> 2388 bytes
.../auxiliary.idx | Bin 0 -> 12800 bytes
.../f243db3b-18a5-4182-8c37-7481361bf628/index.idx | Bin 0 -> 2388 bytes
.../0-c28d1916-321e-48b7-a496-97da54424c30.txn | Bin 0 -> 299 bytes
.../1-cb87fcf7-76e1-4c7e-865c-a191e2f9180a.txn | Bin 0 -> 463 bytes
.../2-b0ac79bc-4068-4cb9-97de-7c27c89e8534.txn | Bin 0 -> 128 bytes
.../3-b7e0ec94-3dc6-4fa7-bc0d-ff28f50e359d.txn | Bin 0 -> 463 bytes
.../4-f99159cc-0549-4cc1-9919-fdecb1aa7f3b.txn | Bin 0 -> 128 bytes
.../_versions/18446744073709551610.manifest | Bin 0 -> 1522 bytes
.../_versions/18446744073709551611.manifest | Bin 0 -> 1770 bytes
.../_versions/18446744073709551612.manifest | Bin 0 -> 1016 bytes
.../_versions/18446744073709551613.manifest | Bin 0 -> 1264 bytes
.../_versions/18446744073709551614.manifest | Bin 0 -> 675 bytes
.../_versions/latest_version_hint.json | 0
...00001011000001059c95649db97fb49f2a95dffb0.lance | Bin 0 -> 338911 bytes
...001001011111111f440e6407e88010674ae90f980.lance | Bin 0 -> 338911 bytes
...1001100011011014b3ee846deabcc7ffc48955629.lance | Bin 0 -> 338911 bytes
.../auxiliary.idx | Bin 0 -> 819 bytes
.../b623b1e5-7461-4a30-880f-394fd4b16617/index.idx | Bin 0 -> 333 bytes
.../0-8e3d8793-6052-45e0-b1ab-43cc9438e25b.txn | Bin 0 -> 224 bytes
.../1-5ddfb769-46bb-4370-91f7-34cc271f6ff7.txn | Bin 0 -> 376 bytes
.../2-5512f591-b4c5-4159-88b5-8973be1a07ca.txn | Bin 0 -> 122 bytes
.../_versions/18446744073709551612.manifest | Bin 0 -> 843 bytes
.../_versions/18446744073709551613.manifest | Bin 0 -> 1016 bytes
.../_versions/18446744073709551614.manifest | Bin 0 -> 525 bytes
.../_versions/latest_version_hint.json | 0
...101000001100111eeb02d4d5c92691235fbf98a81.lance | Bin 0 -> 757 bytes
...00110111101001099e8aa4d8dbb19533ad5d162fe.lance | Bin 0 -> 617 bytes
docs/lance-multivector-search.md | 141 ++
.../main/java/org/apache/doris/common/Config.java | 7 +
.../apache/doris/analysis/InvertedIndexUtil.java | 9 +
.../main/java/org/apache/doris/catalog/Env.java | 22 +-
.../org/apache/doris/cloud/CloudWarmUpJob.java | 42 +-
.../doris/common/util/FileFormatConstants.java | 1 -
.../java/org/apache/doris/common/util/Util.java | 3 -
.../datasource/iceberg/IcebergExternalCatalog.java | 14 +
.../iceberg/IcebergExternalMetaCache.java | 4 +
.../doris/datasource/iceberg/IcebergUtils.java | 38 +-
.../datasource/iceberg/source/IcebergScanNode.java | 8 +-
.../doris/datasource/jdbc/JdbcExternalCatalog.java | 33 +-
.../doris/datasource/lance/LanceCatalogClient.java | 335 ++++
.../doris/datasource/lance/LanceErrorMessages.java | 95 +
.../datasource/lance/LanceExternalCatalog.java | 559 ++----
.../doris/datasource/lance/LanceExternalTable.java | 32 +-
.../datasource/lance/LanceIndexAdmission.java | 12 +-
.../lance/LanceIndexAdmissionSnapshot.java | 35 +-
.../lance/LanceIndexMutationValidator.java | 1 +
.../datasource/lance/LanceNamespaceClient.java | 239 +++
.../lance/LanceSchemaContractBuilder.java | 1 +
.../doris/datasource/lance/LanceTableMetadata.java | 99 -
.../doris/datasource/lance/LanceVectorQuery.java | 259 ---
.../lance/index/LanceDatasetIndexDiscovery.java | 189 ++
.../LanceIndexInspection.java} | 269 +--
.../LanceIndexInspectionExecutor.java} | 10 +-
.../lance/index/LanceIndexSegmentGroup.java | 72 +
.../lance/{ => index}/LanceIndexSegmentInfo.java | 2 +-
.../lance/{ => index}/LancePhysicalIndexEntry.java | 2 +-
.../LanceShowIndexInfo.java} | 8 +-
.../lance/{ => metadata}/LanceFragmentInfo.java | 9 +-
.../lance/{ => metadata}/LanceMetadataLoader.java | 141 +-
.../lance/{ => metadata}/LanceMvccSnapshot.java | 2 +-
.../lance/{ => metadata}/LanceReadOptions.java | 28 +-
.../lance/metadata/LanceSchemaHelper.java | 50 +
.../{ => metadata}/LanceSnapshotResolver.java | 19 +-
.../lance/metadata/LanceTableAccess.java | 33 +-
.../lance/metadata/LanceTableMetadata.java | 128 ++
.../lance/{ => metadata}/LanceTypeConverter.java | 4 +-
.../lance/profile/LanceMetadataMetrics.java | 169 ++
.../lance/source/IndexSegmentSplitPlan.java | 165 --
.../lance/source/LancePredicateConverter.java | 386 +---
.../lance/source/LanceScalarIndexPlanner.java | 30 +-
.../datasource/lance/source/LanceScanNode.java | 443 +----
.../datasource/lance/source/LanceScanPlan.java | 105 ++
.../datasource/lance/source/LanceScanPlanner.java | 394 ++++
.../doris/datasource/lance/source/LanceSplit.java | 116 +-
.../datasource/lance/source/LanceSplitBuilder.java | 117 ++
.../lance/source/LanceSubstraitSerializer.java | 395 ++++
.../{ => storage}/LanceOssStorageProvider.java | 40 +-
.../LancePassThroughStorageProvider.java | 2 +-
.../{ => storage}/LanceS3StorageProvider.java | 38 +-
.../lance/{ => storage}/LanceStorageOptions.java | 23 +-
.../lance/{ => storage}/LanceStorageProvider.java | 2 +-
.../datasource/paimon/PaimonExternalCatalog.java | 17 +
.../datasource/paimon/PaimonExternalTable.java | 53 +-
.../datasource/paimon/PaimonSysExternalTable.java | 14 +-
.../property/fileformat/FileFormatProperties.java | 3 -
.../fileformat/NativeFileFormatProperties.java | 65 -
.../doris/datasource/tvf/source/TVFScanNode.java | 24 +-
.../nereids/load/NereidsLoadScanProvider.java | 5 +-
.../post/materialize/LazyMaterializeTopN.java | 23 +
.../post/materialize/LazySlotPruning.java | 12 +-
.../nereids/rules/analysis/FillUpMissingSlots.java | 4 +-
.../nereids/rules/analysis/NormalizeGenerate.java | 4 +-
.../nereids/rules/rewrite/AddDefaultLimit.java | 8 +-
.../rewrite/MergeOneRowRelationIntoUnion.java | 1 +
.../functions/executable/NumericArithmetic.java | 18 +-
.../trees/plans/commands/ShowIndexCommand.java | 8 +-
.../trees/plans/logical/LogicalAggregate.java | 4 +-
.../plans/physical/PhysicalHashAggregate.java | 4 +-
.../apache/doris/nereids/util/ExpressionUtils.java | 20 +
.../java/org/apache/doris/qe/SessionVariable.java | 8 +-
.../ExternalFileTableValuedFunction.java | 8 +-
.../tablefunction/FileTableValuedFunction.java | 2 +-
.../FullTextSearchTableValuedFunction.java | 16 +-
.../LanceExternalSearchTableValuedFunction.java | 33 +-
.../doris/tablefunction/MetadataGenerator.java | 2 +-
.../doris/tablefunction/S3TableValuedFunction.java | 4 +-
.../VectorSearchTableValuedFunction.java | 297 ++-
.../analysis/InvertedIndexNormsPropertyTest.java | 56 +
.../org/apache/doris/catalog/CreateViewTest.java | 21 +
.../org/apache/doris/cloud/CloudWarmUpJobTest.java | 110 ++
...cebergExternalCatalogDriverUrlSecurityTest.java | 98 +
.../doris/datasource/iceberg/IcebergUtilsTest.java | 28 +-
.../iceberg/source/IcebergScanNodeTest.java | 25 +-
.../datasource/lance/LanceCatalogClientTest.java | 201 ++
.../lance/LanceCatalogLifecycleTest.java | 213 +++
.../LanceExternalCatalogIndexAdmissionTest.java | 31 +-
.../lance/LanceFilesystemCatalogTest.java | 242 +--
.../lance/LanceIndexAdmissionSnapshotTest.java | 14 +-
.../datasource/lance/LanceIndexAdmissionTest.java | 9 +-
.../datasource/lance/LanceSharedSessionTest.java | 250 +++
.../datasource/lance/LanceVectorQueryTest.java | 136 --
.../index/LanceIndexInspectionExecutorTest.java | 218 +++
.../LanceIndexInspectionTest.java} | 133 +-
.../lance/metadata/LanceMetadataLoaderTest.java | 99 +
.../lance/{ => metadata}/LanceSnapshotTest.java | 12 +-
.../{ => metadata}/LanceTypeConverterTest.java | 2 +-
.../lance/profile/LanceMetadataMetricsTest.java | 138 ++
.../datasource/lance/source/LanceScanNodeTest.java | 311 ++--
.../{ => storage}/LanceStorageOptionsTest.java | 2 +-
...PaimonExternalCatalogDriverUrlSecurityTest.java | 112 ++
.../datasource/paimon/PaimonExternalTableTest.java | 8 +-
.../datasource/paimon/PaimonRowCountTest.java | 191 ++
.../datasource/tvf/source/TVFScanNodeTest.java | 2 +-
.../LanceLazyMaterializationUpgradeTest.java | 10 +-
.../nereids/load/NereidsLoadScanProviderTest.java | 13 -
.../post/materialize/LazyMaterializeTopNTest.java | 54 +
.../post/materialize/LazySlotPruningTest.java | 45 +
.../doris/nereids/properties/UniformTest.java | 34 +
.../rules/analysis/FillUpMissingSlotsTest.java | 20 +
.../rules/analysis/NormalizeAggregateTest.java | 37 +-
.../rules/analysis/NormalizeGenerateTest.java | 51 +
.../nereids/rules/rewrite/AddDefaultLimitTest.java | 55 +
.../rewrite/SetOperationOutputMappingTest.java | 44 +
.../executable/NumericArithmeticTest.java | 32 +
.../AlterTableCommandLanceAdmissionTest.java | 10 +-
.../trees/plans/commands/ShowIndexCommandTest.java | 10 +-
.../org/apache/doris/qe/SessionVariablesTest.java | 11 +
.../ExternalFileTableValuedFunctionTest.java | 12 +-
.../FullTextSearchTableValuedFunctionTest.java | 9 +-
...nceIndexEntriesTableValuedFunctionAuthTest.java | 2 +-
.../VectorSearchQueryEncodingTest.java | 238 +++
.../VectorSearchTableValuedFunctionTest.java | 28 +-
.../foundation/security/JdbcDriverUrlSecurity.java | 102 ++
.../security/JdbcDriverUrlSecurityTest.java | 94 +
gensrc/thrift/PaloInternalService.thrift | 3 +
gensrc/thrift/PlanNodes.thrift | 2 +
.../outfile/native/test_outfile_native.out | 25 -
.../lance/test_lance_vector_search_index_types.out | 14 -
.../write/test_paimon_write_external_paths.out | 35 +-
.../test_variant_subcolumn_index_norms.out | 16 +
.../interval_constant_fold_consistency.out | 10 +
.../nereids_p0/test_generate_subquery_output.out | 15 +
.../eliminate_group_by_key_by_uniform.out | 15 +
.../test_having_with_aggregate_function.out | 4 +
.../merge_one_row_relation_into_union.out | 18 +
.../session_variable/test_default_limit.out} | 3 +-
.../test_like_whole_string_match.out | 74 +
.../test_string_function_regexp.out | 35 +-
.../query_p0/topn_lazy/topn_lazy_on_data_model.out | 4 +
.../topNLazyMaterializationUsingIndex.out | 13 +
.../outfile/native/test_outfile_native.groovy | 100 -
.../test_export_variant_10k_columns.groovy | 3 +-
.../lance/test_lance_multivector_coverage.groovy | 226 +++
.../lance/test_lance_multivector_search.groovy | 170 ++
.../test_lance_vector_search_index_types.groovy | 32 +-
.../test_empty_index_file_lifecycle.groovy | 178 ++
.../test_variant_empty_index_file.groovy | 95 +-
.../test_variant_subcolumn_index_norms.groovy | 285 +++
.../interval_constant_fold_consistency.groovy} | 24 +-
.../test_generate_subquery_output.groovy | 51 +
.../eliminate_group_by_key_by_uniform.groovy | 93 +-
.../test_having_with_aggregate_function.groovy | 15 +
.../merge_one_row_relation_into_union.groovy | 25 +
.../session_variable/test_default_limit.groovy | 17 +-
.../test_like_whole_string_match.groovy | 81 +
.../test_string_function_regexp.groovy | 43 +
.../topn_lazy/topn_lazy_on_data_model.groovy | 10 +-
.../topNLazyMaterializationUsingIndex.groovy | 38 +
thirdparty/download-thirdparty.sh | 16 +-
thirdparty/patches/lance-c-0.1.9-pr-83.patch | 1914 ++++++++++++++++++++
380 files changed, 11940 insertions(+), 6599 deletions(-)
delete mode 100644 be/src/format/native/native_format.h
delete mode 100644 be/src/format/native/native_reader.cpp
delete mode 100644 be/src/format/native/native_reader.h
delete mode 100644 be/src/format/transformer/vnative_transformer.cpp
delete mode 100644 be/src/format/transformer/vnative_transformer.h
delete mode 100644 be/src/format_v2/native/native_reader.cpp
delete mode 100644 be/src/format_v2/native/native_reader.h
create mode 100644 be/src/util/hyperscan_util.cpp
copy
regression-test/suites/nereids_rules_p0/merge_one_row_relation/merge_one_row_relation_into_union.groovy
=> be/src/util/hyperscan_util.h (65%)
delete mode 100644 be/test/data/vec/native/all_types_single_row.native
delete mode 100644 be/test/format/native/native_reader_writer_test.cpp
delete mode 100644 be/test/format_v2/native/native_reader_test.cpp
create mode 100644
be/test/format_v2/table/lance/data/multivector.lance/_indices/28f6d79f-9c00-4135-ba4b-fc8de0b635f3/auxiliary.idx
create mode 100644
be/test/format_v2/table/lance/data/multivector.lance/_indices/28f6d79f-9c00-4135-ba4b-fc8de0b635f3/index.idx
create mode 100644
be/test/format_v2/table/lance/data/multivector.lance/_indices/3b5436e9-4e0b-4db4-abaa-6b34a4ec78ea/auxiliary.idx
create mode 100644
be/test/format_v2/table/lance/data/multivector.lance/_indices/3b5436e9-4e0b-4db4-abaa-6b34a4ec78ea/index.idx
create mode 100644
be/test/format_v2/table/lance/data/multivector.lance/_indices/54a5446e-5740-4741-9a84-2c3661ca9dac/auxiliary.idx
create mode 100644
be/test/format_v2/table/lance/data/multivector.lance/_indices/54a5446e-5740-4741-9a84-2c3661ca9dac/index.idx
create mode 100644
be/test/format_v2/table/lance/data/multivector.lance/_indices/a8529803-c50d-4ea3-9fcc-302b385b4dc5/auxiliary.idx
create mode 100644
be/test/format_v2/table/lance/data/multivector.lance/_indices/a8529803-c50d-4ea3-9fcc-302b385b4dc5/index.idx
create mode 100644
be/test/format_v2/table/lance/data/multivector.lance/_transactions/0-98248c30-74fc-4328-955b-93ad3a6572c5.txn
create mode 100644
be/test/format_v2/table/lance/data/multivector.lance/_transactions/1-2bdf0c85-4f23-47ac-b2c4-ae0bc3ba00e2.txn
create mode 100644
be/test/format_v2/table/lance/data/multivector.lance/_transactions/2-2f0814cd-23f1-409f-97a7-f11802ae0000.txn
create mode 100644
be/test/format_v2/table/lance/data/multivector.lance/_transactions/3-85968aa2-1806-42ac-8528-a49e4ac9f753.txn
create mode 100644
be/test/format_v2/table/lance/data/multivector.lance/_transactions/4-f91c85f5-89bd-4744-9cee-4386e21a8734.txn
create mode 100644
be/test/format_v2/table/lance/data/multivector.lance/_transactions/5-8f686607-2ffa-4f37-b5ba-e9afa8748b3b.txn
create mode 100644
be/test/format_v2/table/lance/data/multivector.lance/_versions/18446744073709551609.manifest
create mode 100644
be/test/format_v2/table/lance/data/multivector.lance/_versions/18446744073709551610.manifest
create mode 100644
be/test/format_v2/table/lance/data/multivector.lance/_versions/18446744073709551611.manifest
create mode 100644
be/test/format_v2/table/lance/data/multivector.lance/_versions/18446744073709551612.manifest
create mode 100644
be/test/format_v2/table/lance/data/multivector.lance/_versions/18446744073709551613.manifest
create mode 100644
be/test/format_v2/table/lance/data/multivector.lance/_versions/18446744073709551614.manifest
create mode 100644
be/test/format_v2/table/lance/data/multivector.lance/_versions/latest_version_hint.json
create mode 100644
be/test/format_v2/table/lance/data/multivector.lance/data/010011011000011101010111d600654aa4bf152143f353ff5a.lance
create mode 100644
be/test/format_v2/table/lance/data/multivector.lance/data/01101110111001101011010140ce1040debed650850b651125.lance
create mode 100644
be/test/format_v2/table/lance/data/multivector_dimensions.lance/_transactions/0-0d81521a-dd03-451f-96a6-81f50ff5edec.txn
create mode 100644
be/test/format_v2/table/lance/data/multivector_dimensions.lance/_versions/18446744073709551614.manifest
copy
{docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/all_types.lance
=>
be/test/format_v2/table/lance/data/multivector_dimensions.lance}/_versions/latest_version_hint.json
(100%)
create mode 100644
be/test/format_v2/table/lance/data/multivector_dimensions.lance/data/10000010100100010111001031383243dc838910cb0b1e7016.lance
create mode 100644
be/test/format_v2/table/lance/data/multivector_invalid.lance/_transactions/0-efbaaf9c-8337-4ad8-af21-1b1274f3067e.txn
create mode 100644
be/test/format_v2/table/lance/data/multivector_invalid.lance/_versions/18446744073709551614.manifest
copy
{docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/all_types.lance
=>
be/test/format_v2/table/lance/data/multivector_invalid.lance}/_versions/latest_version_hint.json
(100%)
create mode 100644
be/test/format_v2/table/lance/data/multivector_invalid.lance/data/01001111100111101001000111cfaa495a9857e3b9afc0daee.lance
create mode 100644
be/test/format_v2/table/lance/data/multivector_invalid_types.lance/_transactions/0-d5ed3155-d19a-4ab9-bb2a-9f1aa804ebd9.txn
create mode 100644
be/test/format_v2/table/lance/data/multivector_invalid_types.lance/_versions/18446744073709551614.manifest
copy
{docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/all_types.lance
=>
be/test/format_v2/table/lance/data/multivector_invalid_types.lance}/_versions/latest_version_hint.json
(100%)
create mode 100644
be/test/format_v2/table/lance/data/multivector_invalid_types.lance/data/0001001110011010001111001c07dd49c690bdc468eac853df.lance
create mode 100644
be/test/format_v2/table/lance/data/multivector_ivf_flat.lance/_indices/7fd93781-4cc1-49fb-ac49-292848aca6fd/auxiliary.idx
create mode 100644
be/test/format_v2/table/lance/data/multivector_ivf_flat.lance/_indices/7fd93781-4cc1-49fb-ac49-292848aca6fd/index.idx
create mode 100644
be/test/format_v2/table/lance/data/multivector_ivf_flat.lance/_indices/e6359049-3830-4e8d-93ac-2993bad0837f/auxiliary.idx
create mode 100644
be/test/format_v2/table/lance/data/multivector_ivf_flat.lance/_indices/e6359049-3830-4e8d-93ac-2993bad0837f/index.idx
create mode 100644
be/test/format_v2/table/lance/data/multivector_ivf_flat.lance/_transactions/0-b0e5a832-c817-4b1a-b7fa-b1bb52c0eea9.txn
create mode 100644
be/test/format_v2/table/lance/data/multivector_ivf_flat.lance/_transactions/1-5f2b5ed1-20aa-4902-9e98-08608d0f16a9.txn
create mode 100644
be/test/format_v2/table/lance/data/multivector_ivf_flat.lance/_transactions/2-b1eca4c0-481a-43d9-b1cd-5d9b69e69c94.txn
create mode 100644
be/test/format_v2/table/lance/data/multivector_ivf_flat.lance/_transactions/3-4c0638b9-b006-43ee-b556-81b6d0172760.txn
create mode 100644
be/test/format_v2/table/lance/data/multivector_ivf_flat.lance/_transactions/4-626d52d7-f751-41ec-aba6-4c26d7fc984b.txn
create mode 100644
be/test/format_v2/table/lance/data/multivector_ivf_flat.lance/_versions/18446744073709551610.manifest
create mode 100644
be/test/format_v2/table/lance/data/multivector_ivf_flat.lance/_versions/18446744073709551611.manifest
create mode 100644
be/test/format_v2/table/lance/data/multivector_ivf_flat.lance/_versions/18446744073709551612.manifest
create mode 100644
be/test/format_v2/table/lance/data/multivector_ivf_flat.lance/_versions/18446744073709551613.manifest
create mode 100644
be/test/format_v2/table/lance/data/multivector_ivf_flat.lance/_versions/18446744073709551614.manifest
copy
{docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/482a6218_doris$nested_index
=>
be/test/format_v2/table/lance/data/multivector_ivf_flat.lance}/_versions/latest_version_hint.json
(100%)
create mode 100644
be/test/format_v2/table/lance/data/multivector_ivf_flat.lance/data/00011101001010101010111193f4f74d719dca683e57a6fe9f.lance
create mode 100644
be/test/format_v2/table/lance/data/multivector_ivf_flat.lance/data/0101001000011001110101011d7b8c41c18a187301e7e13396.lance
create mode 100644
be/test/format_v2/table/lance/data/multivector_ivf_flat.lance/data/111011100000010001101010db50ff47508cb03308b653c236.lance
create mode 100644
be/test/format_v2/table/lance/data/multivector_ivf_pq.lance/_indices/a4de8fca-5bd2-4640-ae9f-e03abcc0b552/auxiliary.idx
create mode 100644
be/test/format_v2/table/lance/data/multivector_ivf_pq.lance/_indices/a4de8fca-5bd2-4640-ae9f-e03abcc0b552/index.idx
create mode 100644
be/test/format_v2/table/lance/data/multivector_ivf_pq.lance/_indices/f243db3b-18a5-4182-8c37-7481361bf628/auxiliary.idx
create mode 100644
be/test/format_v2/table/lance/data/multivector_ivf_pq.lance/_indices/f243db3b-18a5-4182-8c37-7481361bf628/index.idx
create mode 100644
be/test/format_v2/table/lance/data/multivector_ivf_pq.lance/_transactions/0-c28d1916-321e-48b7-a496-97da54424c30.txn
create mode 100644
be/test/format_v2/table/lance/data/multivector_ivf_pq.lance/_transactions/1-cb87fcf7-76e1-4c7e-865c-a191e2f9180a.txn
create mode 100644
be/test/format_v2/table/lance/data/multivector_ivf_pq.lance/_transactions/2-b0ac79bc-4068-4cb9-97de-7c27c89e8534.txn
create mode 100644
be/test/format_v2/table/lance/data/multivector_ivf_pq.lance/_transactions/3-b7e0ec94-3dc6-4fa7-bc0d-ff28f50e359d.txn
create mode 100644
be/test/format_v2/table/lance/data/multivector_ivf_pq.lance/_transactions/4-f99159cc-0549-4cc1-9919-fdecb1aa7f3b.txn
create mode 100644
be/test/format_v2/table/lance/data/multivector_ivf_pq.lance/_versions/18446744073709551610.manifest
create mode 100644
be/test/format_v2/table/lance/data/multivector_ivf_pq.lance/_versions/18446744073709551611.manifest
create mode 100644
be/test/format_v2/table/lance/data/multivector_ivf_pq.lance/_versions/18446744073709551612.manifest
create mode 100644
be/test/format_v2/table/lance/data/multivector_ivf_pq.lance/_versions/18446744073709551613.manifest
create mode 100644
be/test/format_v2/table/lance/data/multivector_ivf_pq.lance/_versions/18446744073709551614.manifest
copy
{docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/482a6218_doris$nested_index
=>
be/test/format_v2/table/lance/data/multivector_ivf_pq.lance}/_versions/latest_version_hint.json
(100%)
create mode 100644
be/test/format_v2/table/lance/data/multivector_ivf_pq.lance/data/00011100100001011000001059c95649db97fb49f2a95dffb0.lance
create mode 100644
be/test/format_v2/table/lance/data/multivector_ivf_pq.lance/data/001110010001001011111111f440e6407e88010674ae90f980.lance
create mode 100644
be/test/format_v2/table/lance/data/multivector_ivf_pq.lance/data/1100100111001100011011014b3ee846deabcc7ffc48955629.lance
create mode 100644
be/test/format_v2/table/lance/data/multivector_zero.lance/_indices/b623b1e5-7461-4a30-880f-394fd4b16617/auxiliary.idx
create mode 100644
be/test/format_v2/table/lance/data/multivector_zero.lance/_indices/b623b1e5-7461-4a30-880f-394fd4b16617/index.idx
create mode 100644
be/test/format_v2/table/lance/data/multivector_zero.lance/_transactions/0-8e3d8793-6052-45e0-b1ab-43cc9438e25b.txn
create mode 100644
be/test/format_v2/table/lance/data/multivector_zero.lance/_transactions/1-5ddfb769-46bb-4370-91f7-34cc271f6ff7.txn
create mode 100644
be/test/format_v2/table/lance/data/multivector_zero.lance/_transactions/2-5512f591-b4c5-4159-88b5-8973be1a07ca.txn
create mode 100644
be/test/format_v2/table/lance/data/multivector_zero.lance/_versions/18446744073709551612.manifest
create mode 100644
be/test/format_v2/table/lance/data/multivector_zero.lance/_versions/18446744073709551613.manifest
create mode 100644
be/test/format_v2/table/lance/data/multivector_zero.lance/_versions/18446744073709551614.manifest
copy be/test/format_v2/table/lance/data/{all_types.lance =>
multivector_zero.lance}/_versions/latest_version_hint.json (100%)
create mode 100644
be/test/format_v2/table/lance/data/multivector_zero.lance/data/001011001101000001100111eeb02d4d5c92691235fbf98a81.lance
create mode 100644
be/test/format_v2/table/lance/data/multivector_zero.lance/data/10010111100110111101001099e8aa4d8dbb19533ad5d162fe.lance
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/lance_build_multivector.py
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector.lance/_indices/28f6d79f-9c00-4135-ba4b-fc8de0b635f3/auxiliary.idx
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector.lance/_indices/28f6d79f-9c00-4135-ba4b-fc8de0b635f3/index.idx
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector.lance/_indices/3b5436e9-4e0b-4db4-abaa-6b34a4ec78ea/auxiliary.idx
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector.lance/_indices/3b5436e9-4e0b-4db4-abaa-6b34a4ec78ea/index.idx
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector.lance/_indices/54a5446e-5740-4741-9a84-2c3661ca9dac/auxiliary.idx
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector.lance/_indices/54a5446e-5740-4741-9a84-2c3661ca9dac/index.idx
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector.lance/_indices/a8529803-c50d-4ea3-9fcc-302b385b4dc5/auxiliary.idx
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector.lance/_indices/a8529803-c50d-4ea3-9fcc-302b385b4dc5/index.idx
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector.lance/_transactions/0-98248c30-74fc-4328-955b-93ad3a6572c5.txn
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector.lance/_transactions/1-2bdf0c85-4f23-47ac-b2c4-ae0bc3ba00e2.txn
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector.lance/_transactions/2-2f0814cd-23f1-409f-97a7-f11802ae0000.txn
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector.lance/_transactions/3-85968aa2-1806-42ac-8528-a49e4ac9f753.txn
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector.lance/_transactions/4-f91c85f5-89bd-4744-9cee-4386e21a8734.txn
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector.lance/_transactions/5-8f686607-2ffa-4f37-b5ba-e9afa8748b3b.txn
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector.lance/_versions/18446744073709551609.manifest
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector.lance/_versions/18446744073709551610.manifest
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector.lance/_versions/18446744073709551611.manifest
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector.lance/_versions/18446744073709551612.manifest
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector.lance/_versions/18446744073709551613.manifest
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector.lance/_versions/18446744073709551614.manifest
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector.lance/_versions/latest_version_hint.json
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector.lance/data/010011011000011101010111d600654aa4bf152143f353ff5a.lance
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector.lance/data/01101110111001101011010140ce1040debed650850b651125.lance
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector_dimensions.lance/_transactions/0-0d81521a-dd03-451f-96a6-81f50ff5edec.txn
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector_dimensions.lance/_versions/18446744073709551614.manifest
copy
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/{all_types.lance
=> multivector_dimensions.lance}/_versions/latest_version_hint.json (100%)
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector_dimensions.lance/data/10000010100100010111001031383243dc838910cb0b1e7016.lance
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector_invalid.lance/_transactions/0-efbaaf9c-8337-4ad8-af21-1b1274f3067e.txn
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector_invalid.lance/_versions/18446744073709551614.manifest
copy
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/{all_types.lance
=> multivector_invalid.lance}/_versions/latest_version_hint.json (100%)
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector_invalid.lance/data/01001111100111101001000111cfaa495a9857e3b9afc0daee.lance
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector_invalid_types.lance/_transactions/0-d5ed3155-d19a-4ab9-bb2a-9f1aa804ebd9.txn
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector_invalid_types.lance/_versions/18446744073709551614.manifest
copy
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/{all_types.lance
=> multivector_invalid_types.lance}/_versions/latest_version_hint.json (100%)
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector_invalid_types.lance/data/0001001110011010001111001c07dd49c690bdc468eac853df.lance
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector_ivf_flat.lance/_indices/7fd93781-4cc1-49fb-ac49-292848aca6fd/auxiliary.idx
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector_ivf_flat.lance/_indices/7fd93781-4cc1-49fb-ac49-292848aca6fd/index.idx
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector_ivf_flat.lance/_indices/e6359049-3830-4e8d-93ac-2993bad0837f/auxiliary.idx
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector_ivf_flat.lance/_indices/e6359049-3830-4e8d-93ac-2993bad0837f/index.idx
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector_ivf_flat.lance/_transactions/0-b0e5a832-c817-4b1a-b7fa-b1bb52c0eea9.txn
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector_ivf_flat.lance/_transactions/1-5f2b5ed1-20aa-4902-9e98-08608d0f16a9.txn
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector_ivf_flat.lance/_transactions/2-b1eca4c0-481a-43d9-b1cd-5d9b69e69c94.txn
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector_ivf_flat.lance/_transactions/3-4c0638b9-b006-43ee-b556-81b6d0172760.txn
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector_ivf_flat.lance/_transactions/4-626d52d7-f751-41ec-aba6-4c26d7fc984b.txn
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector_ivf_flat.lance/_versions/18446744073709551610.manifest
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector_ivf_flat.lance/_versions/18446744073709551611.manifest
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector_ivf_flat.lance/_versions/18446744073709551612.manifest
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector_ivf_flat.lance/_versions/18446744073709551613.manifest
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector_ivf_flat.lance/_versions/18446744073709551614.manifest
copy
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/{482a6218_doris$nested_index
=> multivector_ivf_flat.lance}/_versions/latest_version_hint.json (100%)
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector_ivf_flat.lance/data/00011101001010101010111193f4f74d719dca683e57a6fe9f.lance
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector_ivf_flat.lance/data/0101001000011001110101011d7b8c41c18a187301e7e13396.lance
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector_ivf_flat.lance/data/111011100000010001101010db50ff47508cb03308b653c236.lance
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector_ivf_pq.lance/_indices/a4de8fca-5bd2-4640-ae9f-e03abcc0b552/auxiliary.idx
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector_ivf_pq.lance/_indices/a4de8fca-5bd2-4640-ae9f-e03abcc0b552/index.idx
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector_ivf_pq.lance/_indices/f243db3b-18a5-4182-8c37-7481361bf628/auxiliary.idx
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector_ivf_pq.lance/_indices/f243db3b-18a5-4182-8c37-7481361bf628/index.idx
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector_ivf_pq.lance/_transactions/0-c28d1916-321e-48b7-a496-97da54424c30.txn
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector_ivf_pq.lance/_transactions/1-cb87fcf7-76e1-4c7e-865c-a191e2f9180a.txn
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector_ivf_pq.lance/_transactions/2-b0ac79bc-4068-4cb9-97de-7c27c89e8534.txn
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector_ivf_pq.lance/_transactions/3-b7e0ec94-3dc6-4fa7-bc0d-ff28f50e359d.txn
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector_ivf_pq.lance/_transactions/4-f99159cc-0549-4cc1-9919-fdecb1aa7f3b.txn
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector_ivf_pq.lance/_versions/18446744073709551610.manifest
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector_ivf_pq.lance/_versions/18446744073709551611.manifest
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector_ivf_pq.lance/_versions/18446744073709551612.manifest
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector_ivf_pq.lance/_versions/18446744073709551613.manifest
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector_ivf_pq.lance/_versions/18446744073709551614.manifest
copy
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/{482a6218_doris$nested_index
=> multivector_ivf_pq.lance}/_versions/latest_version_hint.json (100%)
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector_ivf_pq.lance/data/00011100100001011000001059c95649db97fb49f2a95dffb0.lance
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector_ivf_pq.lance/data/001110010001001011111111f440e6407e88010674ae90f980.lance
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector_ivf_pq.lance/data/1100100111001100011011014b3ee846deabcc7ffc48955629.lance
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector_zero.lance/_indices/b623b1e5-7461-4a30-880f-394fd4b16617/auxiliary.idx
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector_zero.lance/_indices/b623b1e5-7461-4a30-880f-394fd4b16617/index.idx
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector_zero.lance/_transactions/0-8e3d8793-6052-45e0-b1ab-43cc9438e25b.txn
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector_zero.lance/_transactions/1-5ddfb769-46bb-4370-91f7-34cc271f6ff7.txn
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector_zero.lance/_transactions/2-5512f591-b4c5-4159-88b5-8973be1a07ca.txn
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector_zero.lance/_versions/18446744073709551612.manifest
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector_zero.lance/_versions/18446744073709551613.manifest
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector_zero.lance/_versions/18446744073709551614.manifest
copy {be/test/format_v2/table/lance/data/all_types.lance =>
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector_zero.lance}/_versions/latest_version_hint.json
(100%)
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector_zero.lance/data/001011001101000001100111eeb02d4d5c92691235fbf98a81.lance
create mode 100644
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/multivector_zero.lance/data/10010111100110111101001099e8aa4d8dbb19533ad5d162fe.lance
create mode 100644 docs/lance-multivector-search.md
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceCatalogClient.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceErrorMessages.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceNamespaceClient.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceTableMetadata.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/datasource/lance/LanceVectorQuery.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/datasource/lance/index/LanceDatasetIndexDiscovery.java
rename
fe/fe-core/src/main/java/org/apache/doris/datasource/lance/{LanceIndexMetadataLoader.java
=> index/LanceIndexInspection.java} (60%)
rename
fe/fe-core/src/main/java/org/apache/doris/datasource/lance/{LanceMetadataReadExecutor.java
=> index/LanceIndexInspectionExecutor.java} (94%)
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/datasource/lance/index/LanceIndexSegmentGroup.java
rename fe/fe-core/src/main/java/org/apache/doris/datasource/lance/{ =>
index}/LanceIndexSegmentInfo.java (98%)
rename fe/fe-core/src/main/java/org/apache/doris/datasource/lance/{ =>
index}/LancePhysicalIndexEntry.java (97%)
rename
fe/fe-core/src/main/java/org/apache/doris/datasource/lance/{LanceLogicalIndex.java
=> index/LanceShowIndexInfo.java} (88%)
rename fe/fe-core/src/main/java/org/apache/doris/datasource/lance/{ =>
metadata}/LanceFragmentInfo.java (89%)
rename fe/fe-core/src/main/java/org/apache/doris/datasource/lance/{ =>
metadata}/LanceMetadataLoader.java (50%)
rename fe/fe-core/src/main/java/org/apache/doris/datasource/lance/{ =>
metadata}/LanceMvccSnapshot.java (97%)
rename fe/fe-core/src/main/java/org/apache/doris/datasource/lance/{ =>
metadata}/LanceReadOptions.java (52%)
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/datasource/lance/metadata/LanceSchemaHelper.java
rename fe/fe-core/src/main/java/org/apache/doris/datasource/lance/{ =>
metadata}/LanceSnapshotResolver.java (79%)
copy
regression-test/suites/nereids_rules_p0/fill_up_missing_slots/test_having_with_aggregate_function.groovy
=>
fe/fe-core/src/main/java/org/apache/doris/datasource/lance/metadata/LanceTableAccess.java
(52%)
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/datasource/lance/metadata/LanceTableMetadata.java
rename fe/fe-core/src/main/java/org/apache/doris/datasource/lance/{ =>
metadata}/LanceTypeConverter.java (98%)
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/datasource/lance/profile/LanceMetadataMetrics.java
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/datasource/lance/source/IndexSegmentSplitPlan.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/datasource/lance/source/LanceScanPlan.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/datasource/lance/source/LanceScanPlanner.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/datasource/lance/source/LanceSplitBuilder.java
create mode 100644
fe/fe-core/src/main/java/org/apache/doris/datasource/lance/source/LanceSubstraitSerializer.java
rename fe/fe-core/src/main/java/org/apache/doris/datasource/lance/{ =>
storage}/LanceOssStorageProvider.java (78%)
rename fe/fe-core/src/main/java/org/apache/doris/datasource/lance/{ =>
storage}/LancePassThroughStorageProvider.java (98%)
rename fe/fe-core/src/main/java/org/apache/doris/datasource/lance/{ =>
storage}/LanceS3StorageProvider.java (84%)
rename fe/fe-core/src/main/java/org/apache/doris/datasource/lance/{ =>
storage}/LanceStorageOptions.java (85%)
rename fe/fe-core/src/main/java/org/apache/doris/datasource/lance/{ =>
storage}/LanceStorageProvider.java (98%)
delete mode 100644
fe/fe-core/src/main/java/org/apache/doris/datasource/property/fileformat/NativeFileFormatProperties.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/analysis/InvertedIndexNormsPropertyTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/datasource/iceberg/IcebergExternalCatalogDriverUrlSecurityTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/datasource/lance/LanceCatalogClientTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/datasource/lance/LanceCatalogLifecycleTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/datasource/lance/LanceSharedSessionTest.java
delete mode 100644
fe/fe-core/src/test/java/org/apache/doris/datasource/lance/LanceVectorQueryTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/datasource/lance/index/LanceIndexInspectionExecutorTest.java
rename
fe/fe-core/src/test/java/org/apache/doris/datasource/lance/{LanceIndexMetadataLoaderTest.java
=> index/LanceIndexInspectionTest.java} (90%)
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/datasource/lance/metadata/LanceMetadataLoaderTest.java
rename fe/fe-core/src/test/java/org/apache/doris/datasource/lance/{ =>
metadata}/LanceSnapshotTest.java (89%)
rename fe/fe-core/src/test/java/org/apache/doris/datasource/lance/{ =>
metadata}/LanceTypeConverterTest.java (99%)
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/datasource/lance/profile/LanceMetadataMetricsTest.java
rename fe/fe-core/src/test/java/org/apache/doris/datasource/lance/{ =>
storage}/LanceStorageOptionsTest.java (99%)
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/datasource/paimon/PaimonExternalCatalogDriverUrlSecurityTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/datasource/paimon/PaimonRowCountTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/nereids/processor/post/materialize/LazyMaterializeTopNTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/nereids/processor/post/materialize/LazySlotPruningTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/nereids/rules/analysis/NormalizeGenerateTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/nereids/rules/rewrite/AddDefaultLimitTest.java
create mode 100644
fe/fe-core/src/test/java/org/apache/doris/tablefunction/VectorSearchQueryEncodingTest.java
create mode 100644
fe/fe-foundation/src/main/java/org/apache/doris/foundation/security/JdbcDriverUrlSecurity.java
create mode 100644
fe/fe-foundation/src/test/java/org/apache/doris/foundation/security/JdbcDriverUrlSecurityTest.java
delete mode 100644
regression-test/data/export_p0/outfile/native/test_outfile_native.out
create mode 100644
regression-test/data/inverted_index_p0/test_variant_subcolumn_index_norms.out
create mode 100644
regression-test/data/nereids_function_p0/scalar_function/interval_constant_fold_consistency.out
create mode 100644
regression-test/data/nereids_p0/test_generate_subquery_output.out
create mode 100644
regression-test/data/nereids_rules_p0/merge_one_row_relation/merge_one_row_relation_into_union.out
copy
regression-test/data/{nereids_rules_p0/fill_up_missing_slots/test_having_with_aggregate_function.out
=> query_p0/session_variable/test_default_limit.out} (59%)
create mode 100644
regression-test/data/query_p0/sql_functions/string_functions/test_like_whole_string_match.out
delete mode 100644
regression-test/suites/export_p0/outfile/native/test_outfile_native.groovy
create mode 100644
regression-test/suites/external_table_p0/lance/test_lance_multivector_coverage.groovy
create mode 100644
regression-test/suites/external_table_p0/lance/test_lance_multivector_search.groovy
create mode 100644
regression-test/suites/inverted_index_p0/test_empty_index_file_lifecycle.groovy
create mode 100644
regression-test/suites/inverted_index_p0/test_variant_subcolumn_index_norms.groovy
copy
regression-test/suites/{nereids_rules_p0/fill_up_missing_slots/test_having_with_aggregate_function.groovy
=>
nereids_function_p0/scalar_function/interval_constant_fold_consistency.groovy}
(60%)
create mode 100644
regression-test/suites/nereids_p0/test_generate_subquery_output.groovy
create mode 100644
regression-test/suites/query_p0/sql_functions/string_functions/test_like_whole_string_match.groovy
create mode 100644 thirdparty/patches/lance-c-0.1.9-pr-83.patch
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
