This is an automated email from the ASF dual-hosted git repository.
yiguolei pushed a commit to branch branch-4.1
in repository https://gitbox.apache.org/repos/asf/doris.git
The following commit(s) were added to refs/heads/branch-4.1 by this push:
new 1f021923b1c [fix](test) Keep Lance Unicode fixture paths portable
(#69124)
1f021923b1c is described below
commit 1f021923b1ce6cd1fb4751a0b48a834de0963432
Author: Gabriel <[email protected]>
AuthorDate: Fri Oct 16 14:09:45 2026 +0800
[fix](test) Keep Lance Unicode fixture paths portable (#69124)
### What problem does this PR solve?
Related PR: #68707
The Unicode Lance branch fixture introduced by #68707 tracks five files
below `tree/dev/`. This causes checkout failures on filesystems that
cannot represent that directory name.
Store these files under the ASCII directory `tree/dev_u1c89/` and
restore the original Unicode object prefix when uploading fixtures to
MinIO. The fixture payloads remain byte-for-byte identical, and the
existing Unicode branch regression coverage is preserved. Update the
fixture checker and regeneration instructions to use the portable Git
layout. Initialization exits on upload or relocation failure instead of
continuing with an incomplete fixture.
### Release note
None
### Check List (For Author)
- Test
- [x] Manual test (details below)
- Behavior changed:
- [x] No. Runtime fixture paths and contents are unchanged.
- Does this need documentation?
- [x] No. Fixture regeneration instructions are updated inline.
Validation:
1. Scanned all tracked paths: no non-ASCII names or literal backslashes
remain. Checked out all 10 files in the Unicode fixture using a fresh
Git index with `core.protectNTFS=true` and `core.protectHFS=true` on
Linux.
2. Ran `lance_build_unicode_branch.py --check` on the renamed fixture
with pylance 12.0.0.
3. Executed the upload commands extracted from `iceberg.yaml.tpl`
against a temporary MinIO instance with the pinned mc image. Ran
initialization twice, injecting a stale object before the second run.
After each run, all 515 object keys and blob hashes matched the original
fixtures exactly; neither the ASCII alias nor the stale object remained.
4. Read the main snapshot and both Unicode branch versions with pylance
12.0.0. Confirmed 8 rows in branch version 1, 12 rows in version 2, and
vector search returning the expected branch-only rows.
5. Passed YAML parsing, shell syntax validation, and `git diff --check`.
Full Doris regression tests and a native Windows checkout were not run
locally.
### Check List (For Reviewer who merge this PR)
- [ ] Confirm the release note
- [ ] Confirm test cases
- [ ] Confirm document
- [ ] Add branch pick label
---
.../thirdparties/docker-compose/iceberg/iceberg.yaml.tpl | 5 ++++-
.../iceberg/scripts/lance_build_unicode_branch.py | 15 +++++++++++----
.../1-1be00bd2-8443-4851-9ab8-4bdcee3ffa5e.txn | Bin
.../1-c1998c6f-6213-4b7e-ae7c-cc00cde3b7de.txn | 0
.../dev_u1c89/_versions/18446744073709551613.manifest | Bin
.../dev_u1c89/_versions/18446744073709551614.manifest | Bin
...10100011000100001111040aa57401ab473a6a921298540.lance | Bin
7 files changed, 15 insertions(+), 5 deletions(-)
diff --git a/docker/thirdparties/docker-compose/iceberg/iceberg.yaml.tpl
b/docker/thirdparties/docker-compose/iceberg/iceberg.yaml.tpl
index f2dddc0a319..a9061003f89 100644
--- a/docker/thirdparties/docker-compose/iceberg/iceberg.yaml.tpl
+++ b/docker/thirdparties/docker-compose/iceberg/iceberg.yaml.tpl
@@ -229,6 +229,8 @@ services:
# an earlier fixture would win over the one just published and point
readers at tables
# that no longer exist. This bucket is not recreated between runs, so a
plain copy would
# merge the two catalogs. --remove makes the prefix match the committed
fixture exactly.
+ # The Unicode branch uses an ASCII directory in Git for portable checkout.
Restore its
+ # object keys after each mirror: Lance resolves the original name recorded
in the manifests.
entrypoint: >
/bin/sh -c "
until (/usr/bin/mc config host add minio http://minio:9000 admin
password) do echo '...waiting...' && sleep 1; done;
@@ -240,7 +242,8 @@ services:
/usr/bin/mc policy set public minio/warehouse;
/usr/bin/mc cp -r /mnt/data/input/minio/warehouse/* minio/warehouse/;
fi;
- /usr/bin/mc mirror --overwrite --remove /mnt/preinstalled_data/lance/
minio/warehouse/lance/;
+ /usr/bin/mc mirror --overwrite --remove /mnt/preinstalled_data/lance/
minio/warehouse/lance/ &&
+ /usr/bin/mc mv --recursive
minio/warehouse/lance/unicode_branch.lance/tree/dev_u1c89/
minio/warehouse/lance/unicode_branch.lance/tree/dev/ || exit 1;
/usr/bin/mc cp -r /mnt/preinstalled_data/iceberg/
minio/warehouse/wh/multi_catalog/;
"
diff --git
a/docker/thirdparties/docker-compose/iceberg/scripts/lance_build_unicode_branch.py
b/docker/thirdparties/docker-compose/iceberg/scripts/lance_build_unicode_branch.py
index 1179ddbe6a9..4b219bb6617 100644
---
a/docker/thirdparties/docker-compose/iceberg/scripts/lance_build_unicode_branch.py
+++
b/docker/thirdparties/docker-compose/iceberg/scripts/lance_build_unicode_branch.py
@@ -37,12 +37,18 @@ created against the fixture's final location (see
lance_build_search_snapshot.py
--storage-option secret_access_key=password --storage-option
region=us-east-1 \\
--storage-option allow_http=true
# then sync tree/ and _refs/branches/ from that location back
+ # store the branch under an ASCII name before committing (Bash syntax)
+ mv preinstalled_data/lance/unicode_branch.lance/tree/$'dev\\u1c89' \\
+ preinstalled_data/lance/unicode_branch.lance/tree/dev_u1c89
python3 lance_build_unicode_branch.py --check preinstalled_data/lance
Build it with pylance 12.0.0, the Lance the FE uses, not the writer pinned in
lance_fixture_requirements.txt: Lance 7 stores a non-ASCII branch directory
percent-encoded
(tree/dev%E1%B2%89/), which Lance 12 does not find, while Lance 12 stores it
as tree/dev\u1c89/.
-The catalog build carries the fixture over as-is. --check runs with any reader.
+The repository stores that directory as tree/dev_u1c89/ so checkout does not
depend on
+filesystem Unicode support. iceberg.yaml.tpl restores the original object keys
after
+uploading to MinIO. The catalog build carries the portable fixture over as-is.
+--check verifies this stored layout and runs with any reader.
"""
import argparse
import shutil
@@ -52,7 +58,8 @@ import lance
import pyarrow as pa
UNICODE_BRANCH_DIR = "unicode_branch.lance"
-BRANCH = "dev"
+BRANCH = "dev\u1c89"
+STORED_BRANCH = "dev_u1c89"
PARENT_URI = "s3://warehouse/lance/unicode_branch.lance"
WRITER_VERSION = "12.0.0"
DIMENSION = 4
@@ -102,8 +109,8 @@ def check(output: Path) -> None:
assert [v["version"] for v in dataset.versions()] == [1],
dataset.versions()
assert row_ids(dataset) == list(range(1, 9)), "main rows differ"
assert list(dataset.branches.list()) == [BRANCH], dataset.branches.list()
- # The branch directory keeps the name as is; Lance 12 does not find a
percent-encoded one.
- manifests = sorted((output / "tree" / BRANCH /
"_versions").glob("*.manifest"))
+ # Keep Git paths portable; the upload restores the exact name Lance 12
resolves.
+ manifests = sorted((output / "tree" / STORED_BRANCH /
"_versions").glob("*.manifest"))
assert len(manifests) == 2, f"branch {BRANCH!r} must carry versions 1 and
2: {manifests}"
for manifest in manifests:
assert PARENT_URI.encode() in manifest.read_bytes(), f"{manifest} must
reference {PARENT_URI}"
diff --git
"a/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/unicode_branch.lance/tree/dev\341\262\211/_transactions/1-1be00bd2-8443-4851-9ab8-4bdcee3ffa5e.txn"
b/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/unicode_branch.lance/tree/dev_u1c89/_transactions/1-1be00bd2-8443-4851-9ab8-4bdcee3ffa5e.txn
similarity index 100%
rename from
"docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/unicode_branch.lance/tree/dev\341\262\211/_transactions/1-1be00bd2-8443-4851-9ab8-4bdcee3ffa5e.txn"
rename to
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/unicode_branch.lance/tree/dev_u1c89/_transactions/1-1be00bd2-8443-4851-9ab8-4bdcee3ffa5e.txn
diff --git
"a/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/unicode_branch.lance/tree/dev\341\262\211/_transactions/1-c1998c6f-6213-4b7e-ae7c-cc00cde3b7de.txn"
b/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/unicode_branch.lance/tree/dev_u1c89/_transactions/1-c1998c6f-6213-4b7e-ae7c-cc00cde3b7de.txn
similarity index 100%
rename from
"docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/unicode_branch.lance/tree/dev\341\262\211/_transactions/1-c1998c6f-6213-4b7e-ae7c-cc00cde3b7de.txn"
rename to
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/unicode_branch.lance/tree/dev_u1c89/_transactions/1-c1998c6f-6213-4b7e-ae7c-cc00cde3b7de.txn
diff --git
"a/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/unicode_branch.lance/tree/dev\341\262\211/_versions/18446744073709551613.manifest"
b/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/unicode_branch.lance/tree/dev_u1c89/_versions/18446744073709551613.manifest
similarity index 100%
rename from
"docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/unicode_branch.lance/tree/dev\341\262\211/_versions/18446744073709551613.manifest"
rename to
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/unicode_branch.lance/tree/dev_u1c89/_versions/18446744073709551613.manifest
diff --git
"a/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/unicode_branch.lance/tree/dev\341\262\211/_versions/18446744073709551614.manifest"
b/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/unicode_branch.lance/tree/dev_u1c89/_versions/18446744073709551614.manifest
similarity index 100%
rename from
"docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/unicode_branch.lance/tree/dev\341\262\211/_versions/18446744073709551614.manifest"
rename to
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/unicode_branch.lance/tree/dev_u1c89/_versions/18446744073709551614.manifest
diff --git
"a/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/unicode_branch.lance/tree/dev\341\262\211/data/01110100011000100001111040aa57401ab473a6a921298540.lance"
b/docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/unicode_branch.lance/tree/dev_u1c89/data/01110100011000100001111040aa57401ab473a6a921298540.lance
similarity index 100%
rename from
"docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/unicode_branch.lance/tree/dev\341\262\211/data/01110100011000100001111040aa57401ab473a6a921298540.lance"
rename to
docker/thirdparties/docker-compose/iceberg/scripts/preinstalled_data/lance/unicode_branch.lance/tree/dev_u1c89/data/01110100011000100001111040aa57401ab473a6a921298540.lance
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
