This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new 759a81aadfe5 [fix](website) Recover search and bound publishing
resources (#4533)
759a81aadfe5 is described below
commit 759a81aadfe5c9649e8b7eef8dab84dd6ee913d3
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Tue Oct 27 16:46:47 2026 +0000
[fix](website) Recover search and bound publishing resources (#4533)
Documentation search can remain stuck while loading the 40–58 MB
same-origin indexes, and a failed request poisons the worker cache. Load
source-fingerprinted indexes through the existing asset CDN, fall back
to the website with bounded requests, and offer a localized retry while
preserving the first typed query. Worker scripts remain same-origin.
Temporary redirects preserve the previous index URLs without duplicating
their contents.
Publishing now bounds Node/Terser/Rayon/SSG concurrency, reuses the Yarn
download cache with a frozen install, removes duplicate generation, and
shares compact HTML metadata across strict locale verification. Browser
route payloads retain stable bytes across source commits; full manifests
retain build provenance. UTF-8 client output and level-1 artifact
compression reduce expansion and transfer overhead. Measurements include
per-locale RSS, artifact categories, and exact public generated inputs.
PR admission checks the complete diff, every content
version/locale/plugin, and all changed file sizes with NUL-delimited
filenames. The existing 1,024,000-byte size limit is unchanged.
Validation:
- 391 Node tests: 389 passed, 2 conditional Apache skips; 27 workflow
tests passed.
- Real five-language Rspack search fixture: 16 scenarios passed,
including early input, CDN fallback, and retry in both search UIs; 40
mobile theme/error-layout checks passed.
- Configuration, manual-content digests, and source diff checks passed.
The four existing community-report drift records remain visible.
- CI retains representative validation for this PR; complete
publication, resource headroom, and final public verification are part
of T22 after merge. No full production site was compiled locally.
Self-review against AGENTS.md:
- Goal: recover search and address the measured publishing/gate
problems; production memory/time improvements still require the real
publishing measurement.
- Scope: shared search, publication, and admission mechanisms; no prose
backfill or translation ownership changes.
- Information architecture: all active versions, locales, plugins,
English fallback, and coverage denominators are preserved.
- URLs/navigation: document routes and sidebars are unchanged; legacy
index URLs have temporary aliases to the new files.
- Frontend: retry is localized, keyboard accessible, and uses semantic
theme tokens; the CDN connection is allowed explicitly by CSP.
- Validation: actual worker-cache failure/retry, corrupted static-output
rejection, filename/range gate regressions, numeric registry keys, and
UTF-8 string values are covered.
- Other correctness/maintenance: generated provenance stays in build
reports; fixture selection is scoped to search changes, and
fork/deployment credential boundaries are unchanged.
Co-authored-by: morningman <[email protected]>
---
.github/scripts/publish_metrics.py | 46 ++++++-
.github/scripts/test_pr_gates.py | 108 +++++++++++++++
.github/scripts/test_publish_metrics.py | 31 +++++
.github/scripts/test_review.py | 3 +-
.github/workflows/build-check.yml | 14 +-
.github/workflows/check-md-links.yml | 42 +++---
.github/workflows/check-pr-file-sizes.yml | 22 +--
.github/workflows/cron-deploy-website.yml | 21 +--
.github/workflows/manual-deploy-website.yml | 22 +--
config/custom-docusaurus-plugin.js | 22 ++-
scripts/build-locales.cjs | 26 +++-
scripts/check_move.js | 37 ++---
scripts/i18n/__tests__/content-manifest.test.cjs | 3 +
scripts/i18n/__tests__/search-transport.test.cjs | 100 ++++++++++++++
scripts/i18n/__tests__/static-cache.test.cjs | 41 ++++++
scripts/i18n/ci-plan.cjs | 3 +
scripts/i18n/content-manifest.cjs | 4 +-
scripts/i18n/search-plugin.cjs | 38 +++--
scripts/i18n/search-worker-loader.cjs | 26 ++++
scripts/i18n/verify-search-transport.cjs | 153 +++++++++++++++++++++
scripts/i18n/verify-static.cjs | 58 +++++---
scripts/publish/__tests__/js-minimizer.test.cjs | 23 ++++
scripts/publish/__tests__/optimize.test.cjs | 15 ++
scripts/publish/__tests__/registry-loader.test.cjs | 2 +-
scripts/publish/optimize.cjs | 8 ++
src/theme/SearchBar/SearchBar.jsx | 93 ++++++++-----
src/theme/SearchBar/SearchBar.module.css | 33 +++++
src/theme/SearchPage/SearchPage.jsx | 49 ++++---
src/utils/fetchSearchIndex.js | 32 +++++
src/utils/searchMessages.ts | 8 ++
src/utils/translation.ts | 1 -
static/.htaccess | 2 +-
translations/shared/registry.json | 3 +-
33 files changed, 903 insertions(+), 186 deletions(-)
diff --git a/.github/scripts/publish_metrics.py
b/.github/scripts/publish_metrics.py
index 6d109319ad42..ca125062ee1d 100644
--- a/.github/scripts/publish_metrics.py
+++ b/.github/scripts/publish_metrics.py
@@ -3,6 +3,7 @@
import argparse
from datetime import datetime
import hashlib
+import heapq
import json
import math
import os
@@ -65,7 +66,8 @@ def start(job):
"GITHUB_REPOSITORY", "GITHUB_RUN_ID", "GITHUB_RUN_ATTEMPT",
"GITHUB_SHA",
"GITHUB_WORKFLOW_REF", "GITHUB_EVENT_NAME", "RUNNER_OS", "RUNNER_ARCH",
"ImageOS", "ImageVersion", "DEPLOY_SOURCE_SHA", "DORIS_ENABLE_JA",
- "DORIS_ENABLE_KO", "DORIS_ENABLE_RU", "DOCS_VERSIONS")}
+ "DORIS_ENABLE_KO", "DORIS_ENABLE_RU", "DOCS_VERSIONS", "NODE_OPTIONS",
+ "TERSER_PARALLEL", "RAYON_NUM_THREADS", "DOCUSAURUS_SSR_CONCURRENCY")}
write_json(REPORTS / "context.json", {
"schemaVersion": 1, "job": job, "startedAt": time.time(),
"checkoutSha": git("rev-parse", "HEAD"), "context": context,
@@ -73,24 +75,51 @@ def start(job):
"activeVersions": (["current",
*json.loads(Path("versions.json").read_text())]
if job == "build" else None),
"checkoutActiveVersions": ["current",
*json.loads(Path("versions.json").read_text())],
- "cachePolicy": ("fresh runner; no dependency cache; yarn cache clean
before install"
+ "cachePolicy": ("Yarn download cache keyed by yarn.lock; frozen
install with postinstall"
if job == "build" else "fresh runner; site downloaded
from this run's artifact"),
"initialResources": snapshot(),
})
launch_sampler()
-def site_inventory(site):
+def site_inventory(site, detailed=False):
if not site.is_dir():
return None
files, size = 0, 0
+ extensions, directories, largest = {}, {}, []
for root, _, names in os.walk(site):
for name in names:
item = Path(root) / name
if item.is_file():
+ item_size = item.stat().st_size
files += 1
- size += item.stat().st_size
- return {"files": files, "bytes": size}
+ size += item_size
+ if detailed:
+ relative = item.relative_to(site)
+ directory = "/".join(relative.parts[:2] if
relative.parts[0] == "assets" else relative.parts[:1])
+ for table, key in [(extensions, item.suffix.lower() or
"(none)"), (directories, directory)]:
+ row = table.setdefault(key, {"files": 0, "bytes": 0})
+ row["files"] += 1
+ row["bytes"] += item_size
+ largest.append({"path": relative.as_posix(), "bytes":
item_size})
+ return {"files": files, "bytes": size, **({"byExtension": extensions,
"byDirectory": directories,
+ "largestFiles": heapq.nlargest(20, largest, key=lambda row:
row["bytes"])} if detailed else {})}
+
+
+def archive_generated_inputs():
+ # These are public generated build inputs, not arbitrary tracked files or
+ # environment snapshots. Preserve their exact bytes for reproduction.
+ result = {}
+ for name in ["scripts/last-update/data.json",
"src/constant/github.data.ts", "src/generated/key-features.ts"]:
+ source = Path(name)
+ if not source.is_file() or source.is_symlink() or not
source.resolve().is_relative_to(Path.cwd().resolve()):
+ continue
+ content = source.read_bytes()
+ target = REPORTS / "inputs" / name
+ target.parent.mkdir(parents=True, exist_ok=True)
+ target.write_bytes(content)
+ result[name] = {"bytes": len(content), "sha256":
hashlib.sha256(content).hexdigest()}
+ return result
def api(url):
@@ -145,7 +174,12 @@ def finish(budget):
report["finishedAt"] = time.time()
report["elapsedThroughReportSeconds"] = report["finishedAt"] -
report["startedAt"]
report["finalResources"] = snapshot()
- report["site"] = site_inventory(Path("build"))
+ report["site"] = site_inventory(Path("build"), detailed=True)
+ if report["job"] == "build":
+ report["dependencyCacheHit"] = os.environ.get("DEPENDENCY_CACHE_HIT")
+ report["generatedInputs"] = archive_generated_inputs()
+ changes = subprocess.check_output(["git", "diff", "--name-only", "-z",
"HEAD"]).decode().split("\0")
+ report["unarchivedTrackedChanges"] = [name for name in changes if name
and name not in report["generatedInputs"]]
if report["job"] == "deploy" and Path("build/versions.json").is_file():
# Manual deploy code is checked out at the workflow commit, which
# may have different versions from the protected source branch.
diff --git a/.github/scripts/test_pr_gates.py b/.github/scripts/test_pr_gates.py
new file mode 100644
index 000000000000..138f158a8abb
--- /dev/null
+++ b/.github/scripts/test_pr_gates.py
@@ -0,0 +1,108 @@
+import os
+from pathlib import Path
+import re
+import subprocess
+import tempfile
+import textwrap
+import unittest
+
+ROOT = Path(__file__).resolve().parents[2]
+
+
+def step_script(workflow, name):
+ source = (ROOT / '.github/workflows' / workflow).read_text()
+ match = re.search(r'^(\s*)- name: ' + re.escape(name) + r'\s*$', source,
re.M)
+ assert match, name
+ lines = source[match.end():].splitlines()
+ for i, line in enumerate(lines):
+ if re.match(r'^\s*run: \|\s*$', line):
+ indent = len(line) - len(line.lstrip())
+ body = []
+ for value in lines[i + 1:]:
+ if value.strip() and len(value) - len(value.lstrip()) <=
indent:
+ break
+ body.append(value)
+ return textwrap.dedent('\n'.join(body))
+ raise AssertionError('Missing run block: ' + name)
+
+
+class PrGateRegression(unittest.TestCase):
+ def setUp(self):
+ self.temp = tempfile.TemporaryDirectory(prefix='doris-pr-gates-')
+ self.addCleanup(self.temp.cleanup)
+ self.root = Path(self.temp.name)
+ self.git('init', '-q', '--initial-branch=master')
+ self.git('config', 'user.name', 'Fixture')
+ self.git('config', 'user.email', '[email protected]')
+ self.git('commit', '--allow-empty', '-qm', 'base')
+ self.base = self.git('rev-parse', 'HEAD').strip()
+ self.env = {**os.environ, 'RUNNER_TEMP': str(self.root),
+ 'GITHUB_OUTPUT': str(self.root / 'outputs'), 'BASE_REF':
'master'}
+ self.git('update-ref', 'refs/remotes/origin/master', self.base)
+
+ def git(self, *args):
+ return subprocess.check_output(['git', *args], cwd=self.root,
text=True)
+
+ def write(self, name, value):
+ file = self.root / name
+ file.parent.mkdir(parents=True, exist_ok=True)
+ file.write_bytes(value if isinstance(value, bytes) else value.encode())
+
+ def commit(self):
+ self.git('add', '.')
+ self.git('commit', '-qm', 'change')
+
+ def run_step(self, workflow, name):
+ command = step_script(workflow, name).replace('${{
github.event.pull_request.base.sha }}', self.base)
+ return subprocess.run(['bash', '-e', '-o', 'pipefail', '-c', command],
cwd=self.root,
+ env=self.env, text=True, capture_output=True)
+
+ def
test_size_gate_checks_the_101st_file_and_preserves_boundary_and_exception(self):
+ for i in range(100):
+ self.write(f'a{i:03}.txt', 'small')
+ self.write('z large\n中文.bin', b'x' * 1024001)
+ self.write('scripts/last-update/data.json', b'x' * 1024001)
+ self.commit()
+ result = self.run_step('check-pr-file-sizes.yml', 'Get changed files
in PR')
+ self.assertEqual(result.returncode, 0, result.stderr)
+ selected = (self.root / 'pr-size-files').read_bytes().split(b'\0')
+ self.assertIn('z large\n中文.bin'.encode(), selected)
+ result = self.run_step('check-pr-file-sizes.yml', 'Check file sizes')
+ self.assertEqual(result.returncode, 1, result.stdout + result.stderr)
+ self.assertIn('z large', result.stdout)
+ self.write('z large\n中文.bin', b'x' * 1024000)
+ result = self.run_step('check-pr-file-sizes.yml', 'Check file sizes')
+ self.assertEqual(result.returncode, 0, result.stdout + result.stderr)
+
+ def
test_markdown_gate_selects_all_locales_versions_plugins_and_rename_destinations(self):
+ self.write('docs/old.md', '# Before')
+ self.commit()
+ self.base = self.git('rev-parse', 'HEAD').strip()
+ self.git('update-ref', 'refs/remotes/origin/master', self.base)
+ self.git('mv', 'docs/old.md', 'docs/new name.md')
+ paths = ['versioned_docs/version-4.x/topic.md', 'i18n/ru/русский.md',
+ 'i18n/ko/한국어.md', 'i18n/ja/日本語.md', 'i18n/zh-CN/中文.md',
+ 'community/topic.md', 'course/topic.mdx',
'releasenotes/topic.md',
+ 'blog/topic.md', 'docs/new\nline.md']
+ for name in paths:
+ self.write(name, '# Topic')
+ self.commit()
+ result = self.run_step('check-md-links.yml', 'Collect added/modified
Markdown files')
+ self.assertEqual(result.returncode, 0, result.stdout + result.stderr)
+ selected = {value.decode() for value in (self.root /
'pr-markdown-files').read_bytes().split(b'\0') if value}
+ self.assertEqual(selected, set(paths + ['docs/new name.md']))
+ self.assertIn('has_files=true', (self.root / 'outputs').read_text())
+
+ def
test_node_link_checker_checks_earlier_commits_and_paths_with_spaces(self):
+ self.write('versioned_docs/version-4.x/first topic.md', '#
Topic\n\n[Broken](missing.md)\n')
+ self.commit()
+ self.write('later.txt', 'not Markdown')
+ self.commit()
+ result = subprocess.run(['node', str(ROOT / 'scripts/check_move.js'),
'--base', self.base],
+ cwd=self.root, text=True, capture_output=True)
+ self.assertEqual(result.returncode, 1, result.stdout + result.stderr)
+ self.assertIn('missing.md', result.stderr)
+
+
+if __name__ == '__main__':
+ unittest.main()
diff --git a/.github/scripts/test_publish_metrics.py
b/.github/scripts/test_publish_metrics.py
index b775e7e4d52a..70f8b56e59ca 100644
--- a/.github/scripts/test_publish_metrics.py
+++ b/.github/scripts/test_publish_metrics.py
@@ -18,6 +18,37 @@ spec.loader.exec_module(metrics)
class PublishMetricsTests(unittest.TestCase):
+ def
test_generated_inputs_archive_is_limited_to_public_inputs_and_rejects_symlinks(self):
+ with tempfile.TemporaryDirectory() as directory:
+ previous = Path.cwd()
+ os.chdir(directory)
+ try:
+ source = Path('src/constant/github.data.ts')
+ source.parent.mkdir(parents=True)
+ source.write_text("export const STAR_COUNT = '16.1';")
+ Path('private.env').write_text('SECRET_MUST_NOT_APPEAR')
+ linked = Path('src/generated/key-features.ts')
+ linked.parent.mkdir(parents=True)
+ linked.symlink_to(Path('private.env').resolve())
+ result = metrics.archive_generated_inputs()
+ self.assertEqual(set(result), {'src/constant/github.data.ts'})
+ self.assertEqual((metrics.REPORTS / 'inputs' /
source).read_bytes(), source.read_bytes())
+ self.assertNotIn('SECRET_MUST_NOT_APPEAR', json.dumps(result))
+ finally:
+ os.chdir(previous)
+
+ def
test_detailed_inventory_retains_totals_and_reports_file_categories(self):
+ with tempfile.TemporaryDirectory() as directory:
+ site = Path(directory)
+ (site / 'assets/js').mkdir(parents=True)
+ (site / 'assets/js/main.js').write_bytes(b'12345')
+ (site / '.htaccess').write_bytes(b'abc')
+ detailed = metrics.site_inventory(site, detailed=True)
+ self.assertEqual({key: detailed[key] for key in ['files',
'bytes']}, metrics.site_inventory(site))
+ self.assertEqual(detailed['byExtension']['.js'], {'files': 1,
'bytes': 5})
+ self.assertEqual(detailed['byDirectory']['assets/js'], {'files':
1, 'bytes': 5})
+ self.assertEqual(detailed['largestFiles'][0]['path'],
'assets/js/main.js')
+
def test_resource_summary_preserves_disk_and_memory_extrema(self):
samples = [
{"memory": {"MemAvailableKiB": 100}, "disks": {"/": {"usedBytes":
800, "freeBytes": 200}}},
diff --git a/.github/scripts/test_review.py b/.github/scripts/test_review.py
index 1bbd324fbd55..8e0af1ba5834 100644
--- a/.github/scripts/test_review.py
+++ b/.github/scripts/test_review.py
@@ -241,6 +241,7 @@ class WorkflowTests(unittest.TestCase):
head = git("rev-parse", "HEAD")
git("merge", "--no-ff", "-qm", "PR merge snapshot", "master")
merge = git("rev-parse", "HEAD")
+ git("update-ref", "refs/remotes/origin/master", git("rev-parse",
"master"))
refs = {"${{ github.sha }}": merge, "${{
github.event.pull_request.head.sha }}": head}
git("checkout", "--detach", refs[checkout["with"]["ref"]])
completed = subprocess.run(security["run"], cwd=root, shell=True,
@@ -250,7 +251,7 @@ class WorkflowTests(unittest.TestCase):
# A clean merge has no combined diff; the link checker must still
inspect PR files.
(root / "scripts").symlink_to(ROOT / "scripts",
target_is_directory=True)
completed = subprocess.run(links["run"], cwd=root, shell=True,
executable="/bin/bash",
- env={**os.environ, **{key: refs[value]
+ env={**os.environ, "BASE_REF":
"master", **{key: refs[value]
for key, value in links.get("env",
{}).items()}},
capture_output=True, text=True)
self.assertEqual(completed.returncode, 1, completed.stdout +
completed.stderr)
diff --git a/.github/workflows/build-check.yml
b/.github/workflows/build-check.yml
index d26daa2225be..f8fecc3c6f85 100644
--- a/.github/workflows/build-check.yml
+++ b/.github/workflows/build-check.yml
@@ -79,12 +79,9 @@ jobs:
- name: Run Node.js dead link checker
- env:
- PR_HEAD_SHA: ${{ github.event.pull_request.head.sha }}
- CI_SHA: ${{ github.sha }}
run: |
- # git show on a clean merge omits the PR's changed files.
- node ./scripts/check_move.js "${PR_HEAD_SHA:-$CI_SHA}"
+ BASE_SHA=$(git merge-base "origin/$BASE_REF" HEAD)
+ node ./scripts/check_move.js --base "$BASE_SHA"
- name: Detect changed doc versions and locales
id: detect
@@ -166,7 +163,7 @@ jobs:
done
- name: Install browser verification tools
- if: steps.detect.outputs.validation_mode == 'full' ||
steps.detect.outputs.validation_mode == 'sample' ||
steps.detect.outputs.publish_fixture == 'true' ||
steps.detect.outputs.content_fixture == 'true' ||
steps.detect.outputs.sampling_fixture == 'true'
+ if: steps.detect.outputs.validation_mode == 'full' ||
steps.detect.outputs.validation_mode == 'sample' ||
steps.detect.outputs.publish_fixture == 'true' ||
steps.detect.outputs.content_fixture == 'true' ||
steps.detect.outputs.sampling_fixture == 'true' ||
steps.detect.outputs.search_fixture == 'true'
run: |
DORIS_BROWSER_TOOLS="$RUNNER_TEMP/doris-browser-tools"
npm install --prefix "$DORIS_BROWSER_TOOLS" --no-save
--no-package-lock [email protected]
@@ -177,6 +174,10 @@ jobs:
if: steps.detect.outputs.sampling_fixture == 'true'
run: node scripts/i18n/verify-ci-sampling-fixture.cjs --browser
+ - name: Verify search transport and recovery in Chromium
+ if: steps.detect.outputs.search_fixture == 'true'
+ run: node scripts/i18n/verify-search-transport.cjs --browser
+
- name: Verify five-language publishing fixture in Chromium
if: steps.detect.outputs.publish_fixture == 'true'
env:
@@ -265,6 +266,7 @@ jobs:
.i18n-build/ci-sample/scopes/*.json
.i18n-build/ci-sample/verification.json
.i18n-build/ci-sampling-fixture-proof.json
+ .i18n-build/search-transport-verification.json
.i18n-build/publish-result.json
.i18n-build/build-result.json
.i18n-build/content-fixture-verification.json
diff --git a/.github/workflows/check-md-links.yml
b/.github/workflows/check-md-links.yml
index 8b9f8ea3422e..4c8b17fb10c0 100644
--- a/.github/workflows/check-md-links.yml
+++ b/.github/workflows/check-md-links.yml
@@ -29,23 +29,26 @@ jobs:
env:
BASE_REF: ${{ github.event.pull_request.base.ref }}
run: |
- # Only added (A) or modified (M) files in this PR — deletions and
renames-source are skipped.
+ # Check added/modified files and rename destinations across all
+ # content plugins, versions and locales; deleted files are skipped.
# github.event.pull_request.base.sha is the base branch tip from
when the PR was opened,
# not the current one, so diffing against it drags in every commit
that landed on the base
# branch since then. Resolve the merge base against the base branch
as it is now: the
# checkout is refs/pull/<n>/merge, so this leaves exactly the PR's
own changes.
BASE_SHA="$(git merge-base "origin/$BASE_REF" HEAD)"
echo "Diffing against origin/$BASE_REF ($BASE_SHA)"
- {
- echo 'files<<EOF'
- git diff --name-only --diff-filter=AM \
+ git diff --name-only --diff-filter=AMR -z \
"$BASE_SHA" HEAD \
- -- 'docs/**/*.md' 'docs/**/*.mdx' \
- 'i18n/zh-CN/docusaurus-plugin-content-docs/current/**/*.md' \
- 'i18n/zh-CN/docusaurus-plugin-content-docs/current/**/*.mdx' \
- 'i18n/ja/**/*.md' 'i18n/ja/**/*.mdx'
- echo EOF
- } >> "$GITHUB_OUTPUT"
+ -- ':(glob)docs/**/*.md' ':(glob)docs/**/*.mdx' \
+ ':(glob)versioned_docs/**/*.md'
':(glob)versioned_docs/**/*.mdx' \
+ ':(glob)i18n/**/*.md' ':(glob)i18n/**/*.mdx' \
+ ':(glob)community/**/*.md' ':(glob)community/**/*.mdx' \
+ ':(glob)course/**/*.md' ':(glob)course/**/*.mdx' \
+ ':(glob)releasenotes/**/*.md' ':(glob)releasenotes/**/*.mdx' \
+ ':(glob)blog/**/*.md' ':(glob)blog/**/*.mdx' >
"$RUNNER_TEMP/pr-markdown-files"
+ if [ -s "$RUNNER_TEMP/pr-markdown-files" ]; then
+ echo 'has_files=true' >> "$GITHUB_OUTPUT"
+ fi
{
echo 'scripts<<EOF'
git diff --name-only "$BASE_SHA" HEAD \
@@ -54,7 +57,7 @@ jobs:
} >> "$GITHUB_OUTPUT"
- name: Set up Python
- if: steps.changed.outputs.files != '' || steps.changed.outputs.scripts
!= ''
+ if: steps.changed.outputs.has_files == 'true' ||
steps.changed.outputs.scripts != ''
uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 #
v5
with:
python-version: '3.11'
@@ -64,22 +67,11 @@ jobs:
run: python3 -m unittest scripts/__tests__/test_japanese_links.py
- name: Check links in changed Markdown files
- if: steps.changed.outputs.files != ''
- env:
- FILES: ${{ steps.changed.outputs.files }}
+ if: steps.changed.outputs.has_files == 'true'
run: |
set -u
- if [ -z "$FILES" ]; then
- echo "No Markdown files added or modified in this PR. Skipping."
- exit 0
- fi
-
- echo "Markdown files to check:"
- echo "$FILES"
- echo "------------------------------------------------------------"
-
FAILED=()
- while IFS= read -r file; do
+ while IFS= read -r -d '' file; do
[ -z "$file" ] && continue
if [ ! -f "$file" ]; then
echo "Skip (not found in working tree): $file"
@@ -90,7 +82,7 @@ jobs:
if ! python3 scripts/check_md_links_single.py "$file"
--no-external; then
FAILED+=("$file")
fi
- done <<< "$FILES"
+ done < "$RUNNER_TEMP/pr-markdown-files"
echo
echo "============================================================"
diff --git a/.github/workflows/check-pr-file-sizes.yml
b/.github/workflows/check-pr-file-sizes.yml
index 1a62304bee25..d519936837d9 100644
--- a/.github/workflows/check-pr-file-sizes.yml
+++ b/.github/workflows/check-pr-file-sizes.yml
@@ -27,24 +27,16 @@ jobs:
run: git fetch --depth=1 --filter=blob:none origin ${{
github.event.pull_request.base.sha }}
- name: Get changed files in PR
- id: changed
run: |
- # Get list of files changed in the PR compared to base branch
- # Use delimiter syntax for multi-line output
- {
- echo 'files<<EOF'
- git diff --name-only ${{ github.event.pull_request.base.sha }}
HEAD | head -100
- echo EOF
- } >> "$GITHUB_OUTPUT"
+ git diff --name-only --diff-filter=AMR -z \
+ ${{ github.event.pull_request.base.sha }} HEAD >
"$RUNNER_TEMP/pr-size-files"
- name: Check file sizes
- env:
- FILES: ${{ steps.changed.outputs.files }}
run: |
- MAX_SIZE=1024000 # 500KB in bytes
+ MAX_SIZE=1024000 # 1,000 KiB; preserve the existing byte limit.
OVER_LIMIT=()
- while IFS= read -r file; do
+ while IFS= read -r -d '' file; do
[ -z "$file" ] && continue
# Auto-generated last-update timestamp map (scripts/last-update/):
# legitimately large and regenerated by tooling, not a
hand-committed
@@ -57,10 +49,10 @@ jobs:
OVER_LIMIT+=("❌ $file (${size_kb}KB)")
fi
fi
- done <<< "$FILES"
+ done < "$RUNNER_TEMP/pr-size-files"
if [ ${#OVER_LIMIT[@]} -gt 0 ]; then
- echo "❌ The following files exceed 500KB:"
+ echo "❌ The following files exceed 1,000 KiB:"
for f in "${OVER_LIMIT[@]}"; do
echo " $f"
done
@@ -69,4 +61,4 @@ jobs:
exit 1
fi
- echo "✅ All files are under 500KB"
+ echo "✅ All checked files are at or below 1,000 KiB"
diff --git a/.github/workflows/cron-deploy-website.yml
b/.github/workflows/cron-deploy-website.yml
index b79b81484e3b..44f0cd32a993 100644
--- a/.github/workflows/cron-deploy-website.yml
+++ b/.github/workflows/cron-deploy-website.yml
@@ -53,6 +53,12 @@ jobs:
DORIS_ENABLE_JA: 'true'
DORIS_ENABLE_KO: 'true'
DORIS_ENABLE_RU: 'true'
+ NODE_OPTIONS: '--max-old-space-size=6144'
+ TERSER_PARALLEL: '2'
+ RAYON_NUM_THREADS: '2'
+ DOCUSAURUS_SSR_CONCURRENCY: '8'
+ DOCUSAURUS_PERF_LOGGER: 'true'
+ DORIS_BUILD_METRICS: 'true'
steps:
- name: Checkout
# v4 (the pinned @master commit predates the `filter` input) with
@@ -94,9 +100,11 @@ jobs:
fc-list :lang=zh
- name: Use Node.js
+ id: node
uses: actions/setup-node@3235b876344d2a9aa001b8d1453c930bba69e610 #
v3
with:
node-version: 20
+ cache: yarn
- name: Refresh docs last-update map
# Recompute scripts/last-update/data.json from the history fetched
@@ -124,12 +132,9 @@ jobs:
GITHUB_TOKEN: ${{ github.token }}
run: |
/usr/bin/time -v -o .i18n-build/publish-metrics/build-time.txt
bash -e -o pipefail <<'BUILD'
- npm install -g yarn
- yarn cache clean
- export NODE_OPTIONS=--max-old-space-size=8192
- yarn
+ npm install -g [email protected]
+ yarn --frozen-lockfile
node ./scripts/update_github_info.js
- yarn key-features:generate
PWA_SERVICE_WORKER_URL=https://doris.apache.org/sw.js yarn
build:publish
touch build/.dummy
cp .asf-site.yaml ./build/.asf.yaml
@@ -146,9 +151,8 @@ jobs:
node scripts/i18n/manual-report.cjs --coverage
for LOCALE in $(node scripts/i18n/managed-locales.cjs); do
test -s "build/${LOCALE}/index.html"
- node scripts/i18n/verify-static.cjs \
- --site build --manifest
".i18n-build/manifests/${LOCALE}.json"
done
+ node scripts/i18n/verify-static.cjs --site build --all-managed
VERIFY
- name: Check file sizes for asf-site
@@ -175,7 +179,7 @@ jobs:
name: site-build-${{ github.run_id }}-${{ github.run_attempt }}
path: build
include-hidden-files: true
- compression-level: 0
+ compression-level: 1
if-no-files-found: error
retention-days: 1
@@ -185,6 +189,7 @@ jobs:
GH_TOKEN: ${{ github.token }}
SITE_ARTIFACT_ID: ${{ steps.site-artifact.outputs.artifact-id }}
SITE_ARTIFACT_DIGEST: ${{
steps.site-artifact.outputs.artifact-digest }}
+ DEPENDENCY_CACHE_HIT: ${{ steps.node.outputs.cache-hit }}
run: python3 .publish-tools/.github/scripts/publish_metrics.py
finish --budget 180
- name: Upload publishing measurements
diff --git a/.github/workflows/manual-deploy-website.yml
b/.github/workflows/manual-deploy-website.yml
index 1c9000046ff5..b433d7b49e5f 100644
--- a/.github/workflows/manual-deploy-website.yml
+++ b/.github/workflows/manual-deploy-website.yml
@@ -48,6 +48,12 @@ jobs:
DORIS_ENABLE_JA: 'true'
DORIS_ENABLE_KO: 'true'
DORIS_ENABLE_RU: 'true'
+ NODE_OPTIONS: '--max-old-space-size=6144'
+ TERSER_PARALLEL: '2'
+ RAYON_NUM_THREADS: '2'
+ DOCUSAURUS_SSR_CONCURRENCY: '8'
+ DOCUSAURUS_PERF_LOGGER: 'true'
+ DORIS_BUILD_METRICS: 'true'
steps:
- name: Checkout
uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
@@ -57,6 +63,7 @@ jobs:
# Sitemap lastmod comes from each file's last commit, so a
# shallow clone would date every page to the same commit.
fetch-depth: 0
+ filter: blob:none
- name: Checkout trusted measurement code
uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
@@ -86,9 +93,11 @@ jobs:
fc-list :lang=zh
- name: Use Node.js
+ id: node
uses: actions/setup-node@3235b876344d2a9aa001b8d1453c930bba69e610 #
v3
with:
node-version: 20
+ cache: yarn
- name: Add Swap Space
run: |
@@ -104,12 +113,9 @@ jobs:
GITHUB_TOKEN: ${{ github.token }}
run: |
/usr/bin/time -v -o .i18n-build/publish-metrics/build-time.txt
bash -e -o pipefail <<'BUILD'
- npm install -g yarn
- yarn cache clean
- export NODE_OPTIONS=--max-old-space-size=8192
- yarn
+ npm install -g [email protected]
+ yarn --frozen-lockfile
node ./scripts/update_github_info.js
- yarn key-features:generate
PWA_SERVICE_WORKER_URL=https://doris.apache.org/sw.js yarn
build:publish
touch build/.dummy
cp .asf-site.yaml ./build/.asf.yaml
@@ -126,9 +132,8 @@ jobs:
node scripts/i18n/manual-report.cjs --coverage
for LOCALE in $(node scripts/i18n/managed-locales.cjs); do
test -s "build/${LOCALE}/index.html"
- node scripts/i18n/verify-static.cjs \
- --site build --manifest
".i18n-build/manifests/${LOCALE}.json"
done
+ node scripts/i18n/verify-static.cjs --site build --all-managed
VERIFY
- name: Check file sizes for asf-site
@@ -147,7 +152,7 @@ jobs:
name: site-build-${{ github.run_id }}-${{ github.run_attempt }}
path: build
include-hidden-files: true
- compression-level: 0
+ compression-level: 1
if-no-files-found: error
retention-days: 1
@@ -157,6 +162,7 @@ jobs:
GH_TOKEN: ${{ github.token }}
SITE_ARTIFACT_ID: ${{ steps.site-artifact.outputs.artifact-id }}
SITE_ARTIFACT_DIGEST: ${{
steps.site-artifact.outputs.artifact-digest }}
+ DEPENDENCY_CACHE_HIT: ${{ steps.node.outputs.cache-hit }}
run: python3 .publish-tools/.github/scripts/publish_metrics.py
finish --budget 180
- name: Upload publishing measurements
diff --git a/config/custom-docusaurus-plugin.js
b/config/custom-docusaurus-plugin.js
index 145ee51dc2a0..5de04235899f 100644
--- a/config/custom-docusaurus-plugin.js
+++ b/config/custom-docusaurus-plugin.js
@@ -20,7 +20,9 @@ function clientMinimizers(config) {
parse: {ecma: 2020},
compress: {ecma: 5},
mangle: {safari10: true},
- format: {ecma: 5, comments: false, ascii_only: true},
+ // The site and assets are UTF-8. Keeping translated strings
+ // avoids expanding each CJK character into six ASCII bytes.
+ format: {ecma: 5, comments: false, ascii_only: false},
},
});
});
@@ -38,13 +40,21 @@ module.exports = function (context, options) {
const assetOrigin = getAssetOrigin();
const generated = config.resolve?.alias?.['@generated'] ??
context.generatedFilesDir;
const registry = typeof generated === 'string' ?
path.join(generated, 'registry.js') : null;
+ const rules = [];
+ if (config.mode === 'production' && registry) rules.push({
+ enforce: 'pre',
+ test: file => path.resolve(file) === registry,
+ use: [{loader:
require.resolve('../scripts/publish/registry-loader.cjs')}],
+ });
+ if (!isServer && config.mode === 'production') {
+ const worker =
require.resolve('@yang1666204/docusaurus-search-local/dist/client/client/theme/worker.js');
+ rules.push({enforce: 'pre', test: file => path.resolve(file)
=== worker,
+ use: [{loader:
require.resolve('../scripts/i18n/search-worker-loader.cjs'),
+ options: {assetOrigin, siteOrigin:
context.siteConfig.url ?? ''}}]});
+ }
return {
- ...(config.mode === 'production' && registry ? {module:
{rules: [{
- enforce: 'pre',
- test: file => path.resolve(file) === registry,
- use: [{loader:
require.resolve('../scripts/publish/registry-loader.cjs')}],
- }]}} : {}),
+ ...(rules.length ? {module: {rules}} : {}),
...(!isServer && config.mode === 'production' ? {
mergeStrategy: {'optimization.minimizer': 'replace'},
optimization: {minimizer: clientMinimizers(config)},
diff --git a/scripts/build-locales.cjs b/scripts/build-locales.cjs
index b6a9b477789b..74f62979392d 100644
--- a/scripts/build-locales.cjs
+++ b/scripts/build-locales.cjs
@@ -53,15 +53,33 @@ async function buildLocales({siteDir = process.cwd(),
outDir = 'build', locales:
const stop = signal => child.kill(signal);
const term = () => stop('SIGTERM');
const interrupt = () => stop('SIGINT');
+ const memory = process.env.DORIS_BUILD_METRICS === 'true' &&
process.platform === 'linux'
+ ?
{sampleIntervalMs:1000,samples:0,peakRssKiB:0,minimumSystemAvailableKiB:null} :
null;
+ const sample = () => {
+ try {
+ const rss =
Number(fs.readFileSync(`/proc/${child.pid}/status`,'utf8').match(/^VmRSS:\s+(\d+)/m)?.[1]);
+ const available =
Number(fs.readFileSync('/proc/meminfo','utf8').match(/^MemAvailable:\s+(\d+)/m)?.[1]);
+ if (rss > 0 && available > 0) {
+ memory.samples++;
+ memory.peakRssKiB = Math.max(memory.peakRssKiB,rss);
+ memory.minimumSystemAvailableKiB =
Math.min(memory.minimumSystemAvailableKiB ?? Infinity,available);
+ }
+ } catch {} // A child can exit between the timer and /proc read.
+ };
+ const sampler = memory ? setInterval(sample,1000) : null;
+ const cleanup = () => {
+ if (sampler) clearInterval(sampler);
+ process.removeListener('SIGTERM',term);
+ process.removeListener('SIGINT',interrupt);
+ };
process.once('SIGTERM', term);
process.once('SIGINT', interrupt);
- child.once('error', reject);
+ child.once('error', error => {cleanup();reject(error);});
child.once('exit', (code, signal) => {
- process.removeListener('SIGTERM', term);
- process.removeListener('SIGINT', interrupt);
+ cleanup();
console.log(JSON.stringify({event:'locale_process_end',
mode:args[0], locale:args[0] === 'plan' ? null : args[3], pid:child.pid, code,
signal}));
if (code !== 0) reject(new Error(`Locale ${args[3]} failed
(${signal ?? code}); output was not published`));
- else resolve({pid:child.pid});
+ else resolve({pid:child.pid,...(memory && {memory})});
});
}));
try {
diff --git a/scripts/check_move.js b/scripts/check_move.js
index 011de09d081f..4291bac6f9f2 100644
--- a/scripts/check_move.js
+++ b/scripts/check_move.js
@@ -4,17 +4,18 @@
*/
-const { execSync } = require("child_process");
+const { execFileSync } = require("child_process");
const fs = require("fs");
const path = require("path");
const { sourceForLocalized } = require("./i18n/paths.cjs");
const { managedLocales } = require("./i18n/state.cjs");
const translationLocales = new Set(managedLocales(path.resolve(__dirname,
"..")));
-const commitHash = process.argv[2];
+const baseRef = process.argv[2] === '--base' ? process.argv[3] : null;
+const commitHash = baseRef ? null : process.argv[2];
-if (require.main === module && !commitHash) {
- console.error("❌ Please provide the commit hash, such as: node
check-dead-links.js <commit-hash>");
+if (require.main === module && (!commitHash && !baseRef || commitHash ===
'--base')) {
+ console.error("Provide a commit hash, or --base <merge-base> to check the
complete PR diff.");
process.exit(1);
}
@@ -22,27 +23,11 @@ const linkRegex = /\[.*?\]\((.*?)\)/g;
let hasBrokenLinks = false;
// Get the modified or newly added .md/.mdx files in the commit
-function getModifiedMarkdownFiles(commit) {
- const output = execSync(
- `git show --name-status --diff-filter=AM ${commit} -- '*.md' '*.mdx'`,
- { encoding: "utf-8", maxBuffer: 100 * 1024 * 1024 }
- );
- const lines = output.split("\n");
- console.log('lines',lines);
-
- const files = [];
-
- for (const line of lines) {
- const parts = line.trim().split(/\s+/);
- if (parts.length === 2) {
- const [status, filePath] = parts;
- if ((status === "A" || status === "M") && (filePath.endsWith(".md") ||
filePath.endsWith(".mdx"))) {
- files.push(filePath);
- }
- }
- }
-
- return files;
+function getModifiedMarkdownFiles(commit, base) {
+ const selection = base ? ['diff', '--name-only', '--diff-filter=AMR', '-z',
base, 'HEAD']
+ : ['show', '--format=', '--name-only', '--diff-filter=AMR', '-z', commit];
+ return execFileSync('git', [...selection, '--', '*.md', '*.mdx'],
+ {encoding: 'utf8', maxBuffer: 100 * 1024 *
1024}).split('\0').filter(Boolean);
}
// Checks if the link points to an existing local file
@@ -228,7 +213,7 @@ function checkFileLinks(filePath) {
// Main function
function main() {
- const files = getModifiedMarkdownFiles(commitHash);
+ const files = getModifiedMarkdownFiles(commitHash, baseRef);
if (files.length === 0) {
console.log("✅ Unmodified Markdown files");
return;
diff --git a/scripts/i18n/__tests__/content-manifest.test.cjs
b/scripts/i18n/__tests__/content-manifest.test.cjs
index 686427d3efb3..73fa733db109 100644
--- a/scripts/i18n/__tests__/content-manifest.test.cjs
+++ b/scripts/i18n/__tests__/content-manifest.test.cjs
@@ -57,6 +57,9 @@ test('real native plugin output binds stable identities,
routes and all four cov
assert.equal(missing.navigationRoutes['zh-CN'],
'/zh-CN/docs/dev/english-missing/');
assert(!JSON.stringify(globalData).includes('sourceHash'));
assert(!JSON.stringify(routeData(page)).includes('sourceHash'));
+ assert.deepEqual(routeData({...page, sourceSha:'a'.repeat(40)}),
+ routeData({...page, sourceSha:'b'.repeat(40)}), 'Source identity
cannot invalidate unchanged browser routing data');
+ assert.equal(page.sourceSha, manifest.sourceSha, 'Full build
provenance remains in the manifest');
assert(Buffer.byteLength(JSON.stringify(routeData(page))) < 2048);
}
const buildOnly = await collect(root, 'en', {browserData: false});
diff --git a/scripts/i18n/__tests__/search-transport.test.cjs
b/scripts/i18n/__tests__/search-transport.test.cjs
new file mode 100644
index 000000000000..4552404740e8
--- /dev/null
+++ b/scripts/i18n/__tests__/search-transport.test.cjs
@@ -0,0 +1,100 @@
+const test = require('node:test');
+const assert = require('node:assert/strict');
+const fs = require('node:fs');
+const path = require('node:path');
+const os = require('node:os');
+const ts = require('typescript');
+const lunr = require('lunr');
+const {adaptSearchWorker} = require('../search-worker-loader.cjs');
+const {targetFingerprint} = require('../search-plugin.cjs');
+const root = path.resolve(__dirname, '../../..');
+function compile(source, imports = {}) {
+ const {outputText} = ts.transpileModule(source, {compilerOptions: {
+ module: ts.ModuleKind.CommonJS, target: ts.ScriptTarget.ES2020,
esModuleInterop: true,
+ }});
+ const module = {exports: {}};
+ new Function('require','module','exports','location',outputText)(name => {
+ if (Object.hasOwn(imports, name)) return imports[name];
+ return require(name);
+ },module,module.exports,{origin:'https://doris.apache.org'});
+ return module.exports;
+}
+const {fetchSearchIndex} =
compile(fs.readFileSync(path.join(root,'src/utils/fetchSearchIndex.js'),'utf8'));
+const base =
{origin:'https://doris.apache.org',siteOrigin:'https://doris.apache.org',assetOrigin:'https://cdnd.selectdb.com'};
+
+test('production search reads its configured CDN and preserves version, locale
and fingerprint', async () => {
+ const requests=[],data=[{documents:[]}];
+ const result=await
fetchSearchIndex('/ko/search-index-docs-dev-aabb.json?probe=1', {...base,
+ fetchImpl:async url=>{requests.push(url);return
{ok:true,json:async()=>data};}});
+ assert.equal(result,data);
+
assert.deepEqual(requests,['https://cdnd.selectdb.com/ko/search-index-docs-dev-aabb.json?probe=1']);
+});
+
+test('CDN HTTP, JSON and network failures fall back to the same-origin index',
async () => {
+ for (const failure of ['http','json','network']) {
+ const requests=[];
+ const result=await
fetchSearchIndex('/zh-CN/docs/dev/search-index-docs-aabb.json', {...base,
+ fetchImpl:async url=>{
+ requests.push(url);
+ if(requests.length===1) {
+ if(failure==='network')throw new Error('offline');
+ return
{ok:failure!=='http',status:503,json:async()=>{throw new Error('bad JSON');}};
+ }
+ return {ok:true,json:async()=>[]};
+ }});
+ assert.deepEqual(result,[]);
+
assert.deepEqual(requests,['https://cdnd.selectdb.com/zh-CN/docs/dev/search-index-docs-aabb.json','https://doris.apache.org/zh-CN/docs/dev/search-index-docs-aabb.json']);
+ }
+});
+
+test('local previews do not read production CDN data, and external version
URLs fail closed', async () => {
+ const requests=[];
+ await fetchSearchIndex('/docs/dev/search-index-docs-aabb.json',
{...base,origin:'http://127.0.0.1:3000',
+ fetchImpl:async url=>{requests.push(url);return
{ok:true,json:async()=>[]};}});
+
assert.deepEqual(requests,['http://127.0.0.1:3000/docs/dev/search-index-docs-aabb.json']);
+ await
assert.rejects(fetchSearchIndex('https://example.com/search-index.json',base),/Unexpected
search index origin/);
+});
+
+test('a stalled response is aborted before trying the other host', async () =>
{
+ const requests=[];
+ await fetchSearchIndex('/search-index-aabb.json', {...base,timeoutMs:5,
+ fetchImpl:async(url,{signal})=>{
+ requests.push(url);
+ if(requests.length===1)return new
Promise((resolve,reject)=>signal.addEventListener('abort',()=>reject(new
Error('aborted'))));
+ return {ok:true,json:async()=>[]};
+ }});
+ assert.equal(requests.length,2);
+});
+
+test('the actual worker discards rejected cache entries and reuses successful
ones', async () => {
+ const
source=fs.readFileSync(require.resolve('@yang1666204/docusaurus-search-local/dist/client/client/theme/worker.js'),'utf8');
+ const
index=lunr(function(){this.ref('i');this.field('t');this.add({i:'1',t:'Doris'});});
+ const
payload=[{documents:[{i:1,t:'Doris',u:'/docs/dev/intro/'}],index:index.toJSON()}];
+ let calls=0;
+ const {SearchWorker}=compile(adaptSearchWorker(source,base), {
+
[require.resolve('../../../src/utils/fetchSearchIndex.js')]:{fetchSearchIndex:async()=>{if(++calls===1)throw
new Error('offline');return payload;}},
+ comlink:{expose(){}},
+
'../utils/proxiedGeneratedConstants':{searchIndexUrl:'search-index{dir}-aabb.json',language:['en']},
+
'../utils/tokenize':{},'../utils/smartQueries':{},'../../shared/interfaces':{},
+
'../utils/sortSearchResults':{},'../utils/processTreeStatusOfSearchResults':{},
+ });
+ const worker=new SearchWorker();
+ await assert.rejects(worker.fetchIndexes('/docs/dev/','docs'),/offline/);
+ await worker.fetchIndexes('/docs/dev/','docs');
+ await worker.fetchIndexes('/docs/dev/','docs');
+ assert.equal(calls,2);
+ await
assert.rejects(worker.fetchIndexes('https://example.com/','docs'),/Unexpected
version url/);
+ assert.throws(()=>adaptSearchWorker(source.replace('const cache = new
Map();','const changed = 1;')),/Unexpected search worker/);
+});
+
+test('source-language search fingerprints include historical versions; managed
fallback versions do not', t => {
+ const
fixture=fs.mkdtempSync(path.join(os.tmpdir(),'doris-search-fingerprint-'));
+ t.after(()=>fs.rmSync(fixture,{recursive:true,force:true}));
+ fs.writeFileSync(path.join(fixture,'versions.json'),'["4.x","3.x","2.1"]');
+
fs.mkdirSync(path.join(fixture,'versioned_docs/version-3.x'),{recursive:true});
+ const
file=path.join(fixture,'versioned_docs/version-3.x/topic.md');fs.writeFileSync(file,'Before');
+ const
english=targetFingerprint(fixture,'en',{allVersions:true}),japanese=targetFingerprint(fixture,'ja');
+ fs.writeFileSync(file,'After');
+
assert.notEqual(targetFingerprint(fixture,'en',{allVersions:true}),english);
+ assert.equal(targetFingerprint(fixture,'ja'),japanese);
+});
diff --git a/scripts/i18n/__tests__/static-cache.test.cjs
b/scripts/i18n/__tests__/static-cache.test.cjs
new file mode 100644
index 000000000000..b0456e1068e5
--- /dev/null
+++ b/scripts/i18n/__tests__/static-cache.test.cjs
@@ -0,0 +1,41 @@
+const test=require('node:test');
+const assert=require('node:assert/strict');
+const fs=require('node:fs'),path=require('node:path'),os=require('node:os');
+const cheerio=require('cheerio');
+const {verify,verifyAll}=require('../verify-static.cjs');
+
+function fixture(t) {
+ const root=fs.mkdtempSync(path.join(os.tmpdir(),'doris-static-cache-'));
+ t.after(()=>fs.rmSync(root,{recursive:true,force:true}));
+ const site=path.join(root,'build'),manifestDir=path.join(root,'manifests');
+ fs.mkdirSync(manifestDir);const
alternates={en:'/topic/',ja:'/ja/topic/',ko:'/ko/topic/'};
+ const
write=(file,text)=>{fs.mkdirSync(path.dirname(file),{recursive:true});fs.writeFileSync(file,text);};
+ for(const [locale,route]of Object.entries(alternates)) {
+ const links=Object.entries(alternates).map(([lang,url])=>`<link
rel="alternate" hreflang="${lang==='en'?'en-US':lang}"
href="https://doris.apache.org${url}";>`).join('');
+ write(path.join(site,route,'index.html'),`<html><head><link
rel="canonical"
href="https://doris.apache.org${route}";>${links}</head><body><div
data-translation-status="current"></div><article class="theme-doc-markdown"
lang="${locale}"><h2 id="usage">Usage</h2></article></body></html>`);
+ if(locale==='en')continue;
+
write(path.join(site,locale,'sitemap.xml'),`<urlset><url><loc>https://doris.apache.org${route}</loc></url></urlset>`);
+
write(path.join(site,locale,'search-index.json'),JSON.stringify([{documents:[{u:route}]}]));
+
write(path.join(manifestDir,locale+'.json'),JSON.stringify({locale,pages:[{permalink:route,contentLocale:locale,
+
status:'current',useTarget:true,indexable:true,englishPermalink:'/topic/',alternates}],reports:[]}));
+ }
+ return {site,manifestDir,locales:['ja','ko']};
+}
+test('sharing compact metadata preserves strict results while parsing fewer
complete HTML documents',t=>{
+ const data=fixture(t),original=cheerio.load;let reads=0;
+
cheerio.load=(source,...args)=>{if(String(source).includes('theme-doc-markdown'))reads++;return
original(source,...args);};
+ try {
+ const
separate=data.locales.map(locale=>({locale,...verify({site:data.site,manifest:path.join(data.manifestDir,locale+'.json')})}));
+ const before=reads;reads=0;
+ assert.deepEqual(verifyAll(data),separate);
+ assert(reads<before,`${reads} shared parses must be less than
${before} separate parses`);
+ } finally {cheerio.load=original;}
+});
+test('each invocation rereads the artifact and rejects damaged heading IDs and
duplicate alternates',t=>{
+ const data=fixture(t);verifyAll(data);
+ const
file=path.join(data.site,'topic/index.html'),original=fs.readFileSync(file,'utf8');
+ fs.writeFileSync(file,original.replace('id="usage"','id="wrong"'));
+ assert.throws(()=>verifyAll(data),/Source heading IDs/);
+ const
$=cheerio.load(original);$('head').append($('link[hreflang="ko"]').clone());fs.writeFileSync(file,$.html());
+ assert.throws(()=>verifyAll(data),/Duplicate hreflang/);
+});
diff --git a/scripts/i18n/ci-plan.cjs b/scripts/i18n/ci-plan.cjs
index 541cb5108edd..567c645dc2b8 100644
--- a/scripts/i18n/ci-plan.cjs
+++ b/scripts/i18n/ci-plan.cjs
@@ -69,6 +69,8 @@ function policy(files, {forceFull = false, forceSample =
false} = {}) {
),
runSamplingFixture: files.some(file =>
/^scripts\/(build-locales?\.cjs$|i18n\/(ci-sample|ci-blog-sample|build-cache|verify-ci-sampl(?:e|ing-fixture))\.cjs$)/.test(file)),
+ runSearchFixture: mode === 'full' || files.some(file =>
+
/^(src\/theme\/(SearchBar|SearchPage)\/|src\/utils\/(fetchSearchIndex|searchMessages)\.[jt]s$|scripts\/i18n\/(search-plugin|search-worker-loader|verify-search-transport)\.cjs$)/.test(file)),
runPublishFixture: checks.check_publish_fixture || mode === 'full',
runTranslationTests: mode === 'full' || files.some(file =>
runtime.test(file) || mechanism.test(file) ||
maintenanceConfig(file) || /^scripts\/(i18n\/|build-locales?\.cjs$)/.test(file)
@@ -183,6 +185,7 @@ if (require.main === module) {
content_fixture: result.runContentFixture,
publish_fixture: result.runPublishFixture,
sampling_fixture: result.runSamplingFixture,
+ search_fixture: result.runSearchFixture,
translation_tests: result.runTranslationTests,
docs_report: result.runDocsReport,
})
diff --git a/scripts/i18n/content-manifest.cjs
b/scripts/i18n/content-manifest.cjs
index 114cb7f7de21..c996f75ca0f8 100644
--- a/scripts/i18n/content-manifest.cjs
+++ b/scripts/i18n/content-manifest.cjs
@@ -80,7 +80,9 @@ function validateManifest(manifest) {
// Consumers request one content identity; the full registry never enters
globalData.
function routeData(page) {
- return {schemaVersion, sourceSha: page.sourceSha, contentId:
page.contentId, kind: page.kind,
+ // Build identity stays in the full manifest. Unchanged browser routing
data
+ // must keep stable bytes across source commits for content-addressed
caches.
+ return {schemaVersion, contentId: page.contentId, kind: page.kind,
siteLocale: page.siteLocale, contentLocale: page.contentLocale,
fallback: page.fallback, renderIntegrated: page.renderIntegrated,
indexable: page.indexable, ...(page.aggregate && {aggregate: true}),
navigationRoutes: page.navigationRoutes, availableTranslations:
page.availableTranslations};
diff --git a/scripts/i18n/search-plugin.cjs b/scripts/i18n/search-plugin.cjs
index 3ff07349b56b..5de6e047d0d7 100644
--- a/scripts/i18n/search-plugin.cjs
+++ b/scripts/i18n/search-plugin.cjs
@@ -12,9 +12,11 @@ const {processPluginOptions} =
require('@yang1666204/docusaurus-search-local/dis
const {processDocInfos} =
require('@yang1666204/docusaurus-search-local/dist/server/server/utils/processDocInfos');
const {scanDocuments} =
require('@yang1666204/docusaurus-search-local/dist/server/server/utils/scanDocuments');
const {buildIndex: buildNativeIndex} =
require('@yang1666204/docusaurus-search-local/dist/server/server/utils/buildIndex');
+const {postBuildFactory} =
require('@yang1666204/docusaurus-search-local/dist/server/server/utils/postBuildFactory');
const {loadConfig, managedLocales} = require('./state.cjs');
const {normalizeRussianSearch} =
require('../../src/utils/searchNormalization.js');
const {fallbackOnly} = require('./docs-language-policy.cjs');
+const {sourceSha} = require('./content-manifest.cjs');
function contextsFor(root) {
const versions = JSON.parse(fs.readFileSync(path.join(root,
'versions.json'), 'utf8'));
@@ -52,7 +54,7 @@ function indexGroups(allDocuments, selected, contextNames =
contexts) {
groups.set('', allDocuments.map(documents => documents.filter(doc =>
!selected.has(normalize(doc.u)))));
return groups;
}
-function targetFingerprint(root, locale) {
+function targetFingerprint(root, locale, {allVersions = false} = {}) {
const digester = createHash('sha256');
digester.update(fs.readFileSync(__filename)); // Bust caches when indexing
behavior changes.
// The shared normalizer also changes indexed tokens independently of this
file.
@@ -72,8 +74,16 @@ function targetFingerprint(root, locale) {
scan(path.join(root,'i18n',locale));
scan(path.join(root,'translations/state',locale));
digester.update(fs.readFileSync(path.join(root, 'versions.json')));
- for (const dir of ['docs', ...contextsFor(root).filter(context =>
context.startsWith('docs/') && context !== 'docs/dev')
- .map(context =>
`versioned_docs/version-${context.slice('docs/'.length)}`),
'community','course','releasenotes']) scan(path.join(root,dir));
+ const versions = JSON.parse(fs.readFileSync(path.join(root,
'versions.json')))
+ .filter(version => allVersions || !fallbackOnly('default', version));
+ for (const dir of ['docs', ...versions.map(version =>
`versioned_docs/version-${version}`),
+ 'community','course','releasenotes']) scan(path.join(root,dir));
+ for (const file of
['docusaurus.config.js','config/i18n.js','translations/config.json',
+
'sidebars.ts','sidebarsCommunity.json','sidebarsCourse.json','sidebarsReleases.json'])
{
+ const target = path.join(root, file);
+ if (fs.existsSync(target))
{digester.update(file);digester.update(fs.readFileSync(target));}
+ }
+ digester.update(process.env.DOCS_VERSIONS ?? '');
return digester.digest('hex').slice(0,16);
}
function indexFilename(context) {
@@ -127,23 +137,32 @@ async function buildLocalizedIndex(buildData, config,
filename, locale, contextN
}
function searchPlugin(context, options) {
const locale = context.i18n.currentLocale;
- if (!managedLocales(context.siteDir,
context.i18n.locales).includes(locale)) return native(context,options);
+ const managed = managedLocales(context.siteDir,
context.i18n.locales).includes(locale);
const contextNames = contextsFor(context.siteDir);
- const localized = localizedOptions(options, locale,
loadConfig(context.siteDir).locales[locale]?.searchContextLabels, contextNames);
+ const localized = managed ? localizedOptions(options, locale,
loadConfig(context.siteDir).locales[locale]?.searchContextLabels, contextNames)
: options;
const plugin = native(context,localized);
const config = processPluginOptions(localized,context);
const constants =
path.join(context.generatedFilesDir,'@yang1666204/docusaurus-search-local/default/generated-constants.js');
const generated = fs.readFileSync(constants,'utf8');
- const fingerprint = targetFingerprint(context.siteDir, locale);
+ const fingerprint = targetFingerprint(context.siteDir, locale,
{allVersions: !managed});
+ const revision = sourceSha(context.siteDir);
+ let filename;
const updated = generated.replace(/(export const searchIndexUrl =
)("[^"\n]+")/,
(_,prefix,value) => {
const url = JSON.parse(value);
- return prefix + JSON.stringify(`${url}${url.includes('?') ? '&' :
'?'}${locale}=${fingerprint}`);
+ // A source-derived filename prevents a CDN from reusing an older
+ // index at the same path, even when it ignores query parameters.
+ const contentKey =
createHash('sha256').update(url).update(fingerprint).update(revision).digest('hex').slice(0,16);
+ filename = config.hashed ? `search-index{dir}-${contentKey}.json`
: url.split('?')[0];
+ return prefix + JSON.stringify(config.hashed ? filename
+ : `${url}${url.includes('?') ? '&' :
'?'}${locale}=${fingerprint}`);
});
if (updated === generated) throw new Error('Could not fingerprint the
localized search index');
fs.writeFileSync(constants,updated);
- const filename = indexFilename(context);
- return {...plugin, postBuild: buildData =>
buildLocalizedIndex(buildData,config,filename,locale,contextNames)};
+ if (indexFilename(context) !== filename) throw new Error('Search index
filename differs from its browser URL');
+ return {...plugin, postBuild: managed
+ ? buildData =>
buildLocalizedIndex(buildData,config,filename,locale,contextNames)
+ : postBuildFactory(config,filename)};
}
module.exports = searchPlugin;
module.exports.validateOptions = nativeModule.validateOptions;
@@ -156,3 +175,4 @@ module.exports.indexGroups = indexGroups;
module.exports.buildLocalizedIndex = buildLocalizedIndex;
module.exports.buildLocalizedIndexes = buildLocalizedIndexes;
+module.exports.targetFingerprint = targetFingerprint;
diff --git a/scripts/i18n/search-worker-loader.cjs
b/scripts/i18n/search-worker-loader.cjs
new file mode 100644
index 000000000000..8c852b4ed68b
--- /dev/null
+++ b/scripts/i18n/search-worker-loader.cjs
@@ -0,0 +1,26 @@
+const helper = require.resolve('../../src/utils/fetchSearchIndex.js');
+
+// Keep the pinned plugin's tokenizer, ranking and result shape. Adapt only
+// transport and its rejected-promise cache; fail closed on an upstream change.
+function adaptSearchWorker(source, options = {}) {
+ const replacements = [
+ ['const cache = new Map();',
+ `import {fetchSearchIndex} from ${JSON.stringify(helper)};\nconst
cache = new Map();`],
+ ['promise = legacyFetchIndexes(baseUrl, searchContext);',
+ `promise = legacyFetchIndexes(baseUrl, searchContext).catch(error
=> {\n` +
+ ` if (cache.get(cacheKey) === promise)
cache.delete(cacheKey);\n` +
+ ` throw error;\n });`],
+ ['const json = (await (await fetch(url)).json());',
+ `const json = await fetchSearchIndex(fullUrl,
${JSON.stringify(options)});`],
+ ];
+ for (const [before, after] of replacements) {
+ if (source.split(before).length !== 2) throw new Error('Unexpected
search worker format; revalidate the pinned search plugin');
+ source = source.replace(before, after);
+ }
+ return source;
+}
+module.exports = function searchWorkerLoader(source) {
+ this.cacheable?.();
+ return adaptSearchWorker(source, this.getOptions());
+};
+module.exports.adaptSearchWorker = adaptSearchWorker;
diff --git a/scripts/i18n/verify-search-transport.cjs
b/scripts/i18n/verify-search-transport.cjs
new file mode 100644
index 000000000000..28402e52d7d4
--- /dev/null
+++ b/scripts/i18n/verify-search-transport.cjs
@@ -0,0 +1,153 @@
+const fs = require('node:fs');
+const path = require('node:path');
+const assert = require('node:assert/strict');
+const {spawnSync, execFileSync} = require('node:child_process');
+const {createFixture} = require('./__tests__/fixtures/multi-locale.cjs');
+const {createServer} = require('../publish/serve.cjs');
+const root = path.resolve(__dirname, '../..');
+const fixture = path.join(root, '.i18n-build/search-transport-fixture');
+
+async function build() {
+ fs.rmSync(fixture, {recursive:true, force:true});
+ const {write} = await createFixture(fixture, {searchDocument:true});
+ for (const directory of
['src/theme/SearchBar','src/theme/SearchPage','src/theme/LoadingRing',
+ 'src/utils','src/hooks','src/constant','shared']) {
+
fs.cpSync(path.join(root,directory),path.join(fixture,directory),{recursive:true});
+ }
+
write('src/scss/_variables.scss',fs.readFileSync(path.join(root,'src/scss/_variables.scss'),'utf8'));
+
write('static/images/search-icon.svg',fs.readFileSync(path.join(root,'static/images/search-icon.svg')));
+ write('src/search-fixture.scss','@use "./scss/variables";\n');
+ write('src/theme/Root.tsx',`import React, {useEffect} from 'react';
+export default function
Root({children}){useEffect(()=>{document.documentElement.dataset.hasHydrated='true';},[]);return
children;}\n`);
+ fs.appendFileSync(path.join(fixture,'docusaurus.config.js'),`
+const base = module.exports;
+module.exports = () => {const config = base();
+ config.future = {experimental_faster:true};
+
config.plugins.push('docusaurus-plugin-sass',require(${JSON.stringify(path.join(root,'config/custom-docusaurus-plugin.js'))}));
+ config.presets[0][1].theme =
{customCss:[require.resolve('./src/search-fixture.scss')]};
+ config.themes =
[[require(${JSON.stringify(path.join(root,'scripts/i18n/search-plugin.cjs'))}),{
+
hashed:true,language:['en','zh'],indexDocs:true,indexBlog:false,indexPages:false,
+
docsRouteBasePath:['docs','community'],searchContextByPaths:['docs','community'],
+
useAllContextsWithNoSearchContext:false,highlightSearchTermsOnTargetPage:true}]];
+ config.themeConfig = {navbar:{title:'Search
fixture',items:[{type:'search',position:'right'}]}};
+ return config;};\n`);
+ execFileSync('git',['add','.'],{cwd:fixture});
+
execFileSync('git',['-c','user.name=Fixture','-c','[email protected]','commit','-qm','Search
transport fixture'],{cwd:fixture});
+ const result =
spawnSync(process.execPath,[path.join(root,'scripts/build-locales.cjs'),'--site-dir',fixture],
+
{cwd:root,stdio:'inherit',env:{...process.env,DORIS_ASSET_ORIGIN:'https://cdnd.selectdb.com',
+
NODE_OPTIONS:'--max-old-space-size=4096',TERSER_PARALLEL:'false'}});
+ assert.equal(result.status,0,'Search transport fixture build');
+}
+
+async function verify() {
+ const {chromium,request} = require('playwright');
+ const site = path.join(fixture,'build'),server = createServer(site);
+ await new Promise(resolve=>server.listen(0,'127.0.0.1',resolve));
+ const localOrigin = `http://127.0.0.1:${server.address().port}`;
+ const browser = await chromium.launch({headless:true}),api=await
request.newContext();
+ const
csp=fs.readFileSync(path.join(root,'static/.htaccess'),'utf8').match(/Header
set Content-Security-Policy "([^"]+)"/)[1];
+ const results=[],errors=[],contexts=new Set();
+ async function context(mode={}) {
+ const ctx=await
browser.newContext({serviceWorkers:'block',viewport:{width:1440,height:900}}),requests=[];
+ contexts.add(ctx);
+ await ctx.route('**/*',async route=>{
+ const url=new URL(route.request().url());
+
if(!['doris.apache.org','cdnd.selectdb.com'].includes(url.hostname))return
route.abort();
+ if(/search-index.*\.json$/.test(url.pathname)) {
+ requests.push({url:url.href,origin:url.origin});
+
if(mode.failBoth||mode.failCdn&&url.hostname==='cdnd.selectdb.com')
+ return
route.fulfill({status:503,body:'Unavailable',headers:{'access-control-allow-origin':'*'}});
+ if(mode.delay)await new
Promise(resolve=>setTimeout(resolve,mode.delay));
+ }
+ const response=await api.get(localOrigin+url.pathname+url.search);
+ return
route.fulfill({response,headers:{...response.headers(),'content-security-policy':csp,
+ 'access-control-allow-origin':'*'}});
+ });
+ const page=await
ctx.newPage();page.on('pageerror',error=>errors.push(error.message));
+ return {ctx,page,requests};
+ }
+ async function close(ctx) {
+ await ctx.unrouteAll({behavior:'wait'});
+ await ctx.close();contexts.delete(ctx);
+ }
+ const open=async(page,url)=>{
+ assert.equal((await
page.goto('https://doris.apache.org'+url,{waitUntil:'domcontentloaded'})).status(),200);
+ await page.waitForSelector('html[data-has-hydrated="true"]');
+ };
+ const
input=page=>page.locator('input.navbar__search-input:not([readonly]):visible').first();
+ const suggestions=page=>page.locator('[class*="suggestion_"]');
+ try {
+ for(const locale of ['en','zh-CN','ja','ko','ru']) {
+ const {ctx,page,requests}=await context({delay:1000});
+ const
manifest=JSON.parse(fs.readFileSync(path.join(fixture,'.i18n-build/content-manifests',locale+'.json')));
+ const
doc=manifest.pages.find(p=>p.kind==='docs'&&p.pluginId==='default'&&p.version==='current'&&p.docId==='shared');
+ assert(doc);await open(page,doc.route);
+ await input(page).fill('Doris');
+ await suggestions(page).first().waitFor();
+ assert.equal(await input(page).inputValue(),'Doris');
+ assert(requests.some(r=>r.origin==='https://cdnd.selectdb.com'));
+
assert(requests.every(r=>/search-index.*-[a-f0-9]{16}\.json$/.test(new
URL(r.url).pathname)));
+ const ax=await page.locator('.navbar__search').ariaSnapshot();
+ assert.match(ax,/combobox/);
+ results.push({locale,earlyQueryPreserved:true,results:await
suggestions(page).count(),requests,accessibility:ax});
+ await close(ctx);
+ }
+ {
+ const {ctx,page,requests}=await context({failCdn:true});
+ await open(page,'/docs/dev/english-shared/');await
input(page).fill('Doris');
+ await suggestions(page).first().waitFor();
+ assert(requests.some(r=>r.origin==='https://cdnd.selectdb.com'));
+ assert(requests.some(r=>r.origin==='https://doris.apache.org'));
+ results.push({fallbackToOrigin:true,requests});await close(ctx);
+ }
+ const retryLabels={en:'Retry','zh-CN':'重试',ja:'再試行',ko:'다시
시도',ru:'Повторить'};
+ for(const locale of Object.keys(retryLabels)) for(const fullPage of
[false,true]) {
+ const mode={failBoth:true},{ctx,page,requests}=await context(mode);
+ const prefix=locale==='en'?'':'/'+locale;
+ const
manifest=JSON.parse(fs.readFileSync(path.join(fixture,'.i18n-build/content-manifests',locale+'.json')));
+ const
doc=manifest.pages.find(p=>p.kind==='docs'&&p.pluginId==='default'&&p.version==='current'&&p.docId==='shared');
+ const
searchContext=['ja','ko','ru'].includes(locale)?'docs/dev':'docs';
+ await
open(page,fullPage?prefix+'/search/?q=Doris&ctx='+encodeURIComponent(searchContext)+'&version=docs/dev/':doc.route);
+ if(!fullPage)await input(page).fill('Doris');
+ const
retryButton=(fullPage?page.locator('.container.margin-vert--lg'):page.locator('.navbar__search'))
+ .getByRole('button',{name:retryLabels[locale],exact:true});
+ await retryButton.waitFor();
+ const themeChecks=[];
+ if(!fullPage) {
+ await page.setViewportSize({width:390,height:844});
+ for(const theme of
['doris','golden','blue','read','yellow-blue','purple','yellow-black','sky']) {
+ await
page.evaluate(value=>{document.documentElement.dataset.brandTheme=value;},theme);
+ const status=page.locator('.navbar__search
[role="status"]');
+ const box=await status.boundingBox();
+ const style=await
status.evaluate(e=>({color:getComputedStyle(e).color,background:getComputedStyle(e).backgroundColor,
+ scrollWidth:e.scrollWidth,clientWidth:e.clientWidth}));
+ assert(box.x>=-1&&box.x+box.width<=391,'Mobile search
retry overflows');
+ assert(style.scrollWidth<=style.clientWidth+1);
+ assert.notEqual(style.background,'rgba(0, 0, 0, 0)');
+ themeChecks.push({theme,box,...style});
+ }
+ }
+ const failedRequests=requests.length;mode.failBoth=false;
+ await retryButton.click();
+ if(fullPage)await page.locator('section
article').first().waitFor();
+ else {await suggestions(page).first().waitFor();assert.equal(await
input(page).inputValue(),'Doris');}
+ await retryButton.waitFor({state:'detached'});
+ assert(requests.length>failedRequests,'Retry must discard the
failed worker cache entry');
+
results.push({locale,fullPage,recoversWithoutReload:true,themeChecks,requests});await
close(ctx);
+ }
+ assert.deepEqual(errors,[]);
+ return {scope:'Small real five-language Rspack fixture only; not a
public deployment',
+
implementationSha:execFileSync('git',['rev-parse','HEAD'],{cwd:root,encoding:'utf8'}).trim(),results,errors,passed:true};
+ } finally {
+ for(const ctx of contexts)await close(ctx);
+ await browser.close();await
api.dispose();server.closeAllConnections?.();await new
Promise(resolve=>server.close(resolve));
+ }
+}
+async function main() {
+ if(!process.argv.includes('--verify-existing'))await build();
+ if(process.argv.includes('--browser')) {
+ const result=await
verify();fs.writeFileSync(path.join(root,'.i18n-build/search-transport-verification.json'),JSON.stringify(result,null,2)+'\n');
+
console.log(JSON.stringify({passed:result.passed,cases:result.results.length}));
+ }
+}
+if(require.main===module)main().catch(error=>{console.error(error);process.exitCode=1;});
diff --git a/scripts/i18n/verify-static.cjs b/scripts/i18n/verify-static.cjs
index cc8c2f433d4e..68389918b403 100644
--- a/scripts/i18n/verify-static.cjs
+++ b/scripts/i18n/verify-static.cjs
@@ -12,7 +12,7 @@ const htmlLangFor = locale => localeRegistry.find(item =>
item.code === locale)?
// current locale itself still run; a counterpart page that the subset did not
// build is reported as skipped instead of read. Publishing always verifies the
// complete site, where every counterpart must be present.
-function verify({site, manifest, partial = false}) {
+function verify({site, manifest, partial = false, headCache = new Map()}) {
const data = JSON.parse(fs.readFileSync(manifest, 'utf8'));
const locale = data.locale;
const contentFile = path.join(path.dirname(path.dirname(manifest)),
'content-manifests', locale + '.json');
@@ -40,9 +40,25 @@ function verify({site, manifest, partial = false}) {
readIndexes(path.join(site, locale));
let verified = 0;
let skipped = 0;
+ const headingIds = $ => $('.theme-doc-markdown h1,.theme-doc-markdown
h2,.theme-doc-markdown h3,.theme-doc-markdown h4,.theme-doc-markdown
h5,.theme-doc-markdown h6')
+ .map((i, node) => $(node).attr('id') ?? '').get();
+ function head(file, $) {
+ if (!headCache.has(file)) {
+ $ ??= cheerio.load(fs.readFileSync(file, 'utf8'));
+ // Retain only immutable metadata, not tens of thousands of DOMs.
+ const alternates = new Map();
+ for (const node of $('link[rel="alternate"][hreflang]').toArray())
{
+ const lang = $(node).attr('hreflang');
+ if (!alternates.has(lang)) alternates.set(lang, []);
+ alternates.get(lang).push($(node).attr('href'));
+ }
+ headCache.set(file, {headingIds:headingIds($), alternates});
+ }
+ return headCache.get(file);
+ }
function counterpart(permalink, check) {
- const file = path.join(site, permalink, 'index.html');
- if (fs.existsSync(file)) return cheerio.load(fs.readFileSync(file,
'utf8'));
+ const file = path.join(site, decodeURIComponent(permalink),
'index.html');
+ if (fs.existsSync(file)) return head(file);
assert(partial, `Missing counterpart page (${check}): ${permalink}`);
skipped++;
return null;
@@ -56,6 +72,7 @@ function verify({site, manifest, partial = false}) {
continue;
}
const $ = cheerio.load(fs.readFileSync(filename, 'utf8'));
+ head(filename, $);
const body = $('.theme-doc-markdown,.course-doc-content');
assert.equal(body.attr('lang'), page.contentLocale, `Body language:
${page.permalink}`);
assert.equal($('[data-translation-status]').attr('data-translation-status'),
page.status, `Notice: ${page.permalink}`);
@@ -72,26 +89,24 @@ function verify({site, manifest, partial = false}) {
for (const permalink of Object.values(alternateGroup)) {
const file = path.join(site, permalink, 'index.html');
if (fs.existsSync(file)) {
- const counterpart = cheerio.load(fs.readFileSync(file,
'utf8'));
-
assert.equal(counterpart(`link[rel="alternate"][hreflang="${htmlLang}"]`).length,
0,
+ assert.equal(head(file).alternates.get(htmlLang)?.length
?? 0, 0,
`Fallback/stale localized alternate in ${permalink}`);
}
}
} else {
assert(!($('meta[name="robots"]').attr('content') ??
'').includes('noindex'), page.permalink);
assert(sitemapPaths.has(page.permalink), `Current page missing
from sitemap: ${page.permalink}`);
- const ids = root => root('.theme-doc-markdown
h1,.theme-doc-markdown h2,.theme-doc-markdown h3,.theme-doc-markdown
h4,.theme-doc-markdown h5,.theme-doc-markdown h6').map((i, node) =>
root(node).attr('id') ?? '').get();
const english = counterpart(page.englishPermalink, 'source heading
IDs');
- if (english) assert.deepEqual(ids($), ids(english), `Source
heading IDs: ${page.permalink}`);
+ if (english) assert.deepEqual(headingIds($), english.headingIds,
`Source heading IDs: ${page.permalink}`);
for (const [lang, permalink] of Object.entries(alternateGroup)) {
const counterpart$ = counterpart(permalink, 'reciprocal
alternates');
if (!counterpart$) continue;
- const links = counterpart$('link[rel="alternate"][hreflang]');
for (const [otherLang, otherPath] of
Object.entries(alternateGroup)) {
const otherHtmlLang = htmlLangFor(otherLang);
-
assert.equal(links.filter(`[hreflang="${otherHtmlLang}"]`).attr('href'),
`https://doris.apache.org${otherPath}`,
+ const links = counterpart$.alternates.get(otherHtmlLang)
?? [];
+ assert.equal(links[0],
`https://doris.apache.org${otherPath}`,
`Non-reciprocal alternates ${lang} -> ${otherLang}:
${permalink}`);
-
assert.equal(links.filter(`[hreflang="${otherHtmlLang}"]`).length, 1,
`Duplicate hreflang: ${permalink}`);
+ assert.equal(links.length, 1, `Duplicate hreflang:
${permalink}`);
}
}
}
@@ -118,7 +133,7 @@ function verify({site, manifest, partial = false}) {
assert.equal($(`link[rel="alternate"][hreflang="${htmlLang}"]`).length, 0,
`Ordinary fallback alternate: ${permalink}`);
for (const sourceRoute of (decision ?
Object.values(decision.navigationRoutes) : [source, `/zh-CN${source}`])) {
const source$ = counterpart(sourceRoute, 'ordinary fallback
alternates');
- if (source$)
assert.equal(source$(`link[rel="alternate"][hreflang="${htmlLang}"]`).length, 0,
+ if (source$)
assert.equal(source$.alternates.get(htmlLang)?.length ?? 0, 0,
`Source advertises ordinary fallback: ${sourceRoute}`);
}
ordinaryPages.fallback++;
@@ -132,13 +147,24 @@ function verify({site, manifest, partial = false}) {
.map(status => [status, data.pages.filter(page => page.status ===
status).length])), reports: data.reports};
}
+function verifyAll({site, manifestDir, locales}) {
+ // One immutable artifact per invocation. Never share this cache across
runs.
+ const headCache = new Map();
+ return locales.map(locale => ({locale, ...verify({site,
+ manifest:path.join(manifestDir, locale + '.json'), headCache})}));
+}
+
if (require.main === module) {
const args = process.argv.slice(2);
const option = name => args[args.indexOf(name) + 1];
- const result = verify({site: path.resolve(option('--site')), manifest:
path.resolve(option('--manifest')),
- partial: args.includes('--partial')});
+ const site = path.resolve(option('--site'));
+ const result = args.includes('--all-managed') ? verifyAll({site,
+ manifestDir:path.resolve(args.includes('--manifest-dir') ?
option('--manifest-dir') : '.i18n-build/manifests'),
+ locales:require('./state.cjs').managedLocales(process.cwd(),
require('../../config/i18n').enabledLocales())})
+ : verify({site, manifest: path.resolve(option('--manifest')), partial:
args.includes('--partial')});
if (args.includes('--output')) fs.writeFileSync(option('--output'),
JSON.stringify(result, null, 2) + '\n');
- console.log(JSON.stringify({verified: result.verified, skipped:
result.skipped, partial: result.partial,
- states: result.states, ordinaryPages: result.ordinaryPages, reports:
result.reports.length}));
+ const summary = row =>
({locale:row.locale,verified:row.verified,skipped:row.skipped,partial:row.partial,
+
states:row.states,ordinaryPages:row.ordinaryPages,reports:row.reports.length});
+ console.log(JSON.stringify(Array.isArray(result) ? result.map(summary) :
summary(result)));
}
-module.exports = {verify};
+module.exports = {verify, verifyAll};
diff --git a/scripts/publish/__tests__/js-minimizer.test.cjs
b/scripts/publish/__tests__/js-minimizer.test.cjs
index f88f9c4a0a63..177a14875b2e 100644
--- a/scripts/publish/__tests__/js-minimizer.test.cjs
+++ b/scripts/publish/__tests__/js-minimizer.test.cjs
@@ -67,3 +67,26 @@ test('the production bundler preserves numeric-looking route
hashes as string ke
new vm.Script(code).runInNewContext(context);
assert.equal(JSON.stringify(context.window.registry),
JSON.stringify(registry));
});
+
+test('UTF-8 client strings remain exact without expanding translated prose to
ASCII escapes', async t => {
+ const root = fs.mkdtempSync(path.join(os.tmpdir(),
'doris-utf8-minifier-'));
+ t.after(() => fs.rmSync(root, {recursive: true, force: true}));
+ const values = {zh: '查询数据库', ja: 'データベースを検索', ko: '데이터베이스 검색',
+ ru: 'Поиск данных: ёж', special: '\u2028\u2029😀'};
+ fs.writeFileSync(path.join(root, 'entry.js'),
`window.prose=${JSON.stringify(values)};`);
+ const config = {mode:'production',optimization:{minimizer:[new
rspack.SwcJsMinimizerRspackPlugin()]}};
+ const optimization = plugin.configureWebpack(config, false).optimization;
+ const compiler = rspack({...config,context:root,entry:'./entry.js',
+ output:{path:path.join(root,'out'),filename:'main.js'},optimization});
+ const stats = await new
Promise((resolve,reject)=>compiler.run((error,result)=>error?reject(error):resolve(result)));
+ await new
Promise((resolve,reject)=>compiler.close(error=>error?reject(error):resolve()));
+ assert(!stats.hasErrors(),stats.toString({all:false,errors:true}));
+ const code = fs.readFileSync(path.join(root,'out/main.js'),'utf8');
+ const context = {window:{}};
+ new vm.Script(code).runInNewContext(context);
+ assert.equal(JSON.stringify(context.window.prose),JSON.stringify(values));
+ assert(code.includes(values.zh) && code.includes(values.ko));
+ const ascii = await require('terser').minify(code,
{compress:false,mangle:false,
+ format:{ecma:5,comments:false,ascii_only:true}});
+ assert(Buffer.byteLength(code) < Buffer.byteLength(ascii.code));
+});
diff --git a/scripts/publish/__tests__/optimize.test.cjs
b/scripts/publish/__tests__/optimize.test.cjs
index d96e69375ae7..b2407dafa878 100644
--- a/scripts/publish/__tests__/optimize.test.cjs
+++ b/scripts/publish/__tests__/optimize.test.cjs
@@ -70,6 +70,21 @@ test('hash filename collisions fail instead of silently
mixing languages', t =>
assert.throws(() => optimize({site,locales}),/filename collision/);
});
+test('legacy search URLs retain locale, context and query without copying the
index', async t => {
+ const {site,write} = fixture(t);
+ const target = 'ja/search-index-docs-dev-aabbccddeeff0011.json';
+ write(target, '[{"documents":[]}]');
+ optimize({site,locales});
+ assert(!fs.existsSync(path.join(site,'ja/search-index-docs-dev.json')));
+ const server = require('../serve.cjs').createServer(site);
+ await new Promise(resolve=>server.listen(0,'127.0.0.1',resolve));
+ t.after(()=>new
Promise(resolve=>{server.closeAllConnections?.();server.close(resolve);}));
+ const response = await
fetch(`http://127.0.0.1:${server.address().port}/ja/search-index-docs-dev.json?old=1`,{redirect:'manual'});
+ assert.equal(response.status,302);
+ assert.equal(response.headers.get('location'),'/'+target+'?old=1');
+ require('../verify.cjs').verify(site);
+});
+
test('retains distinct compiled URLs even when their binary contents match', t
=> {
const {site,write} = fixture(t);
write('assets/shared/other-name.PNG',Buffer.alloc(65536,42));
diff --git a/scripts/publish/__tests__/registry-loader.test.cjs
b/scripts/publish/__tests__/registry-loader.test.cjs
index c12b4bad553e..d64bd3e46464 100644
--- a/scripts/publish/__tests__/registry-loader.test.cjs
+++ b/scripts/publish/__tests__/registry-loader.test.cjs
@@ -54,7 +54,7 @@ for (const isServer of [true, false]) test(`production
${isServer ? 'SSR' : 'cli
optimization:{minimize:!isServer,minimizer:[new
rspack.SwcJsMinimizerRspackPlugin()]}};
const plugin =
createPlugin({siteDir:root,generatedFilesDir:generated,siteConfig:{baseUrl:'/'}});
const changes = plugin.configureWebpack(base,isServer);
- assert.equal(changes.module.rules.length,1);
+ assert.equal(changes.module.rules.filter(rule =>
rule.test(path.join(generated,'registry.js'))).length,1);
assert(changes.module.rules[0].test(path.join(generated,'registry.js')));
assert(!changes.module.rules[0].test(path.join(root,'registry.js')));
assert.equal(plugin.configureWebpack({...base,mode:'development'},isServer).module,undefined);
diff --git a/scripts/publish/optimize.cjs b/scripts/publish/optimize.cjs
index d593430ba478..f07e3ccb5c94 100644
--- a/scripts/publish/optimize.cjs
+++ b/scripts/publish/optimize.cjs
@@ -50,6 +50,14 @@ function optimize({site, locales, reportFile}) {
};
let sharedFiles = 0, redirectPages = 0;
+ // Cached clients may still request the old query-versioned index URL.
+ // Preserve it with a temporary alias, without duplicating the large JSON.
+ for (const item of before) {
+ if
(!/(?:^|\/)search-index(?:-[a-z0-9-]+)?-[a-f0-9]{16}\.json$/.test(item.relative))
continue;
+ const legacy = item.relative.replace(/-[a-f0-9]{16}\.json$/, '.json');
+ add(url(legacy), url(item.relative), true, 302);
+ }
+
// All webpack runtimes already reference the root pool, including lazy
// chunks and same-origin search workers. Never rewrite a hashed bundle.
for (const item of before) {
diff --git a/src/theme/SearchBar/SearchBar.jsx
b/src/theme/SearchBar/SearchBar.jsx
index 9128bb7afc54..2f713672ccfa 100644
--- a/src/theme/SearchBar/SearchBar.jsx
+++ b/src/theme/SearchBar/SearchBar.jsx
@@ -19,7 +19,7 @@ import {
useAllContextsWithNoSearchContext,
} from '../../utils/proxiedGenerated';
import LoadingRing from '../LoadingRing/LoadingRing';
-import { VERSIONS, DEFAULT_VERSION } from '@site/src/constant/version';
+import { VERSIONS } from '@site/src/constant/version';
import styles from './SearchBar.module.css';
import { normalizeContextByPath } from '../../utils/normalizeContextByPath';
import { searchResultLimits } from "../../utils/proxiedGeneratedConstants";
@@ -30,6 +30,7 @@ import {searchWithLocaleFallback} from
'@site/src/utils/searchInput';
import {fetchContextIndexes, searchAcrossContexts} from
'@site/src/utils/searchContexts';
import {isTranslatedLocale} from '@site/src/utils/localization';
import {searchIndexBaseUrl} from '@site/src/utils/searchVersion';
+import {searchMessages} from '@site/src/utils/searchMessages';
const searchContextsByWorker = searchAcrossContexts(searchByWorker);
@@ -49,7 +50,6 @@ async function fetchAutoCompleteJS() {
const SEARCH_PARAM_HIGHLIGHT = '_highlight';
export default function SearchBar({ handleSearchBarToggle }) {
const isBrowser = useIsBrowser();
- const [curVersion, setCurVersion] = useState(DEFAULT_VERSION);
const location = useLocation();
const {
siteConfig: { baseUrl },
@@ -69,6 +69,7 @@ export default function SearchBar({ handleSearchBarToggle }) {
// Should the input be focused after the index is loaded?
const focusAfterIndexLoaded = useRef(false);
const [loading, setLoading] = useState(false);
+ const [loadError, setLoadError] = useState(false);
const [inputChanged, setInputChanged] = useState(false);
const [inputValue, setInputValue] = useState('');
const search = useRef(null);
@@ -76,7 +77,6 @@ export default function SearchBar({ handleSearchBarToggle }) {
// finished after a newer call started) can abort before wrapping the
// input with a second autocomplete instance — see #3655 race.
const loadToken = useRef(0);
- const prevSearchContext = useRef('');
const [searchContext, setSearchContext] = useState('');
useEffect(() => {
if (!Array.isArray(searchContextByPaths)) {
@@ -108,30 +108,47 @@ export default function SearchBar({ handleSearchBarToggle
}) {
if (matchedPath) {
nextSearchContext = matchedPath;
}
- if (prevSearchContext.current !== nextSearchContext) {
- // Reset index state map once search context is changed.
- indexStateMap.current.delete(nextSearchContext);
- prevSearchContext.current = nextSearchContext;
- }
setSearchContext(nextSearchContext);
}, [location.pathname, locales, currentLocale]);
const hidden = !!hideSearchBarWithNoSearchContext &&
Array.isArray(searchContextByPaths) && searchContext === '';
+ const indexKey = `${versionUrl}:${searchContext}`;
+
+ useEffect(() => {
+ setLoadError(false);
+ setLoading(false);
+ return () => {
+ ++loadToken.current;
+ indexStateMap.current.clear();
+ search.current?.autocomplete.destroy();
+ search.current = null;
+ };
+ }, [versionUrl, searchContext]);
const loadIndex = useCallback(
async (forceLoad = false) => {
- if (hidden || (indexStateMap.current.get(searchContext) &&
!forceLoad)) {
+ if (hidden || (indexStateMap.current.get(indexKey) && !forceLoad))
{
// Do not load the index (again) if its already loaded or in
the process of being loaded.
return;
}
const myToken = ++loadToken.current;
- indexStateMap.current.set(searchContext, 'loading');
+ indexStateMap.current.set(indexKey, 'loading');
search.current?.autocomplete.destroy();
search.current = null;
setLoading(true);
- const [autoComplete] = await Promise.all([
- fetchAutoCompleteJS(),
- fetchContextIndexes(fetchIndexesByWorker, versionUrl,
searchContext),
- ]);
+ setLoadError(false);
+ let autoComplete;
+ try {
+ [autoComplete] = await Promise.all([
+ fetchAutoCompleteJS(),
+ fetchContextIndexes(fetchIndexesByWorker, versionUrl,
searchContext),
+ ]);
+ } catch {
+ if (myToken !== loadToken.current) return;
+ indexStateMap.current.delete(indexKey);
+ setLoading(false);
+ setLoadError(true);
+ return;
+ }
if (myToken !== loadToken.current) {
// Superseded by a newer loadIndex; don't mount a second
autocomplete on the same input.
return;
@@ -201,6 +218,7 @@ export default function SearchBar({ handleSearchBarToggle
}) {
});
return a;
};
+ const pendingQuery = searchBarRef.current.value;
search.current = autoComplete(
searchBarRef.current,
{
@@ -230,8 +248,15 @@ export default function SearchBar({ handleSearchBarToggle
}) {
[
{
source: debounce(async (input, callback) => {
- const result = await
searchWithLocaleFallback(searchContextsByWorker, versionUrl, searchContext,
input, searchResultLimits);
- callback(result);
+ try {
+ const result = await
searchWithLocaleFallback(searchContextsByWorker, versionUrl, searchContext,
input, searchResultLimits);
+ callback(result);
+ } catch {
+ if (myToken !== loadToken.current) return;
+ indexStateMap.current.delete(indexKey);
+ setLoadError(true);
+ callback([]);
+ }
}, 300),
templates: {
suggestion: SuggestionTemplate,
@@ -268,7 +293,13 @@ export default function SearchBar({ handleSearchBarToggle
}) {
.on('autocomplete:closed', () => {
searchBarRef.current?.blur();
});
- indexStateMap.current.set(searchContext, 'done');
+ // Autocomplete was not listening while its index loaded. Replay
+ // the current input so the user's first query needs no extra key.
+ if (pendingQuery) {
+ search.current.autocomplete.setVal('');
+ search.current.autocomplete.setVal(pendingQuery);
+ }
+ indexStateMap.current.set(indexKey, 'done');
setLoading(false);
if (focusAfterIndexLoaded.current) {
const input = searchBarRef.current;
@@ -278,7 +309,7 @@ export default function SearchBar({ handleSearchBarToggle
}) {
input.focus();
}
},
- [hidden, searchContext, versionUrl, baseUrl, history],
+ [hidden, indexKey, searchContext, versionUrl, baseUrl, history],
);
useEffect(() => {
if (!Mark) {
@@ -336,26 +367,15 @@ export default function SearchBar({ handleSearchBarToggle
}) {
// Implement hint icons for the search shortcuts on mac and the rest
operating systems.
const isMac = isBrowser ? /mac/i.test(navigator.userAgentData?.platform ??
navigator.platform) : false;
- useEffect(() => {
- const pathHaveVer = VERSIONS.some(item =>
location.pathname.includes(item));
- if (!pathHaveVer && curVersion !== '2.1') {
- setCurVersion('2.1');
- } else {
- VERSIONS.forEach(item => {
- if (location.pathname.includes(item) && item !== curVersion) {
- setCurVersion(item);
- }
- });
- }
- }, [location.pathname]);
useEffect(() => {
// On mount the page's search context is not resolved yet, and an empty
- // one would load every context index. Leave that to focus or hover.
- if (!searchContext && useAllContextsWithNoSearchContext) {
+ // one can load the wrong index or every index. Leave outside-context
+ // search to focus or hover, including the source-language search page.
+ if (!searchContext) {
return;
}
loadIndex(true);
- }, [curVersion]);
+ }, [loadIndex, searchContext]);
useEffect(() => {
if (!searchBarShortcut) {
return;
@@ -413,6 +433,7 @@ export default function SearchBar({ handleSearchBarToggle
}) {
[styles.focused]: focused,
})}
hidden={hidden}
+ aria-busy={loading || undefined}
// Manually make the search bar be LTR even if in RTL
dir="ltr"
>
@@ -428,6 +449,12 @@ export default function SearchBar({ handleSearchBarToggle
}) {
value={inputValue}
/>
<LoadingRing className={styles.searchBarLoadingRing} />
+ {loadError && <div className={styles.searchLoadError}
role="status">
+ <span>{searchMessages(currentLocale).error}</span>
+ <button type="button" onClick={() => loadIndex(true)}>
+ {searchMessages(currentLocale).retry}
+ </button>
+ </div>}
{searchBarShortcut &&
searchBarShortcutHint &&
(inputValue !== '' ? (
diff --git a/src/theme/SearchBar/SearchBar.module.css
b/src/theme/SearchBar/SearchBar.module.css
index f39dc20019b3..e9c6a418aa85 100644
--- a/src/theme/SearchBar/SearchBar.module.css
+++ b/src/theme/SearchBar/SearchBar.module.css
@@ -12,6 +12,39 @@
padding: var(--search-local-spacing, 12px);
}
+.searchLoadError {
+ position: absolute;
+ z-index: 20;
+ top: 100%;
+ right: 0;
+ display: flex;
+ align-items: center;
+ gap: 0.5rem;
+ max-width: min(22rem, calc(100vw - 2rem));
+ padding: 0.5rem 0.75rem;
+ border: 1px solid var(--brand-border-soft);
+ border-radius: 6px;
+ background: var(--brand-surface-note);
+ color: var(--brand-ink);
+ font-size: 0.875rem;
+}
+
+.searchLoadError button {
+ flex-shrink: 0;
+ border: 1px solid currentColor;
+ border-radius: 4px;
+ padding: 0.15rem 0.5rem;
+ background: transparent;
+ color: inherit;
+ font: inherit;
+ cursor: pointer;
+}
+
+.searchLoadError > span {
+ min-width: 0;
+ white-space: normal;
+}
+
.searchBar .mobileDropdownMenu {
left: 0 !important;
}
diff --git a/src/theme/SearchPage/SearchPage.jsx
b/src/theme/SearchPage/SearchPage.jsx
index 1c791614f61a..0530fe0cadaa 100644
--- a/src/theme/SearchPage/SearchPage.jsx
+++ b/src/theme/SearchPage/SearchPage.jsx
@@ -24,6 +24,7 @@ import {searchWithLocaleFallback} from
"../../utils/searchInput";
import {fetchContextIndexes, searchAcrossContexts} from
"../../utils/searchContexts";
import {isTranslatedLocale} from "../../utils/localization";
import {searchIndexBaseUrl} from "../../utils/searchVersion";
+import {searchMessages} from "../../utils/searchMessages";
const searchContextsByWorker = searchAcrossContexts(searchByWorker);
export default function SearchPage() {
return (<Layout>
@@ -37,6 +38,9 @@ function SearchPageContent() {
const { searchValue, searchContext, searchVersion, updateSearchPath,
updateSearchContext, } = useSearchQuery();
const [searchQuery, setSearchQuery] = useState(searchValue);
const [searchResults, setSearchResults] = useState();
+ const [searchWorkerReady, setSearchWorkerReady] = useState(false);
+ const [loadError, setLoadError] = useState(false);
+ const [retry, setRetry] = useState(0);
const sourceVersions = useAllDocsData()[docsPluginIdForPreferredVersion ??
'default']?.versions ?? [];
const versionUrl = isTranslatedLocale(currentLocale) ? baseUrl
: searchIndexBaseUrl(baseUrl, sourceVersions, {versionPath:
searchVersion});
@@ -55,18 +59,9 @@ function SearchPageContent() {
}), [searchQuery]);
useEffect(() => {
updateSearchPath(searchQuery);
- if (searchQuery) {
- (async () => {
- const results = await
searchWithLocaleFallback(searchContextsByWorker, versionUrl, searchContext,
searchQuery, searchResultLimits);
- setSearchResults(results);
- })();
- }
- else {
- setSearchResults(undefined);
- }
// `updateSearchPath` should not be in the deps,
// otherwise will cause call stack overflow.
- }, [searchQuery, versionUrl, searchContext]);
+ }, [searchQuery]);
const handleSearchInputChange = useCallback((e) => {
setSearchQuery(e.target.value);
}, []);
@@ -75,7 +70,6 @@ function SearchPageContent() {
setSearchQuery(searchValue);
}
}, [searchValue]);
- const [searchWorkerReady, setSearchWorkerReady] = useState(false);
// The source index lives at the site root, so drop this locale's prefix.
const sourceBaseUrl = isTranslatedLocale(currentLocale)
? baseUrl.replace(new
RegExp(`${currentLocale.replace(/[.*+?^${}()|[\]\\]/g, '\\$&')}/$`), '')
@@ -84,20 +78,33 @@ function SearchPageContent() {
...(searchContext?.startsWith('docs/') ? {ctx: 'docs'} : {}),
...(searchContext === 'docs/dev' ? {version: 'docs/dev/'} :
{})}).toString()}`;
useEffect(() => {
+ let active = true;
+ setLoadError(false);
+ setSearchWorkerReady(false);
+ setSearchResults(undefined);
async function doFetchIndexes() {
- if (!Array.isArray(searchContextByPaths) ||
- searchContext ||
- useAllContextsWithNoSearchContext) {
- await fetchContextIndexes(fetchIndexesByWorker, versionUrl,
searchContext);
+ try {
+ if (!Array.isArray(searchContextByPaths) ||
+ searchContext ||
+ useAllContextsWithNoSearchContext) {
+ await fetchContextIndexes(fetchIndexesByWorker,
versionUrl, searchContext);
+ }
+ if (searchQuery) {
+ const results = await
searchWithLocaleFallback(searchContextsByWorker, versionUrl, searchContext,
searchQuery, searchResultLimits);
+ if (active) setSearchResults(results);
+ }
+ if (active) setSearchWorkerReady(true);
+ } catch {
+ if (active) setLoadError(true);
}
- setSearchWorkerReady(true);
}
// The context comes from the query string, which is only read in the
// browser; before that an empty one would load every context index.
if (isBrowser) {
doFetchIndexes();
}
- }, [isBrowser, searchContext, versionUrl]);
+ return () => {active = false;};
+ }, [isBrowser, searchContext, versionUrl, searchQuery, retry]);
return (<React.Fragment>
<Head>
{/*
@@ -137,9 +144,15 @@ function SearchPageContent() {
</div>) : null}
</div>
- {!searchWorkerReady && searchQuery && (<div>
+ {!searchWorkerReady && !loadError && searchQuery && (<div>
<LoadingRing />
</div>)}
+ {loadError && <p role="status">
+ {searchMessages(currentLocale).error}{' '}
+ <button type="button" className="button button--secondary
button--sm" onClick={() => setRetry(value => value + 1)}>
+ {searchMessages(currentLocale).retry}
+ </button>
+ </p>}
{searchResults &&
(searchResults.length > 0 ? (<p>
diff --git a/src/utils/fetchSearchIndex.js b/src/utils/fetchSearchIndex.js
new file mode 100644
index 000000000000..7389b3e96636
--- /dev/null
+++ b/src/utils/fetchSearchIndex.js
@@ -0,0 +1,32 @@
+// Only production pages use the configured asset CDN. Local previews and
+// alternate hosts read their own index, just like their own same-origin
worker.
+export async function fetchSearchIndex(url, {
+ assetOrigin = '', siteOrigin = '', origin = globalThis.location?.origin,
+ timeoutMs = 60000, fetchImpl = globalThis.fetch,
+} = {}) {
+ const source = new URL(url, origin);
+ if (source.origin !== origin) throw new Error('Unexpected search index
origin');
+ const candidates = [source];
+ if (assetOrigin && siteOrigin && new URL(siteOrigin).origin === origin) {
+ const cdn = new URL(source.pathname + source.search, assetOrigin);
+ if (!['http:', 'https:'].includes(cdn.protocol)) throw new
Error('Invalid search asset origin');
+ if (cdn.origin !== source.origin) candidates.unshift(cdn);
+ }
+ let failure;
+ for (const candidate of candidates) {
+ const controller = new AbortController();
+ const timeout = setTimeout(() => controller.abort(), timeoutMs);
+ try {
+ const response = await fetchImpl(candidate.href, {signal:
controller.signal});
+ if (!response.ok) throw new Error(`Search index HTTP
${response.status}`);
+ const data = await response.json();
+ if (!Array.isArray(data)) throw new Error('Invalid search index');
+ return data;
+ } catch (error) {
+ failure = error;
+ } finally {
+ clearTimeout(timeout);
+ }
+ }
+ throw failure;
+}
diff --git a/src/utils/searchMessages.ts b/src/utils/searchMessages.ts
new file mode 100644
index 000000000000..97ac699b1b47
--- /dev/null
+++ b/src/utils/searchMessages.ts
@@ -0,0 +1,8 @@
+const messages = {
+ en: {error: 'Search could not load.', retry: 'Retry'},
+ 'zh-CN': {error: '搜索加载失败。', retry: '重试'},
+ ja: {error: '検索を読み込めませんでした。', retry: '再試行'},
+ ko: {error: '검색을 불러오지 못했습니다.', retry: '다시 시도'},
+ ru: {error: 'Не удалось загрузить поиск.', retry: 'Повторить'},
+};
+export function searchMessages(locale: string) {return messages[locale] ??
messages.en;}
diff --git a/src/utils/translation.ts b/src/utils/translation.ts
index aad55b6e93f1..bab809121edb 100644
--- a/src/utils/translation.ts
+++ b/src/utils/translation.ts
@@ -63,7 +63,6 @@ export type ContentRouteData = {
aggregate?: boolean;
schemaVersion: 1;
renderIntegrated: boolean;
- sourceSha: string;
contentId: string;
kind: 'docs' | 'page' | 'blog' | 'utility';
siteLocale: string;
diff --git a/static/.htaccess b/static/.htaccess
index b4144a54009b..06d4949483ab 100644
--- a/static/.htaccess
+++ b/static/.htaccess
@@ -1,5 +1,5 @@
<IfModule mod_headers.c>
- Header set Content-Security-Policy "script-src 'self'
https://cdnd.selectdb.com https://analytics.apache.org widget.kapa.ai
www.google.com https://hcaptcha.com https://*.hcaptcha.com
https://www.gstatic.com 'unsafe-inline' 'unsafe-eval'; connect-src 'self'
https://analytics.apache.org proxy.kapa.ai
kapa-widget-proxy-la7dkmplpq-uc.a.run.app metrics.kapa.ai https://hcaptcha.com
https://*.hcaptcha.com www.google.com https://agent.velodb.io; frame-src 'self'
www.google.com https://hcaptc [...]
+ Header set Content-Security-Policy "script-src 'self'
https://cdnd.selectdb.com https://analytics.apache.org widget.kapa.ai
www.google.com https://hcaptcha.com https://*.hcaptcha.com
https://www.gstatic.com 'unsafe-inline' 'unsafe-eval'; connect-src 'self'
https://cdnd.selectdb.com https://analytics.apache.org proxy.kapa.ai
kapa-widget-proxy-la7dkmplpq-uc.a.run.app metrics.kapa.ai https://hcaptcha.com
https://*.hcaptcha.com www.google.com https://agent.velodb.io; frame-src 'self'
www [...]
</IfModule>
# Native and optimized redirect manifests retain only known historical aliases.
diff --git a/translations/shared/registry.json
b/translations/shared/registry.json
index 1abdd72ebf85..753300e52ef7 100644
--- a/translations/shared/registry.json
+++ b/translations/shared/registry.json
@@ -65,6 +65,7 @@
"src/utils/proxiedGeneratedConstants.ts",
"src/utils/searchContexts.js",
"src/utils/searchInput.js",
+ "src/utils/searchMessages.ts",
"src/utils/searchNormalization.js",
"src/utils/searchVersion.js",
"src/utils/tokenize.ts",
@@ -173,7 +174,7 @@
"theme.SearchPage.noResultsText": "No documents were found",
"theme.SearchPage.searchSource": "Search the source documents"
},
- "sourceHash":
"67a454376f885cd92b6784830b91fe6a8e495897a74bcd44bb9ca706445e78c1",
+ "sourceHash":
"9032b3ab8106811aad12c49fddb2b59f83e80c7f8892f2e532db634f446e1595",
"entryFiles": [
"src/components/home-next/NavbarNext.tsx",
"src/components/home-next/SlackCommunityNudge.tsx",
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
