This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new e621d097fb5d ci: use scoped document compilation for pull request
validation (#4497)
e621d097fb5d is described below
commit e621d097fb5d77ffddff47fe5f5179511e10304c
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Sat Oct 24 13:33:48 2026 +0100
ci: use scoped document compilation for pull request validation (#4497)
`Build Check` currently compiles complete documentation sets even for CI
tooling or small content changes. This makes PR feedback depend on a
five-language publication build. This change selects validation from the
merge-base diff while keeping complete native document metadata
available for route/sidebar and translation decisions.
| Changed inputs | PR validation |
| --- | --- |
| CI/tooling | Unit tests, a five-language native compilation fixture,
publishing/Apache/Chromium fixture |
| Shared UI/routing/SEO/sidebars | Five-language representative real
documents across every active version, plus native content and
publishing fixtures |
| Documents/assets | Changed documents, import dependants and
representative documents in affected languages; English source edits
select all five languages |
| Dependencies, active versions, unknown inputs, `ci/full` label | Full
five-language publication build and artifact/browser verification |
The CI-only adapter prunes unselected document modules before
compilation and SSR. It retains native IDs, front matter, sidebars,
routes and translation state for the complete document set, and
preserves non-document pages. Changes to document/state records are
mandatory; import dependency discovery is transitive and conservative
for dynamic/unparseable imports. Samples above 500 documents fail with
instructions to request full validation instead of truncating the
affected set.
The plan, actual compiled MDX modules, rendered counts and skipped
destination-anchor checks are recorded in artifacts and the job summary.
Sample output is confined to `.i18n-build/ci-sample`, carries a marker
rejected by publication optimization/verification, and is never uploaded
as a publishable site. The adapter is pinned to Docusaurus 3.6.3.
Unselected body syntax and destination anchors are outside sample
coverage; full scheduled/manual deployment builds and explicit full PR
validation remain available.
The workflow also avoids empty translation inventory scans (reporting
freshness as unscanned), batches unit tests, removes the duplicate link
scan already included in docs governance, and fixes the Linux Apache
MIME path and HTTP probe resource handling so publishing fixtures run
reliably. PR checkout remains read-only, credential-free and bound to
the workflow's merge SHA.
Validation:
- i18n tests: 246 passed, one existing opt-in test skipped; publishing
tests: 14 passed with real Apache; workflow security: 22 passed;
actionlint passed.
- Five-language sampler fixture and Chromium hydration passed. A
deliberately broken import is excluded when unchanged and fails
compilation after its document changes.
- Final-head real-document samples passed in all five languages: each
loads metadata for 5,948 documents and renders 32 representative
document routes across all active versions. Each compiles 254 actual MDX
modules including blogs/imports and 272 routes including non-document
pages. Static/translation SEO and Chromium hydration verification
passed; one omitted destination-anchor check per language is disclosed.
Local five-language compilation totaled 11m42s.
- Final-head GitHub [Build
Check](https://github.com/apache/doris-website/actions/runs/40093073635)
passed in **4m49s** in tooling/fixture mode. All five PR checks passed.
The job's downloaded plan and fixture proof bind to merge SHA
`c7979cd84482bf87b2f4e1e6b55972580f3d02df`, matching PR head
`ee53f81d0bb87bdc24f78c43bd22dc3cba590a0d`.
- These timings reflect reduced validation scope, not a controlled
performance comparison of equivalent full builds.
This PR is based directly on Apache master and can be reviewed
independently of #4494. See `scripts/i18n/CI_VALIDATION.md` for policy,
limits, reports and local commands.
---------
Co-authored-by: morningman <[email protected]>
---
.github/workflows/build-check.yml | 140 ++---
scripts/build-locale.cjs | 7 +-
scripts/build-locales.cjs | 14 +-
scripts/i18n/CI_VALIDATION.md | 90 +++
scripts/i18n/__tests__/admission-scope.test.cjs | 49 ++
scripts/i18n/__tests__/ci-sampling.test.cjs | 217 +++++++
scripts/i18n/__tests__/multi-locale.test.cjs | 2 +-
scripts/i18n/build-ci-sample.cjs | 20 +
scripts/i18n/check-translations.cjs | 38 +-
scripts/i18n/ci-plan.cjs | 166 ++++++
scripts/i18n/ci-sample.cjs | 655 ++++++++++++++++++++++
scripts/i18n/pr-preflight.cjs | 3 +-
scripts/i18n/verify-ci-sample.cjs | 115 ++++
scripts/i18n/verify-ci-sampling-fixture.cjs | 120 ++++
scripts/publish/__tests__/apache-startup.test.cjs | 23 +
scripts/publish/apache.cjs | 43 +-
scripts/publish/optimize.cjs | 1 +
scripts/publish/verify-fixture.cjs | 13 +-
scripts/publish/verify.cjs | 1 +
19 files changed, 1605 insertions(+), 112 deletions(-)
diff --git a/.github/workflows/build-check.yml
b/.github/workflows/build-check.yml
index d6ec352771b8..0f43010be13e 100644
--- a/.github/workflows/build-check.yml
+++ b/.github/workflows/build-check.yml
@@ -20,6 +20,14 @@ name: Build Check
on:
pull_request:
+ types: [opened, synchronize, reopened, labeled, unlabeled]
+ workflow_dispatch:
+ inputs:
+ validation:
+ description: Validation scope
+ type: choice
+ options: [full, sample]
+ default: full
concurrency:
group: build-check-${{ github.event.pull_request.number || github.ref }}
@@ -32,6 +40,8 @@ jobs:
build-and-deploy:
name: Build Check
runs-on: ubuntu-22.04
+ env:
+ BASE_REF: ${{ github.base_ref ||
github.event.repository.default_branch }}
steps:
- name: Checkout PR merge commit (${{ github.sha }})
uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262
# v4
@@ -62,7 +72,7 @@ jobs:
git config --global user.name "github-action-test"
# Get the base commit of the PR
- BASE_COMMIT=$(git merge-base origin/${{ github.base_ref }}
HEAD)
+ BASE_COMMIT=$(git merge-base "origin/$BASE_REF" HEAD)
# Run the Python script with the base commit range
python check_move.py "$BASE_COMMIT"
@@ -71,27 +81,31 @@ jobs:
- name: Run Node.js dead link checker
env:
PR_HEAD_SHA: ${{ github.event.pull_request.head.sha }}
+ CI_SHA: ${{ github.sha }}
run: |
# git show on a clean merge omits the PR's changed files.
- node ./scripts/check_move.js "$PR_HEAD_SHA"
+ node ./scripts/check_move.js "${PR_HEAD_SHA:-$CI_SHA}"
- name: Detect changed doc versions and locales
id: detect
run: |
- BASE_SHA=$(git merge-base origin/${{ github.base_ref }} HEAD)
+ BASE_SHA=$(git merge-base "origin/$BASE_REF" HEAD)
node scripts/i18n/check-configuration.cjs --all
node scripts/i18n/build-scope.cjs "$BASE_SHA"
+ node scripts/i18n/ci-plan.cjs "$BASE_SHA"
- name: Prepare swap for the localized build
- if: steps.detect.outputs.enabled_locales != '' &&
steps.detect.outputs.skip_build != 'true'
+ if: steps.detect.outputs.validation_mode == 'full' ||
steps.detect.outputs.validation_mode == 'sample'
run: |
sudo fallocate -l 8G /mnt/japanese-build-check-swap
sudo chmod 600 /mnt/japanese-build-check-swap
sudo mkswap /mnt/japanese-build-check-swap
sudo swapon /mnt/japanese-build-check-swap
- # Dependencies are required even when the scoped site build is
skipped.
+ # Fixture validation also needs dependencies; prose-only changes
do not.
- name: Install dependencies
+ id: install
+ if: steps.detect.outputs.validation_mode != 'skip'
env:
NODE_OPTIONS: --max-old-space-size=8192
run: |
@@ -102,41 +116,29 @@ jobs:
- name: Translation unit tests
env:
NODE_OPTIONS: --max-old-space-size=8192
+ if: steps.detect.outputs.validation_mode != 'skip'
+ run: node --test scripts/i18n/__tests__/*.test.cjs
+
+ - name: Publish artifact optimization tests
+ if: steps.detect.outputs.publish_fixture == 'true'
+ env:
+ NODE_OPTIONS: --max-old-space-size=2048
+ DORIS_TEST_APACHE: "true"
run: |
- node --test scripts/i18n/__tests__/t04.test.cjs
- node --test scripts/i18n/__tests__/heading-ids.test.cjs
- node --test scripts/i18n/__tests__/search-input.test.cjs
- node --test scripts/i18n/__tests__/search-contexts.test.cjs
- node --test scripts/i18n/__tests__/search-version.test.cjs
- node --test scripts/i18n/__tests__/configuration.test.cjs
- node --test
scripts/i18n/__tests__/search-verification.test.cjs
- node --test scripts/i18n/__tests__/korean-tokenizer.test.cjs
- node --test scripts/i18n/__tests__/russian-search.test.cjs
- node --test scripts/i18n/__tests__/doc-navigation.test.cjs
- node --test scripts/i18n/__tests__/marketing-pages.test.cjs
- node --test scripts/i18n/__tests__/locale-registry.test.cjs
- node --test scripts/i18n/__tests__/multi-locale.test.cjs
- node --test scripts/i18n/__tests__/content-manifest.test.cjs
- node --test scripts/i18n/__tests__/inventory.test.cjs
- node --test scripts/i18n/__tests__/planner.test.cjs
- node --test
scripts/i18n/__tests__/translation-engine.test.cjs
- node --test scripts/i18n/__tests__/korean-toolchain.test.cjs
- node --test
scripts/i18n/__tests__/translation-workflows.test.cjs
- node --test scripts/i18n/__tests__/workflow.test.cjs
- node --test scripts/i18n/__tests__/codex-adapter.test.cjs
- node --test
scripts/i18n/__tests__/translate-document.test.cjs
- node --test
scripts/i18n/__tests__/incremental-document.test.cjs
+ node --test scripts/publish/__tests__/*.test.cjs
+ node scripts/publish/verify-fixture.cjs
- name: Verify native content manifest fixture
- if: steps.detect.outputs.optimize_publish == 'true'
+ if: steps.detect.outputs.content_fixture == 'true'
run: node scripts/i18n/verify-content-fixture.cjs
- name: Workflow security boundary tests
run: PYTHONDONTWRITEBYTECODE=1 python3 -m unittest discover -s
.github/scripts -p 'test_*.py' -v
- name: Translation gate for changed locales
+ if: steps.detect.outputs.validation_mode != 'skip'
run: |
- BASE_SHA=$(git merge-base origin/${{ github.base_ref }} HEAD)
+ BASE_SHA=$(git merge-base "origin/$BASE_REF" HEAD)
CHANGED=$(git diff --name-only "$BASE_SHA" HEAD)
# A shared input can change every locale's admission result;
# otherwise only locales whose own translation inputs changed
@@ -153,70 +155,62 @@ jobs:
fi
done
- - name: Publish artifact optimization tests
- env:
- NODE_OPTIONS: --max-old-space-size=2048
- run: |
- node --test scripts/publish/__tests__/*.test.cjs
- if [ "${{ steps.detect.outputs.optimize_publish }}" = "true"
]; then
- node scripts/publish/verify-fixture.cjs
- fi
-
- name: Install browser verification tools
- if: steps.detect.outputs.optimize_publish == 'true'
+ if: steps.detect.outputs.validation_mode == 'full' ||
steps.detect.outputs.validation_mode == 'sample' ||
steps.detect.outputs.publish_fixture == 'true'
run: |
DORIS_BROWSER_TOOLS="$RUNNER_TEMP/doris-browser-tools"
npm install --prefix "$DORIS_BROWSER_TOOLS" --no-save
--no-package-lock [email protected]
"$DORIS_BROWSER_TOOLS/node_modules/.bin/playwright" install
--with-deps chromium
echo "NODE_PATH=$DORIS_BROWSER_TOOLS/node_modules" >>
"$GITHUB_ENV"
+ - name: Verify CI sampling adapter
+ if: steps.detect.outputs.sampling_fixture == 'true'
+ run: node scripts/i18n/verify-ci-sampling-fixture.cjs --browser
+
- name: Verify five-language publishing fixture in Chromium
- if: steps.detect.outputs.optimize_publish == 'true'
+ if: steps.detect.outputs.publish_fixture == 'true'
env:
NODE_OPTIONS: --max-old-space-size=4096
run: node scripts/publish/verify-fixture.cjs --verify-existing
--browser
- name: Build
env:
- DORIS_OPTIMIZE_PUBLISH: ${{
steps.detect.outputs.optimize_publish }}
+ DORIS_OPTIMIZE_PUBLISH: ${{
steps.detect.outputs.validation_mode == 'full' }}
DORIS_ASSET_ORIGIN: ""
- DOCS_VERSIONS: ${{ steps.detect.outputs.docs_versions }}
- DORIS_ENABLE_LOCALES: ${{
steps.detect.outputs.enabled_locales }}
+ DOCS_VERSIONS: ""
+ DORIS_ENABLE_LOCALES: ${{
steps.detect.outputs.validation_enabled_locales }}
PWA_SERVICE_WORKER_URL: https://doris.apache.org/sw.js
NODE_OPTIONS: --max-old-space-size=8192
run: |
- if [ "${{ steps.detect.outputs.skip_build }}" = "true" ];
then
- echo "No site-affecting changes detected — skipping the
docusaurus build."
+ MODE="${{ steps.detect.outputs.validation_mode }}"
+ if [ "$MODE" = "skip" ] || [ "$MODE" = "fixtures" ]; then
+ echo "No real-site build required: validation mode=$MODE"
exit 0
fi
-
- echo "Building with DOCS_VERSIONS=${DOCS_VERSIONS:-all}
LOCALES=${{ steps.detect.outputs.locale_csv }}"
mkdir -p .i18n-build
free -m > .i18n-build/ci-memory-before.txt
- if [ "${{ steps.detect.outputs.optimize_publish }}" = "true"
]; then
- /usr/bin/time -v -o .i18n-build/ci-build-time.txt yarn
build:publish --locales "${{ steps.detect.outputs.locale_csv }}"
- elif [ "${{ steps.detect.outputs.locale_count }}" = "1" ] &&
[ "${{ steps.detect.outputs.enabled_locales }}" = "" ]; then
- PWA_SERVICE_WORKER_URL=https://doris.apache.org/sw.js
yarn docusaurus build ${{ steps.detect.outputs.locale_args }}
+ if [ "$MODE" = "sample" ]; then
+ /usr/bin/time -v -o .i18n-build/ci-build-time.txt node
scripts/i18n/build-ci-sample.cjs
else
- # Every locale gets a separate Node process and its real
prefix.
- yarn build:locales --locales "${{
steps.detect.outputs.locale_csv }}"
+ /usr/bin/time -v -o .i18n-build/ci-build-time.txt yarn
build:publish --locales "${{ steps.detect.outputs.validation_locales }}"
fi
free -m > .i18n-build/ci-memory-after.txt
+ - name: Verify sampled compilation
+ if: steps.detect.outputs.validation_mode == 'sample'
+ run: node scripts/i18n/verify-ci-sample.cjs
+
- name: Verify localized static pages
- if: steps.detect.outputs.skip_build != 'true'
+ if: steps.detect.outputs.validation_mode == 'full'
env:
NODE_OPTIONS: --max-old-space-size=8192
run: |
- # A locale subset cannot prove cross-locale reciprocity.
- VERIFY_PARTIAL=()
- if [ "${{ steps.detect.outputs.partial_build }}" = "true" ];
then VERIFY_PARTIAL=(--partial); fi
- for LOCALE in $(echo "${{
steps.detect.outputs.enabled_locales }}" | tr ',' ' '); do
- node scripts/i18n/verify-static.cjs --site build
--manifest ".i18n-build/manifests/${LOCALE}.json" "${VERIFY_PARTIAL[@]}"
+ for LOCALE in $(echo "${{
steps.detect.outputs.validation_enabled_locales }}" | tr ',' ' '); do
+ node scripts/i18n/verify-static.cjs --site build
--manifest ".i18n-build/manifests/${LOCALE}.json"
done
- if [ "${{ steps.detect.outputs.optimize_publish }}" = "true"
]; then
+ if [ "${{ steps.detect.outputs.validation_mode }}" = "full"
]; then
node scripts/publish/verify.cjs build
- for LOCALE in $(echo "${{
steps.detect.outputs.enabled_locales }}" | tr ',' ' '); do
+ for LOCALE in $(echo "${{
steps.detect.outputs.validation_enabled_locales }}" | tr ',' ' '); do
node scripts/i18n/verify-doc-navigation.cjs --site
build --locale "$LOCALE" --manifest ".i18n-build/manifests/${LOCALE}.json"
--output ".i18n-build/browser-navigation-${LOCALE}.json"
EXTRA_ARGS=()
if [ "$LOCALE" = "ko" ] || [ "$LOCALE" = "ru" ];
then EXTRA_ARGS=(--source-locales "en,zh-CN" --themes); fi
@@ -226,12 +220,16 @@ jobs:
rm -rf build
- name: Upload publish artifact measurements
- if: always() && steps.detect.outputs.optimize_publish == 'true'
+ if: always() && hashFiles('.i18n-build/ci-plan.json') != ''
uses:
actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
with:
name: publish-artifact-measurements
include-hidden-files: true
path: |
+ .i18n-build/ci-plan.json
+ .i18n-build/ci-sample/scopes/*.json
+ .i18n-build/ci-sample/verification.json
+ .i18n-build/ci-sampling-fixture-proof.json
.i18n-build/publish-result.json
.i18n-build/build-result.json
.i18n-build/content-fixture-verification.json
@@ -242,22 +240,8 @@ jobs:
if-no-files-found: error
retention-days: 14
- - name: Docs changed link gate
- continue-on-error: true
- run: |
- # Report-only: the bulk-rename in the new-homepage launch PR
- # exposes ~30 latent anchor/link issues that pre-date this
- # change. They're being chipped away in follow-up PRs and
- # should not block the launch. The full findings still write
- # to the report file below for audit.
- yarn docs:links:changed --output
website-quality-governance/generated/docs-links-report.json
-
- name: Docs governance report
- if: always()
+ if: always() && steps.install.outcome == 'success' &&
steps.detect.outputs.validation_mode != 'fixtures'
continue-on-error: true
run: |
- if [ ! -d node_modules ]; then
- npm install -g yarn
- yarn --frozen-lockfile || yarn
- fi
yarn docs:lint:changed --output
website-quality-governance/generated/docs-governance-report.json
diff --git a/scripts/build-locale.cjs b/scripts/build-locale.cjs
index 2fa37bf65c33..d83de9a42d11 100644
--- a/scripts/build-locale.cjs
+++ b/scripts/build-locale.cjs
@@ -6,7 +6,7 @@ const path = require('node:path');
const {createRequire} = require('node:module');
async function main() {
- const [mode, siteDir, outDir, value] = process.argv.slice(2);
+ const [mode, siteDir, outDir, value, samplePlan] = process.argv.slice(2);
const fromSite = createRequire(path.join(siteDir, 'package.json'));
const version = fromSite('@docusaurus/core/package.json').version;
if (version !== '3.6.3') throw new Error(`Revalidate the locale build
adapter for Docusaurus ${version}`);
@@ -24,9 +24,10 @@ async function main() {
directory:locale === i18n.defaultLocale ? '.' : locale,
baseUrl:localizePath({path:siteConfig.baseUrl, pathType:'url',
i18n:{...i18n,currentLocale:locale}}),
}))}, null, 2) + '\n');
- } else if (mode === 'build') {
+ } else if (mode === 'build' || mode === 'sample') {
+ const restore = mode === 'sample' ? await
require('./i18n/ci-sample.cjs').install({root:siteDir,outDir,locale:value,planFile:samplePlan})
: () => {};
const {buildLocale} =
fromSite('@docusaurus/core/lib/commands/build/buildLocale.js');
- await buildLocale({siteDir, locale:value, cliOptions:{outDir}});
+ try {await buildLocale({siteDir, locale:value, cliOptions:{outDir}});}
finally {restore();}
} else throw new Error(`Unknown locale build mode: ${mode}`);
}
main().catch(error => {console.error(error); process.exitCode = 1;});
diff --git a/scripts/build-locales.cjs b/scripts/build-locales.cjs
index 34f2475747f0..5fb336e424a7 100644
--- a/scripts/build-locales.cjs
+++ b/scripts/build-locales.cjs
@@ -24,9 +24,13 @@ function selectLocales(plan, requested) {
return plan.locales.filter(item => wanted.has(item.locale));
}
-async function buildLocales({siteDir = process.cwd(), outDir = 'build',
locales: requestedLocales, run, stamp = sourceStamp}) {
+async function buildLocales({siteDir = process.cwd(), outDir = 'build',
locales: requestedLocales, samplePlan, run, stamp = sourceStamp}) {
siteDir = path.resolve(siteDir);
outDir = path.resolve(siteDir, outDir);
+ if (samplePlan) {
+ if
(!outDir.startsWith(path.join(siteDir,'.i18n-build','ci-sample')+path.sep))
throw new Error('Sample output must stay under .i18n-build/ci-sample');
+ if (require('./publish/shared-assets.cjs').enabled()) throw new
Error('Sample validation cannot produce optimized publication artifacts');
+ }
if (outDir === siteDir || siteDir.startsWith(outDir + path.sep)) throw new
Error('Build output must not contain the source checkout');
fs.mkdirSync(path.dirname(outDir), {recursive:true});
const stage = fs.mkdtempSync(path.join(path.dirname(outDir),
'.doris-build-'));
@@ -36,12 +40,13 @@ async function buildLocales({siteDir = process.cwd(),
outDir = 'build', locales:
};
const report = {schemaVersion:1, source:initialStamp,
mode:'separate-processes',
requested:requestedLocales ?? null, locales:[]};
+ if(samplePlan)report.validation={mode:'sample',publicationAllowed:false};
const reportDir = path.join(siteDir, '.i18n-build');
fs.mkdirSync(reportDir, {recursive:true});
const reportPath = path.join(reportDir, 'build-result.json');
const execute = run ?? ((args) => new Promise((resolve, reject) => {
const child =
spawn(process.execPath,[path.join(__dirname,'build-locale.cjs'),
...args],{cwd:siteDir,stdio:'inherit'});
- console.log(JSON.stringify({event:'locale_process_start',
mode:args[0], locale:args[0] === 'build' ? args[3] : null, pid:child.pid}));
+ console.log(JSON.stringify({event:'locale_process_start',
mode:args[0], locale:args[0] === 'plan' ? null : args[3], pid:child.pid}));
const stop = signal => child.kill(signal);
const term = () => stop('SIGTERM');
const interrupt = () => stop('SIGINT');
@@ -51,7 +56,7 @@ async function buildLocales({siteDir = process.cwd(), outDir
= 'build', locales:
child.once('exit', (code, signal) => {
process.removeListener('SIGTERM', term);
process.removeListener('SIGINT', interrupt);
- console.log(JSON.stringify({event:'locale_process_end',
mode:args[0], locale:args[0] === 'build' ? args[3] : null, pid:child.pid, code,
signal}));
+ console.log(JSON.stringify({event:'locale_process_end',
mode:args[0], locale:args[0] === 'plan' ? null : args[3], pid:child.pid, code,
signal}));
if (code !== 0) reject(new Error(`Locale ${args[3]} failed
(${signal ?? code}); output was not published`));
else resolve({pid:child.pid});
});
@@ -68,7 +73,7 @@ async function buildLocales({siteDir = process.cwd(), outDir
= 'build', locales:
verifyStamp();
const localeOutput = path.join(stage, item.locale, 'site');
const started = Date.now();
- const processInfo = await
execute(['build',siteDir,localeOutput,item.locale]);
+ const processInfo = await
execute([samplePlan?'sample':'build',siteDir,localeOutput,item.locale,...(samplePlan?[path.resolve(samplePlan)]:[])]);
const resultDir = path.join(localeOutput,item.directory);
if (!fs.existsSync(path.join(resultDir,'index.html'))) throw new
Error(`Missing locale homepage: ${item.locale}`);
report.locales.push({...item, ...processInfo,
elapsedMs:Date.now()-started, resultDir});
@@ -87,6 +92,7 @@ async function buildLocales({siteDir = process.cwd(), outDir
= 'build', locales:
fs.rmSync(destination,{recursive:true,force:true});
fs.renameSync(item.resultDir,destination);
}
+
if(samplePlan)fs.writeFileSync(path.join(assembled,'.ci-validation.json'),JSON.stringify({schemaVersion:1,mode:'sample',publicationAllowed:false,source:initialStamp})+'\n');
if (require('./publish/shared-assets.cjs').enabled()) {
report.optimization =
require('./publish/optimize.cjs').optimize({site:assembled,
locales:report.locales});
report.optimization.verification =
require('./publish/verify.cjs').verify(assembled);
diff --git a/scripts/i18n/CI_VALIDATION.md b/scripts/i18n/CI_VALIDATION.md
new file mode 100644
index 000000000000..83f90b7c0071
--- /dev/null
+++ b/scripts/i18n/CI_VALIDATION.md
@@ -0,0 +1,90 @@
+# Pull request validation
+
+`Build Check` uses `ci-plan.cjs` to select validation from the merge-base diff.
+The required check keeps the same job name. The plan and measurements are
+uploaded even when a later check fails.
+
+| Change | Validation |
+| --- | --- |
+| README, planning or review notes only | Configuration, move/link and
workflow security checks; no dependency installation or site build |
+| CI and tooling | Unit tests, five-language native compilation fixture and
publishing/Apache/Chromium fixture |
+| Shared UI, routing, SEO, localization or sidebars | Representative real
documents in five languages and every active version, plus native content and
publishing fixtures |
+| Documentation or assets | Changed documents and import dependants, plus
representative real documents; English source changes select all five
languages, localized changes select English and the affected locales |
+| Dependencies, active versions or unclassified inputs | Full five-language
publication build and verification |
+| `ci/full` PR label or manual `validation=full` | Full five-language
publication build and verification |
+
+Manual `validation=sample` requests representative samples, but cannot
downgrade
+dependency/version changes or unknown inputs from full validation. Removing a
+`ci/full` label runs the automatic policy again. Scheduled and manual
deployment
+workflows still build complete publication artifacts; this PR check never
deploys.
+
+## What a sample proves
+
+The CI adapter first loads native Docusaurus metadata for **all documents and
+all active versions**, including IDs, front matter, routes, sidebars, source
and
+translation state. Optional locales remain enabled during metadata collection.
+It then removes unselected document modules from the generated route registry
+before Webpack compilation and static rendering. Non-document pages remain in
+the build. The production build does not load this adapter.
+
+Every document plugin/version contributes its native main document and
available
+examples of MDX/imports, SQL, math and media/relative links. Real translated
bodies
+and English fallbacks are sampled where available. Changed document bodies and
+translation-state records are mandatory. Local MDX/JavaScript imports are
followed
+transitively when an imported file changes; dynamic or unparseable imports
select
+their owners conservatively. A selection above 500 documents fails with a
request
+for `ci/full`, rather than silently dropping affected documents.
+
+Links from rendered pages use the complete native route graph, generated
redirect
+aliases and existing static files. Anchors on rendered destinations receive the
+normal Docusaurus check. Anchors on omitted destinations cannot be verified and
+are counted explicitly. Unselected document bodies are not compiled, rendered
or
+checked for body syntax/anchors. Existing Docusaurus warning/error policies
still
+apply; sample success is not full-site compilation success.
+
+Selected pages receive HTML/canonical checks, managed-translation SEO checks
and a
+Chromium hydration smoke test per language. The separate publishing fixture
tests
+resource delivery, redirects, language 404s and optimized artifacts on a small
site.
+It does not claim to validate every real page's search/navigation behavior;
that
+coverage remains in full validation and deployment.
+
+## Reports and publication boundary
+
+- `.i18n-build/ci-plan.json`: source SHA, diff hash, changed files, selected
mode,
+ languages and active versions.
+- `.i18n-build/ci-sample/scopes/<locale>.json`: complete metadata count,
selected
+ document routes with reasons, actual client/server MDX modules compiled,
+ metadata loading time and unverified anchor count.
+- `.i18n-build/ci-sample/verification.json`: explicitly reports `mode=sample`,
+ `completeDocumentCompilation=false` and `publicationAllowed=false`.
+- `.i18n-build/ci-sampling-fixture-proof.json`: proves an omitted broken import
+ does not compile, while editing the same document makes compilation fail.
+
+Samples write only under `.i18n-build/ci-sample/`, never `build/`. Their root
and
+locale outputs carry `.ci-validation.json`; publication optimization and
artifact
+verification reject that marker. The workflow uploads reports, not the sampled
+site. Locale builds also reject a changed source snapshot, preventing
mixed-head
+artifacts. The adapter is pinned to Docusaurus 3.6.3 and must be revalidated
on an
+upgrade.
+
+The translation admission gate skips native inventory collection when no target
+body or state changed for that language. Its output says
`inventorySkipped=true`
+and `freshness=null`; protection exceptions are still validated. Edited targets
+and state continue through the original native registration and technical
checks.
+
+## Local commands
+
+Commit the inputs to validate before planning, so the merge-base diff includes
+new files and the evidence refers to a reproducible source snapshot.
+
+```bash
+yarn key-features:generate
+node scripts/i18n/ci-plan.cjs <merge-base> --sample
+DORIS_ENABLE_LOCALES=ja,ko,ru DORIS_OPTIMIZE_PUBLISH=false \
+ node scripts/i18n/build-ci-sample.cjs
+node scripts/i18n/verify-ci-sample.cjs
+```
+
+The verifier needs Playwright/Chromium, installed like the workflow; use
+`--no-browser` only for a local static check. The fixture can be run
independently
+with `node scripts/i18n/verify-ci-sampling-fixture.cjs --browser`.
diff --git a/scripts/i18n/__tests__/admission-scope.test.cjs
b/scripts/i18n/__tests__/admission-scope.test.cjs
new file mode 100644
index 000000000000..2346ca8ad7ac
--- /dev/null
+++ b/scripts/i18n/__tests__/admission-scope.test.cjs
@@ -0,0 +1,49 @@
+const test = require('node:test');
+const assert = require('node:assert/strict');
+const fs = require('node:fs');
+const path = require('node:path');
+const os = require('node:os');
+const {execFileSync} = require('node:child_process');
+const {checkChangedTranslations, needsNativeInventory} =
require('../check-translations.cjs');
+function fixture(t) {
+ const root = fs.mkdtempSync(path.join(os.tmpdir(), 'admission-scope-'));
+ t.after(() => fs.rmSync(root, {recursive: true, force: true}));
+ const write = (file, value) => {fs.mkdirSync(path.dirname(path.join(root,
file)), {recursive: true}); fs.writeFileSync(path.join(root, file), value);};
+ write('translations/config.json', JSON.stringify({schemaVersion: 1,
locales: {ja: {sourceLocale: 'en', sourceOverrides: [], languageName:
'Japanese', scriptPattern: '[日]'}}}));
+ const git = (...args) => execFileSync('git', args, {cwd: root, encoding:
'utf8', stdio: 'pipe'}).trim();
+ const commit = () => {git('add', '.'); git('-c', 'user.name=Test', '-c',
'[email protected]', 'commit', '-qm', 'change');};
+ git('init', '-q'); commit(); const base = git('rev-parse', 'HEAD');
+ return {root, write, base, commit};
+}
+
+test('shared renderer and English source edits skip empty native admission
without claiming freshness', async t => {
+ const {root, write, base, commit} = fixture(t);
+ write('src/utils/translation.ts', '// Shared implementation\n');
write('docs/topic.md', '# Updated English\n'); commit();
+ const result = await checkChangedTranslations({root, base, locale: 'ja',
collectPages: () => {throw Error('Unnecessary full inventory');}});
+ assert.equal(result.checked, 0);
+ assert.deepEqual(result.errors, []);
+ assert.equal(result.inventorySkipped, true);
+ assert.equal(result.freshness, null);
+ assert.match(result.inventorySkipReason, /freshness was not scanned/);
+});
+
+test('edited translations still require native registration and cannot pass as
an empty check', async t => {
+ const {root, write, base, commit} = fixture(t);
+ write('i18n/ja/docusaurus-plugin-content-docs/current/orphan.md', '#
Unregistered target\n'); commit();
+ let calls = 0;
+ const result = await checkChangedTranslations({root, base, locale: 'ja',
collectPages: async () => {calls++; return {pages: [], reports: []};}});
+ assert.equal(calls, 1);
+ assert.equal(result.inventorySkipped, false);
+ assert(result.errors.some(error => /no registered
source/.test(error.message)));
+ assert(needsNativeInventory(['translations/state/ja/docs.json'], 'ja'));
+
assert(needsNativeInventory(['translations/state/ja/pages/default/current/topic.json'],
'ja'));
+
assert(needsNativeInventory(['i18n/ja/docusaurus-plugin-content-docs-course-ui/current/page.tsx'],
'ja'));
+
assert(!needsNativeInventory(['i18n/ko/docusaurus-plugin-content-docs/current/topic.md'],
'ja'));
+});
+
+test('the empty-admission path still rejects invalid protection exceptions and
unknown locales', async t => {
+ const {root, write, base, commit} = fixture(t);
+ write('translations/protection-exceptions/ja.json',
JSON.stringify({schemaVersion: 1, pages: [{reason: 'invalid baseline'}]}));
commit();
+ await assert.rejects(checkChangedTranslations({root, base, locale: 'ja'}),
/baseline commit/);
+ await assert.rejects(checkChangedTranslations({root, base, locale:
'unconfigured'}), /Unconfigured/);
+});
diff --git a/scripts/i18n/__tests__/ci-sampling.test.cjs
b/scripts/i18n/__tests__/ci-sampling.test.cjs
new file mode 100644
index 000000000000..7275965cd430
--- /dev/null
+++ b/scripts/i18n/__tests__/ci-sampling.test.cjs
@@ -0,0 +1,217 @@
+const test = require('node:test'),
+ assert = require('node:assert/strict'),
+ fs = require('node:fs'),
+ path = require('node:path'),
+ os = require('node:os');
+const {policy} = require('../ci-plan.cjs');
+const {prune, dependencies, stripUnbuiltAnchors, selectDocs} =
require('../ci-sample.cjs');
+const {buildLocales} = require('../../build-locales.cjs');
+function directory(t) {
+ const root = fs.mkdtempSync(path.join(os.tmpdir(), 'ci-sampling-'));
+ t.after(() => fs.rmSync(root, {recursive: true, force: true}));
+ return root;
+}
+test('PR policy separates fixtures, real samples, full upgrades and
lightweight prose', () => {
+ assert.equal(policy(['.github/workflows/build-check.yml']).mode,
'fixtures');
+ assert.equal(policy(['scripts/publish/apache.cjs']).mode, 'fixtures');
+ assert.equal(policy(['scripts/i18n/ci-sample.cjs']).mode, 'fixtures');
+ assert.equal(policy(['src/theme/DocItem/Content/index.tsx']).mode,
'sample');
+ assert.equal(policy(['docs/topic.mdx']).mode, 'sample');
+ assert.equal(
+
policy(['i18n/ja/docusaurus-plugin-content-docs/current/topic.md']).mode,
+ 'sample',
+ );
+ assert.equal(policy(['static/images/logo.svg']).mode, 'sample');
+ for (const file of ['package.json', 'yarn.lock', 'versions.json',
'unknown-input.bin'])
+ assert.equal(policy([file]).mode, 'full');
+ assert.equal(policy(['plan-doc/notes.md']).mode, 'skip');
+ assert.equal(policy(['README.md']).mode, 'skip');
+ assert.equal(policy(['docs/topic.md'], {forceFull: true}).mode, 'full');
+ assert.equal(policy([], {forceSample: true}).mode, 'sample');
+});
+test('unselected MDX modules leave the compiler graph while native layouts and
non-doc pages remain', () => {
+ const routes = [
+ {
+ path: '/docs',
+ plugin: {name: 'docusaurus-plugin-content-docs'},
+ routes: [
+ {
+ path: '/docs/4.x',
+ component: 'Layout',
+ routes: [
+ {path: '/docs/4.x/a/', component: 'Doc', modules:
{content: 'a.mdx'}},
+ {path: '/docs/4.x/b/', component: 'Doc', modules:
{content: 'b.mdx'}},
+ ],
+ },
+ ],
+ },
+ {path: '/', component: 'Home'},
+ ];
+ const output = prune(
+ routes,
+ new Set(['/docs/4.x/a', '/docs/4.x/b']),
+ new Set(['/docs/4.x/a']),
+ [],
+ );
+ assert.equal(output[0].routes[0].component, 'Layout');
+ assert.deepEqual(
+ output[0].routes[0].routes.map(r => r.modules.content),
+ ['a.mdx'],
+ );
+ assert.equal(output[1].component, 'Home');
+ assert.equal(routes[0].routes[0].routes.length, 2);
+});
+test('dependency discovery follows MDX imports and their transitive code
imports, not fenced examples', async t => {
+ const root = directory(t);
+ fs.mkdirSync(path.join(root, 'docs'));
+ fs.mkdirSync(path.join(root, 'src'));
+ fs.writeFileSync(
+ path.join(root, 'docs/a.mdx'),
+ 'import Child from "./part.mdx";\n\n<Child/>\n\n```js\nimport Fake
from "./absent.mdx"\n```\n',
+ );
+ fs.writeFileSync(
+ path.join(root, 'docs/part.mdx'),
+ 'import {value} from "@site/src/value.js";\n\n{value}\n',
+ );
+ fs.writeFileSync(path.join(root, 'src/value.js'), 'export const value =
"real";');
+ const output = await dependencies(root, path.join(root, 'docs/a.mdx'));
+ assert.deepEqual([...output.files].map(f => path.relative(root,
f)).sort(), [
+ 'docs/a.mdx',
+ 'docs/part.mdx',
+ 'src/value.js',
+ ]);
+ assert.equal(output.uncertain, false);
+});
+test('known unbuilt anchors are reported as unverified, while unknown paths
and compiled anchors remain checked', () => {
+ const result = stripUnbuiltAnchors(
+ {
+ '/a/': {
+ links: ['/b/#not-rendered', '/a/#must-exist', '/unknown/'],
+ anchors: ['must-exist'],
+ },
+ },
+ new Set(['/a', '/b']),
+ new Set(['/a']),
+ new Map(),
+ );
+ assert.equal(result.skipped, 1);
+ assert.deepEqual(result.collected['/a/'].links, ['/a/#must-exist',
'/unknown/']);
+});
+test('sample artifacts cannot become publication artifacts or overwrite the
full build output', async t => {
+ const root = directory(t);
+ fs.writeFileSync(path.join(root, '.ci-validation.json'), '{}');
+ assert.throws(() => require('../../publish/verify.cjs').verify(root),
/must not be published/);
+ assert.throws(
+ () => require('../../publish/optimize.cjs').optimize({site: root,
locales: []}),
+ /must not be published/,
+ );
+ await assert.rejects(
+ buildLocales({siteDir: root, outDir: 'build', samplePlan:
'plan.json'}),
+ /must stay under/,
+ );
+});
+
+test('changed transitive imports and sharded translation state select their
owning documents', async t => {
+ const root = directory(t);
+ fs.mkdirSync(path.join(root, 'docs'));
+ fs.mkdirSync(path.join(root, 'src'));
+ fs.mkdirSync(path.join(root, 'generated'));
+ const docs = ['entry', 'owner', 'state-only'].map(name => ({
+ id: name,
+ source: '@site/docs/' + name + '.mdx',
+ permalink: '/docs/' + name + '/',
+ }));
+ fs.writeFileSync(path.join(root, 'docs/entry.mdx'), 'Entry body');
+ fs.writeFileSync(
+ path.join(root, 'docs/owner.mdx'),
+ 'import {value} from "@site/src/value.js";\nimport Nested from
"./state-only.mdx";\n\n{value}\n<Nested/>',
+ );
+ fs.writeFileSync(path.join(root, 'docs/state-only.mdx'), 'State body');
+ fs.writeFileSync(
+ path.join(root, 'src/value.js'),
+ 'export {value} from "./nested.js"; export {default as data} from
"./data.json";',
+ );
+ fs.writeFileSync(path.join(root, 'src/nested.js'), 'export const value =
1;');
+ fs.writeFileSync(path.join(root, 'src/data.json'), '{}');
+ fs.writeFileSync(
+ path.join(root, 'generated/globalData.json'),
+ JSON.stringify({
+ 'docusaurus-plugin-content-docs': {
+ default: {versions: [{name: 'current', mainDocId: 'entry'}]},
+ },
+ }),
+ );
+ const props = {
+ siteDir: root,
+ generatedFilesDir: path.join(root, 'generated'),
+ i18n: {currentLocale: 'en', locales: ['en']},
+ plugins: [
+ {
+ name: 'docusaurus-plugin-content-docs',
+ options: {id: 'default'},
+ content: {
+ loadedVersions: [
+ {versionName: 'current', contentPath: path.join(root,
'docs'), docs},
+ ],
+ contentIdentity: {
+ entries: docs.map(doc => ({
+ siteLocale: 'en',
+ pluginId: 'default',
+ version: 'current',
+ docId: doc.id,
+ englishPath: 'docs/' + doc.id + '.mdx',
+ indexable: true,
+ fallback: false,
+ })),
+ },
+ },
+ },
+ ],
+ };
+ const imported = await selectDocs(props, {files: ['src/nested.js'],
baseSha: 'HEAD'});
+ assert(
+ imported.selected
+ .get(JSON.stringify(['default', 'current', 'owner']))
+ .reasons.includes('changed import dependency'),
+ );
+ const data = await selectDocs(props, {files: ['src/data.json'], baseSha:
'HEAD'});
+ assert(
+ data.selected
+ .get(JSON.stringify(['default', 'current', 'owner']))
+ .reasons.includes('changed import dependency'),
+ );
+ const document = await selectDocs(props, {files: ['docs/state-only.mdx'],
baseSha: 'HEAD'});
+ assert(
+ document.selected
+ .get(JSON.stringify(['default', 'current', 'owner']))
+ .reasons.includes('changed import dependency'),
+ );
+ fs.rmSync(path.join(root, 'src/nested.js'));
+ const deleted = await selectDocs(props, {files: ['src/nested.js'],
baseSha: 'HEAD'});
+ assert(
+ deleted.selected
+ .get(JSON.stringify(['default', 'current', 'owner']))
+ .reasons.includes('conservative dynamic/unparsed import'),
+ );
+ fs.writeFileSync(path.join(root, 'src/nested.js'), 'export const value =
1;');
+ const state = 'translations/state/ja/pages/topic.json';
+ fs.mkdirSync(path.dirname(path.join(root, state)), {recursive: true});
+ fs.writeFileSync(
+ path.join(root, state),
+ JSON.stringify({
+ schemaVersion: 1,
+ page: {pluginId: 'default', version: 'current', docId:
'state-only'},
+ }),
+ );
+ const changed = await selectDocs(props, {files: [state], baseSha: 'HEAD'});
+ assert(
+ changed.selected
+ .get(JSON.stringify(['default', 'current', 'state-only']))
+ .reasons.includes('changed document/state'),
+ );
+ assert(
+ changed.selected
+ .get(JSON.stringify(['default', 'current', 'owner']))
+ .reasons.includes('changed import dependency'),
+ );
+});
diff --git a/scripts/i18n/__tests__/multi-locale.test.cjs
b/scripts/i18n/__tests__/multi-locale.test.cjs
index 1c0fac66e63e..6910b26b0759 100644
--- a/scripts/i18n/__tests__/multi-locale.test.cjs
+++ b/scripts/i18n/__tests__/multi-locale.test.cjs
@@ -126,7 +126,7 @@ test('scheduled Japanese jobs select ja and PR admission
checks only changed loc
const run = fail => {
write('calls', '');
const result = spawnSync('bash', ['-e', '-c', gate.run.replace('${{
github.base_ref }}', 'master')],
- {cwd: root, encoding: 'utf8', env: {...process.env, FAIL_LOCALE:
fail, DORIS_ENABLE_JA: '', DORIS_ENABLE_LOCALES: ''}});
+ {cwd: root, encoding: 'utf8', env: {...process.env, BASE_REF:
'master', FAIL_LOCALE: fail, DORIS_ENABLE_JA: '', DORIS_ENABLE_LOCALES: ''}});
return {status: result.status, calls: fs.readFileSync(path.join(root,
'calls'), 'utf8'), stderr: result.stderr};
};
const jaTarget =
'i18n/ja/docusaurus-plugin-content-docs/current/01-shared.md';
diff --git a/scripts/i18n/build-ci-sample.cjs b/scripts/i18n/build-ci-sample.cjs
new file mode 100644
index 000000000000..529d380b1b6e
--- /dev/null
+++ b/scripts/i18n/build-ci-sample.cjs
@@ -0,0 +1,20 @@
+const fs = require('node:fs');
+const path = require('node:path');
+const assert = require('node:assert/strict');
+const {buildLocales} = require('../build-locales.cjs');
+async function main() {
+ const planFile = path.resolve(process.argv[2] ??
'.i18n-build/ci-plan.json');
+ const plan = JSON.parse(fs.readFileSync(planFile));
+ assert.equal(plan.mode, 'sample');
+ const report = await buildLocales({
+ outDir: '.i18n-build/ci-sample/site',
+ locales: plan.locales,
+ samplePlan: planFile,
+ });
+ console.log(JSON.stringify(report));
+}
+if (require.main === module)
+ main().catch(e => {
+ console.error(e);
+ process.exitCode = 1;
+ });
diff --git a/scripts/i18n/check-translations.cjs
b/scripts/i18n/check-translations.cjs
index 3a1290357193..72524afc0b45 100644
--- a/scripts/i18n/check-translations.cjs
+++ b/scripts/i18n/check-translations.cjs
@@ -2,7 +2,7 @@
const fs = require('node:fs');
const path = require('node:path');
const {execFileSync} = require('node:child_process');
-const {identity, hash, loadRecords, loadRecordsAt, pipelineLocale} =
require('./state.cjs');
+const {identity, hash, loadRecords, loadRecordsAt, pipelineLocale, loadConfig}
= require('./state.cjs');
const {structure, alignHeadings} = require('./structure.cjs');
const {collect} = require('./inventory.cjs');
const {protectionSignature} = require('./protection.cjs');
@@ -86,18 +86,38 @@ async function checkTranslations({root, locale =
pipelineLocale(root), pages, re
return {checked: selected.length, errors, warnings, freshness:
Object.fromEntries([...new Set(pages.map(page => page.status))]
.map(status => [status, pages.filter(page => page.status ===
status).length])), reports};
}
+function needsNativeInventory(files, locale) {
+ const prefix = `i18n/${locale}/`;
+ return files.some(file => (file.startsWith(prefix)
+ &&
/^docusaurus-plugin-content-docs[^/]*\/.+\.(mdx?|tsx?)$/.test(file.slice(prefix.length)))
+ || file === `translations/state/${locale}/docs.json` ||
file.startsWith(`translations/state/${locale}/pages/`));
+}
+
+async function checkChangedTranslations({root, base, locale =
pipelineLocale(root), collectPages = collect}) {
+ if (!Object.hasOwn(loadConfig(root).locales, locale)) throw new
Error(`Unconfigured translation locale: ${locale}`);
+ const files =
execFileSync('git',['diff','--name-only','-z',base,'HEAD'],{cwd:root,encoding:'utf8'}).split('\0').filter(Boolean);
+ const inventoryRequired = needsNativeInventory(files, locale);
+ const stateChanged = files.some(file => file ===
`translations/state/${locale}/docs.json`
+ || file.startsWith(`translations/state/${locale}/pages/`));
+ const records = inventoryRequired ? loadRecords(root, locale) : new Map();
+ const previousRecords = stateChanged ? loadRecordsAt(root, locale, base) :
new Map();
+ const native = inventoryRequired ? await collectPages(root) : {pages: [],
reports: []};
+ // Admission blocks only edited targets/state. Keep validating pinned
+ // protection exceptions even when there are no such edits. Full freshness
+ // belongs to build manifests/coverage audits, not an empty admission
result.
+ const result = await checkTranslations({root, locale, ...native, files,
records, previousRecords});
+ return {...result, inventorySkipped: !inventoryRequired,
+ freshness: inventoryRequired ? result.freshness : null,
+ ...(!inventoryRequired && {inventorySkipReason: 'No edited translation
body or state for this locale; freshness was not scanned'})};
+}
+
async function main() {
const root = path.resolve(__dirname,'../..');
const args = process.argv.slice(2), base = args[args.indexOf('--base')+1];
- if (!args.includes('--base')) throw new Error('Usage:
check-translations.cjs --base <PR merge base>');
- const files =
execFileSync('git',['diff','--name-only',base,'HEAD'],{cwd:root,encoding:'utf8'}).trim().split('\n').filter(Boolean);
- const locale = pipelineLocale(root);
- const records = loadRecords(root, locale);
- const previousRecords = loadRecordsAt(root, locale, base);
- const native = await collect(root);
- const result = await checkTranslations({root, locale, ...native, files,
records, previousRecords});
+ if (!args.includes('--base') || !base) throw new Error('Usage:
check-translations.cjs --base <PR merge base>');
+ const result = await checkChangedTranslations({root, base});
console.log(JSON.stringify(result,null,2));
if (result.errors.length) process.exitCode = 1;
}
if (require.main === module) main().catch(error =>
{console.error(error);process.exitCode=1;});
-module.exports = {checkTranslations};
+module.exports = {checkTranslations, checkChangedTranslations,
needsNativeInventory};
diff --git a/scripts/i18n/ci-plan.cjs b/scripts/i18n/ci-plan.cjs
new file mode 100644
index 000000000000..e62a49707e1d
--- /dev/null
+++ b/scripts/i18n/ci-plan.cjs
@@ -0,0 +1,166 @@
+// CI policy is separate from the production build scope. Sample artifacts
never
+// claim full compilation coverage and are not valid publication inputs.
+const fs = require('node:fs');
+const path = require('node:path');
+const {execFileSync} = require('node:child_process');
+const {createHash} = require('node:crypto');
+const {locales, localeRegistry} = require('../../config/i18n');
+const {buildScope} = require('./build-scope.cjs');
+const {managedLocales} = require('./state.cjs');
+const runtime =
+
/^(src\/|config\/|docusaurus\.config\.js$|sidebars[^/]*\.(ts|json)$|versioned_sidebars\/|scripts\/i18n\/(docs-plugin|status-plugin|state|structure|content-plugin|content-loader|source-groups|code-prose|protection|content-manifest|manual-state|manual-plugins|native-context|route-context|page-body|page-body-loader|sitemap)\.cjs$)/;
+const content =
+
/^(static\/|docs\/|versioned_docs\/|community\/|course\/|releasenotes\/|blog\/|i18n\/|translations\/(pages|blog|state)\/)/;
+const highRisk =
/^(package\.json|yarn\.lock|versions\.json|config\/versions-plugin\.js)$/;
+function policy(files, {forceFull = false, forceSample = false} = {}) {
+ const risk = files.filter(file => highRisk.test(file));
+ const changed = files.some(file => content.test(file));
+ const shared = files.some(file => runtime.test(file));
+ const ci = files.some(file => /^(scripts\/|\.github\/)/.test(file));
+ const unknown = files.filter(
+ file =>
+ !runtime.test(file) &&
+ !content.test(file) &&
+
!/^((scripts|\.github|plan-doc|review-docs|\.agents)\/|README\.md$|AGENTS\.md$|LICENSE$|NOTICE$)/.test(
+ file,
+ ),
+ );
+ const mode =
+ forceFull || risk.length || unknown.length
+ ? 'full'
+ : shared || changed || forceSample
+ ? 'sample'
+ : ci
+ ? 'fixtures'
+ : 'skip';
+ return {
+ mode,
+ selection: shared || forceSample ? 'representative' : 'changed',
+ reasons: forceFull
+ ? ['explicit full validation']
+ : risk.length
+ ? risk
+ : unknown.length
+ ? unknown
+ : [mode],
+ runContentFixture:
+ shared ||
+ files.some(file =>
+
/verify-content-fixture|__tests__\/fixtures\/content-manifest/.test(file),
+ ),
+ runSamplingFixture: mode === 'fixtures',
+ runPublishFixture: mode === 'fixtures' || shared || mode === 'full',
+ publicationAllowed: mode === 'full',
+ };
+}
+function plan({root, base, event = {}, forceFull = false, forceSample =
false}) {
+ const files = execFileSync('git', ['diff', '--name-only', '-z', base,
'HEAD'], {
+ cwd: root,
+ encoding: 'utf8',
+ maxBuffer: 16 * 1024 * 1024,
+ })
+ .split('\0')
+ .filter(Boolean);
+ const labels = event.pull_request?.labels ?? [];
+ const selected = policy(files, {
+ forceFull:
+ forceFull ||
+ event.inputs?.validation === 'full' ||
+ labels.some(label => label.name === 'ci/full'),
+ forceSample: forceSample || event.inputs?.validation === 'sample',
+ });
+ const impact = buildScope(
+ files,
+ JSON.parse(fs.readFileSync(path.join(root, 'versions.json'))),
+ managedLocales(root),
+ );
+ let requested = impact.locales;
+ if (selected.mode === 'full' || selected.selection === 'representative')
requested = locales;
+ else if (selected.mode === 'sample') {
+ // English source changes may alter any target's fallback or source
+ // freshness; use every locale until an exact dependency proof exists.
+ requested = files.some(file =>
+
/^(docs|versioned_docs|community|course|releasenotes|blog)\//.test(file),
+ )
+ ? locales
+ : [...new Set(['en', ...requested])];
+ }
+ if (selected.mode === 'sample' && !requested.length) requested = locales;
+ const sourceSha = execFileSync('git', ['rev-parse', 'HEAD'], {
+ cwd: root,
+ encoding: 'utf8',
+ }).trim();
+ const trackedDiffHash = createHash('sha256')
+ .update(
+ execFileSync('git', ['diff', 'HEAD', '--binary'], {
+ cwd: root,
+ maxBuffer: 16 * 1024 * 1024,
+ }),
+ )
+ .digest('hex');
+ return {
+ schemaVersion: 1,
+ sourceSha,
+ trackedDiffHash,
+ baseSha: base,
+ files,
+ ...selected,
+ impact,
+ locales: locales.filter(locale => requested.includes(locale)),
+ enabledLocales: localeRegistry.filter(locale =>
locale.optional).map(locale => locale.code),
+ activeVersions: [
+ 'current',
+ ...JSON.parse(fs.readFileSync(path.join(root, 'versions.json'))),
+ ],
+ };
+}
+if (require.main === module) {
+ try {
+ const root = process.cwd(),
+ args = process.argv.slice(2),
+ base = args[0];
+ if (!base) throw new Error('Usage: ci-plan.cjs <base SHA>
[--full|--sample]');
+ const event = process.env.GITHUB_EVENT_PATH
+ ? JSON.parse(fs.readFileSync(process.env.GITHUB_EVENT_PATH))
+ : {};
+ const result = plan({
+ root,
+ base,
+ event,
+ forceFull: args.includes('--full'),
+ forceSample: args.includes('--sample'),
+ });
+ fs.mkdirSync(path.join(root, '.i18n-build'), {recursive: true});
+ fs.writeFileSync(
+ path.join(root, '.i18n-build/ci-plan.json'),
+ JSON.stringify(result, null, 2) + '\n',
+ );
+ console.log(JSON.stringify(result, null, 2));
+ if (process.env.GITHUB_STEP_SUMMARY)
+ fs.appendFileSync(
+ process.env.GITHUB_STEP_SUMMARY,
+ `### Validation scope: ${result.mode}\n\nSource:
\`${result.sourceSha}\`. Languages: ${result.locales.join(', ') ||
'none'}.\n\n` +
+ (result.mode === 'full'
+ ? 'Full document compilation and publication artifact
verification requested.\n\n'
+ : 'This check does **not** claim full document
compilation or produce a publication artifact.\n\n'),
+ );
+ if (process.env.GITHUB_OUTPUT)
+ fs.appendFileSync(
+ process.env.GITHUB_OUTPUT,
+ Object.entries({
+ validation_mode: result.mode,
+ validation_locales: result.locales.join(','),
+ validation_enabled_locales:
result.enabledLocales.join(','),
+ content_fixture: result.runContentFixture,
+ publish_fixture: result.runPublishFixture,
+ sampling_fixture: result.runSamplingFixture,
+ })
+ .map(([k, v]) => `${k}=${v}\n`)
+ .join(''),
+ );
+ } catch (error) {
+ console.error(error);
+ process.exitCode = 1;
+ }
+}
+module.exports = {policy, plan};
diff --git a/scripts/i18n/ci-sample.cjs b/scripts/i18n/ci-sample.cjs
new file mode 100644
index 000000000000..b2d8e8caf8bf
--- /dev/null
+++ b/scripts/i18n/ci-sample.cjs
@@ -0,0 +1,655 @@
+// Disposable-process adapter for Docusaurus 3.6.3. Production builds never
load
+// this module. Native metadata stays complete; only selected routes enter the
+// bundler and SSG. Unrendered public routes remain valid *logical* link
targets.
+const fs = require('node:fs');
+const path = require('node:path');
+const assert = require('node:assert/strict');
+const {execFileSync} = require('node:child_process');
+const key = value => decodeURI(value).replace(/\/+$/, '') || '/';
+const id = row => JSON.stringify([row.pluginId, row.version, row.docId]);
+const sourceFile = (root, file) => path.resolve(root, file.replace(/^@site\//,
''));
+const gitPath = (root, file) =>
+ path
+ .relative(root, file)
+ .split(path.sep)
+ .join('/')
+ .replace(/^\.i18n-build\/([^/]+)\//, 'i18n/$1/');
+const sha = root =>
+ execFileSync('git', ['rev-parse', 'HEAD'], {cwd: root, encoding:
'utf8'}).trim();
+function flatten(routes) {
+ return routes.flatMap(route =>
+ route.routes ? flatten(route.routes) : route.path === '*' ? [] :
[route],
+ );
+}
+function rowsFor(props) {
+ const rows = [];
+ const global = JSON.parse(
+ fs.readFileSync(path.join(props.generatedFilesDir, 'globalData.json'),
'utf8'),
+ );
+ for (const plugin of props.plugins.filter(p => p.name ===
'docusaurus-plugin-content-docs')) {
+ const entries = plugin.content.contentIdentity?.entries ?? [];
+ const lookup = new Map(
+ entries.map(e => [JSON.stringify([e.siteLocale, e.version,
e.docId]), e]),
+ );
+ const grouped = new Map();
+ for (const entry of entries) {
+ const k = JSON.stringify([entry.version, entry.docId]);
+ if (!grouped.has(k)) grouped.set(k, []);
+ grouped.get(k).push(entry);
+ }
+ for (const version of plugin.content.loadedVersions)
+ for (const doc of version.docs) {
+ const entry = lookup.get(
+ JSON.stringify([props.i18n.currentLocale,
version.versionName, doc.id]),
+ );
+ const english = lookup.get(JSON.stringify(['en',
version.versionName, doc.id]));
+ const englishPath =
+ english?.englishPath ??
+ entry?.englishPath ??
+ gitPath(props.siteDir, sourceFile(props.siteDir,
doc.source));
+ const filename = path.relative(
+ version.contentPath,
+ path.resolve(props.siteDir, englishPath),
+ );
+ const aliases = new Set([
+ englishPath,
+ gitPath(props.siteDir, sourceFile(props.siteDir,
doc.source)),
+ ]);
+ for (const item of
grouped.get(JSON.stringify([version.versionName, doc.id])) ?? [])
+ for (const file of [item.englishPath, item.sourcePath,
item.targetPath].filter(
+ Boolean,
+ ))
+ aliases.add(file);
+ if (!filename.startsWith('..'))
+ for (const locale of props.i18n.locales)
+ aliases.add(
+
`i18n/${locale}/docusaurus-plugin-content-docs${plugin.options.id === 'default'
? '' : '-' + plugin.options.id}/${version.versionName === 'current' ? 'current'
: 'version-' + version.versionName}/${filename.split(path.sep).join('/')}`,
+ );
+ rows.push({
+ pluginId: plugin.options.id,
+ version: version.versionName,
+ docId: doc.id,
+ route: doc.permalink,
+ englishPath,
+ source: doc.source,
+ aliases: [...aliases],
+ main:
+ doc.id ===
+ global['docusaurus-plugin-content-docs']?.[
+ plugin.options.id
+ ]?.versions.find(v => v.name ===
version.versionName)?.mainDocId,
+ decisions: (
+ grouped.get(JSON.stringify([version.versionName,
doc.id])) ?? []
+ ).map(e => ({
+ locale: e.siteLocale,
+ fallback: e.fallback,
+ indexable: e.indexable,
+ state: e.state,
+ })),
+ });
+ }
+ }
+ return rows.sort((a, b) => id(a).localeCompare(id(b)));
+}
+function chooseRepresentatives(rows, root) {
+ const selected = new Map();
+ const add = (row, reason) => {
+ if (!row) return;
+ const k = id(row);
+ if (!selected.has(k)) selected.set(k, {row, reasons: []});
+ selected.get(k).reasons.push(reason);
+ };
+ const groups = new Map();
+ for (const row of rows) {
+ const g = JSON.stringify([row.pluginId, row.version]);
+ if (!groups.has(g)) groups.set(g, []);
+ groups.get(g).push(row);
+ }
+ for (const group of groups.values()) {
+ add(group.find(r => r.main) ?? group[0], 'version/plugin entry');
+ const texts = new Map();
+ const text = r => {
+ if (!texts.has(id(r)))
+ texts.set(id(r), fs.readFileSync(path.resolve(root,
r.englishPath), 'utf8'));
+ return texts.get(id(r));
+ };
+ for (const [feature, pattern] of [
+ ['MDX/import', /\bimport\s|<[A-Z][A-Za-z]+/],
+ ['SQL', /```sql\b/i],
+ ['math', /\$\$|\\\[/],
+ ['media/link', /!\[|\]\(\.\.?\//],
+ ])
+ add(
+ group.find(r => pattern.test(text(r))),
+ feature,
+ );
+ const languages = new Set(group.flatMap(r => r.decisions.map(d =>
d.locale)));
+ for (const locale of languages) {
+ add(
+ group.find(r =>
+ r.decisions.some(d => d.locale === locale && !d.fallback
&& d.indexable),
+ ),
+ `${locale} real body`,
+ );
+ add(
+ group.find(r => r.decisions.some(d => d.locale === locale &&
d.fallback)),
+ `${locale} fallback`,
+ );
+ }
+ }
+ return {selected, add};
+}
+async function dependencies(root, file, cache = new Map(), seen = new Set()) {
+ if (seen.has(file)) return {files: seen, uncertain: false};
+ seen.add(file);
+ if (!/\.(mdx?|[cm]?[jt]sx?)$/.test(file)) return {files: seen, uncertain:
false};
+ let imports,
+ uncertain = false;
+ if (cache.has(file)) ({imports, uncertain} = cache.get(file));
+ else {
+ const source = fs.readFileSync(file, 'utf8');
+ imports = [];
+ // A lexical hint only: actual imports are read from AST, so fenced
code
+ // examples cannot accidentally turn the entire site into dependencies.
+ if (/\b(import|export|require)\b/.test(source)) {
+ const ts = require('typescript');
+ const blocks = [];
+ if (/\.mdx?$/.test(file)) {
+ try {
+ const {createProcessor} = await import('@mdx-js/mdx');
+ const matter = require('gray-matter');
+ const tree = createProcessor({format:
'mdx'}).parse(matter(source).content);
+ const walk = n => {
+ if (n.type === 'mdxjsEsm') blocks.push(n.value);
+ for (const child of n.children ?? []) walk(child);
+ };
+ walk(tree);
+ } catch {
+ uncertain = true;
+ }
+ } else blocks.push(source);
+ for (const block of blocks) {
+ const ast = ts.createSourceFile(
+ file,
+ block,
+ ts.ScriptTarget.Latest,
+ true,
+ ts.ScriptKind.TSX,
+ );
+ if (ast.parseDiagnostics.length) uncertain = true;
+ const walk = node => {
+ if (
+ (ts.isImportDeclaration(node) ||
ts.isExportDeclaration(node)) &&
+ node.moduleSpecifier &&
+ ts.isStringLiteral(node.moduleSpecifier)
+ )
+ imports.push(node.moduleSpecifier.text);
+ if (
+ ts.isCallExpression(node) &&
+ (node.expression.kind === ts.SyntaxKind.ImportKeyword
||
+ node.expression.getText(ast) === 'require')
+ ) {
+ if (node.arguments[0] &&
ts.isStringLiteral(node.arguments[0]))
+ imports.push(node.arguments[0].text);
+ else uncertain = true;
+ }
+ ts.forEachChild(node, walk);
+ };
+ walk(ast);
+ }
+ }
+ cache.set(file, {imports, uncertain});
+ }
+ for (const name of imports) {
+ const base = name.startsWith('@site/')
+ ? path.join(root, name.slice(6))
+ : name.startsWith('.')
+ ? path.resolve(path.dirname(file), name)
+ : undefined;
+ if (!base) continue;
+ const child = [
+ '',
+ '.js',
+ '.jsx',
+ '.mjs',
+ '.cjs',
+ '.ts',
+ '.tsx',
+ '.mts',
+ '.cts',
+ '.mdx',
+ '.md',
+ '.json',
+ '/index.js',
+ '/index.jsx',
+ '/index.ts',
+ '/index.tsx',
+ ]
+ .map(ext => base + ext)
+ .find(p => fs.existsSync(p) && fs.statSync(p).isFile());
+ if (child) {
+ const result = await dependencies(root, child, cache, seen);
+ uncertain ||= result.uncertain;
+ } else uncertain = true; // Deleted imports must still select their
owners.
+ }
+ return {files: seen, uncertain};
+}
+function stateIdentities(root, files, base) {
+ const changed = new Set();
+ for (const file of files.filter(
+ f => /^translations\/state\/[^/]+\//.test(f) && f.endsWith('.json'),
+ )) {
+ const parse = text => {
+ const data = JSON.parse(text);
+ return data.pages ?? (data.page ? [data.page] : data.docId ?
[data] : []);
+ };
+ let before = [];
+ try {
+ before = parse(
+ execFileSync('git', ['show', `${base}:${file}`], {
+ cwd: root,
+ encoding: 'utf8',
+ stdio: ['ignore', 'pipe', 'ignore'],
+ maxBuffer: 16 * 1024 * 1024,
+ }),
+ );
+ } catch {}
+ const after = fs.existsSync(path.join(root, file))
+ ? parse(fs.readFileSync(path.join(root, file), 'utf8'))
+ : [];
+ const a = new Map(before.map(r => [id(r), JSON.stringify(r)])),
+ b = new Map(after.map(r => [id(r), JSON.stringify(r)]));
+ for (const k of new Set([...a.keys(), ...b.keys()]))
+ if (a.get(k) !== b.get(k)) changed.add(k);
+ }
+ return changed;
+}
+async function selectDocs(props, plan) {
+ const rows = rowsFor(props),
+ {selected, add} = chooseRepresentatives(rows, props.siteDir);
+ const changed = new Set(plan.files),
+ changedStates = stateIdentities(props.siteDir, plan.files,
plan.baseSha);
+ for (const row of rows)
+ if (row.aliases.some(f => changed.has(f)) ||
changedStates.has(id(row))) {
+ add(row, 'changed document/state');
+ // A state transition can replace the body in a generated locale
+ // view even when its source file was not edited. Importers of that
+ // document must be compiled too, including importers of doc
routes.
+ changed.add(gitPath(props.siteDir, sourceFile(props.siteDir,
row.source)));
+ }
+ const dependenciesChanged = [...changed].filter(f =>
+
/^(docs\/|versioned_docs\/|community\/|course\/|releasenotes\/|i18n\/|src\/|static\/)/.test(
+ f,
+ ),
+ );
+ if (dependenciesChanged.length) {
+ const cache = new Map();
+ for (const row of rows) {
+ const result = await dependencies(
+ props.siteDir,
+ sourceFile(props.siteDir, row.source),
+ cache,
+ );
+ if (
+ result.uncertain ||
+ [...result.files].some(file =>
changed.has(gitPath(props.siteDir, file)))
+ )
+ add(
+ row,
+ result.uncertain
+ ? 'conservative dynamic/unparsed import'
+ : 'changed import dependency',
+ );
+ }
+ }
+ if (selected.size > 500)
+ throw new Error(
+ `Sample expands to ${selected.size} documents; use the ci/full
label or manual full validation instead of truncating affected documents`,
+ );
+ return {rows, selected};
+}
+function prune(routes, docPaths, chosen, versions, inDocs = false) {
+ const categories = new Set();
+ function walk(route, parentDocs) {
+ const docs = parentDocs || route.plugin?.name ===
'docusaurus-plugin-content-docs';
+ if (route.routes) {
+ const children = route.routes.map(child => walk(child,
docs)).filter(Boolean);
+ return children.length ? {...route, routes: children} : undefined;
+ }
+ if (!docs) return route;
+ if (docPaths.has(key(route.path))) return chosen.has(key(route.path))
? route : undefined;
+ if (/Redirect/.test(route.component)) return route;
+ const group =
+ versions.find(v => key(route.path).startsWith(key(v.path) +
'/'))?.key ?? 'root';
+ if (!categories.has(group)) {
+ categories.add(group);
+ return route;
+ }
+ return undefined;
+ }
+ return routes.map(route => walk(route, inDocs)).filter(Boolean);
+}
+function stripUnbuiltAnchors(collected, known, rendered, aliases) {
+ let skipped = 0,
+ metadataValidated = 0;
+ const result = {};
+ for (const [from, data] of Object.entries(collected))
+ result[from] = {
+ ...data,
+ links: data.links.flatMap(link => {
+ const url = new URL(link, 'https://ci.invalid' + from);
+ const alias = aliases.get(key(url.pathname));
+ const target = alias ? new URL(alias, 'https://ci.invalid') :
url;
+ if (alias && !target.hash) target.hash = url.hash;
+ const targetKey = key(target.pathname);
+ if (url.hash && known.has(targetKey) &&
!rendered.has(targetKey)) {
+ skipped++;
+ metadataValidated++;
+ return [];
+ }
+ if (!url.hash && known.has(targetKey)) {
+ metadataValidated++;
+ return [];
+ }
+ return [alias ? target.pathname + target.search + target.hash
: link];
+ }),
+ };
+ return {collected: result, skipped, metadataValidated};
+}
+
+async function install({root, outDir, locale, planFile}) {
+ assert.equal(
+ require('@docusaurus/core/package.json').version,
+ '3.6.3',
+ 'Revalidate the CI-only Docusaurus adapter',
+ );
+ const plan = JSON.parse(fs.readFileSync(planFile));
+ assert.equal(plan.mode, 'sample');
+ assert.equal(plan.sourceSha, sha(root), 'Sample plan is stale');
+ assert(plan.locales.includes(locale), 'Locale is outside the requested
sample');
+ const diff = require('node:crypto')
+ .createHash('sha256')
+ .update(
+ execFileSync('git', ['diff', 'HEAD', '--binary'], {
+ cwd: root,
+ maxBuffer: 16 * 1024 * 1024,
+ }),
+ )
+ .digest('hex');
+ assert.equal(plan.trackedDiffHash, diff, 'Source changed after sample
planning');
+ const siteModule = require('@docusaurus/core/lib/server/site.js');
+ const linksModule = require('@docusaurus/core/lib/server/brokenLinks.js');
+ const originalSite = siteModule.loadSite,
+ originalLinks = linksModule.handleBrokenLinks;
+ const updates = require('@docusaurus/utils/lib/lastUpdateUtils.js'),
+ readUpdate = updates.readLastUpdateData;
+ let deferredDates = 0;
+ // Timestamps are irrelevant to the full ID/route/sidebar census. Resolve
+ // their real source data only for documents that will actually be
rendered.
+ updates.readLastUpdateData = async () => {
+ deferredDates++;
+ return {};
+ };
+ let scope,
+ fullRoutes,
+ aliases = new Map(),
+ known,
+ rendered;
+ const scopeDir = path.join(root, '.i18n-build/ci-sample/scopes');
+ fs.mkdirSync(scopeDir, {recursive: true});
+ const save = () =>
+ fs.writeFileSync(
+ path.join(scopeDir, locale + '.json'),
+ JSON.stringify(scope, null, 2) + '\n',
+ );
+ siteModule.loadSite = async args => {
+ const started = Date.now();
+ const site = await originalSite(args),
+ props = site.props;
+ fullRoutes = props.routes;
+ const {rows, selected} = await selectDocs(props, plan);
+ const docPaths = new Set(rows.map(r => key(r.route))),
+ chosen = new Set([...selected.values()].map(x =>
key(x.row.route)));
+ const versions = props.plugins
+ .filter(p => p.name === 'docusaurus-plugin-content-docs')
+ .flatMap(p =>
+ p.content.loadedVersions.map(v => ({
+ path: v.path,
+ key: p.options.id + ':' + v.versionName,
+ })),
+ )
+ .sort((a, b) => b.path.length - a.path.length);
+ for (const plugin of props.plugins.filter(p => p.name ===
'docusaurus-plugin-content-docs'))
+ for (const version of plugin.content.loadedVersions)
+ for (const doc of version.docs.filter(d =>
chosen.has(key(d.permalink)))) {
+ const state = doc.translation;
+ const file = state
+ ? path.resolve(
+ root,
+ state.contentLocale === 'en' ? state.englishPath
: state.sourcePath,
+ )
+ : sourceFile(root, doc.source);
+ const utils = require('@docusaurus/utils');
+ const parsed = await
props.siteConfig.markdown.parseFrontMatter({
+ filePath: file,
+ fileContent: fs.readFileSync(file, 'utf8'),
+ defaultParseFrontMatter:
utils.DEFAULT_PARSE_FRONT_MATTER,
+ });
+ Object.assign(
+ doc,
+ await readUpdate(
+ file,
+ state?.useTarget
+ ? {...plugin.options, showLastUpdateTime: true}
+ : plugin.options,
+ parsed.frontMatter.last_update,
+ ),
+ );
+ fs.writeFileSync(
+ path.join(
+ props.generatedFilesDir,
+ 'docusaurus-plugin-content-docs',
+ plugin.options.id,
+ utils.docuHash(doc.source) + '.json',
+ ),
+ JSON.stringify(doc, null, 2),
+ );
+ }
+ for (const route of flatten(fullRoutes)) {
+ const row = rows.find(r => key(r.route) === key(route.path));
+ if (row && chosen.has(key(row.route))) {
+ const plugin = props.plugins.find(
+ p =>
+ p.name === 'docusaurus-plugin-content-docs' &&
+ p.options.id === row.pluginId,
+ );
+ const doc = plugin.content.loadedVersions
+ .find(v => v.versionName === row.version)
+ .docs.find(d => d.id === row.docId);
+ route.metadata = {...route.metadata, lastUpdatedAt:
doc.lastUpdatedAt};
+ }
+ }
+ const pruned = prune(fullRoutes, docPaths, chosen, versions);
+ const {getRoutesPaths} =
require('@docusaurus/core/lib/server/routes.js');
+ const allPaths = getRoutesPaths(fullRoutes, props.baseUrl);
+ known = new Set(allPaths.map(key));
+ const metadata = props.plugins.flatMap(p =>
p.content?.contentIdentity?.entries ?? []);
+ for (const entry of metadata) known.add(key(entry.route));
+ const pathSet = new Set(allPaths.map(key));
+ const logical = [
+ ...fullRoutes,
+ ...metadata
+ .filter(e => !pathSet.has(key(e.route)))
+ .map(e => ({path: e.route, exact: true, component:
'metadata-only'})),
+ ];
+ for (const plugin of props.plugins.filter(
+ p => p.name === 'docusaurus-plugin-client-redirects',
+ )) {
+ const context = {
+ relativeRoutesPaths: allPaths.map(
+ p => '/' + p.slice(props.baseUrl.length).replace(/^\//,
''),
+ ),
+ baseUrl: props.baseUrl,
+ outDir: props.outDir,
+ options: plugin.options,
+ siteConfig: props.siteConfig,
+ };
+ const redirects =
+
require('@docusaurus/plugin-client-redirects/lib/collectRedirects').default(
+ context,
+ props.siteConfig.trailingSlash,
+ );
+ const absolute = value =>
+ props.baseUrl.replace(/\/$/, '') + '/' + value.replace(/^\//,
'');
+ for (const redirect of redirects) {
+ const from = absolute(redirect.from),
+ to = redirect.to.startsWith('/') ? absolute(redirect.to) :
redirect.to;
+ known.add(key(from));
+ aliases.set(key(from), to);
+ logical.push({path: from, exact: true, component:
'redirect-metadata'});
+ }
+ plugin.postBuild = async () => {
+ const kept = redirects.filter(
+ r =>
+ !r.to.startsWith('/') ||
rendered.has(key(absolute(r.to).split(/[?#]/)[0])),
+ );
+ const write =
require('@docusaurus/plugin-client-redirects/lib/writeRedirectFiles');
+ await write.default(
+ write.toRedirectFiles(kept, context,
props.siteConfig.trailingSlash),
+ );
+ };
+ }
+ // Static files are real public targets too. Do not invent routes for
+ // missing resources merely to silence partial-build link checks.
+ const walkStatic = (directory, prefix = '') => {
+ if (!fs.existsSync(directory)) return;
+ for (const entry of fs.readdirSync(directory, {withFileTypes:
true})) {
+ if (entry.name.startsWith('.')) continue;
+ const relative = prefix + entry.name,
+ file = path.join(directory, entry.name);
+ if (entry.isDirectory()) walkStatic(file, relative + '/');
+ else {
+ known.add(key('/' + relative));
+ known.add(key(props.baseUrl + relative));
+ }
+ }
+ };
+ for (const directory of props.siteConfig.staticDirectories)
+ walkStatic(path.resolve(root, directory));
+ fullRoutes = logical;
+ props.routes = pruned;
+ props.routesPaths = getRoutesPaths(pruned, props.baseUrl);
+ rendered = new Set(props.routesPaths.map(key));
+ // Native search must index compiled bodies only, never metadata-only
routes.
+ for (const plugin of props.plugins.filter(
+ p => p.name === 'docusaurus-plugin-content-docs',
+ )) {
+ for (const version of plugin.content.loadedVersions)
+ version.docs = version.docs.filter(d =>
chosen.has(key(d.permalink)));
+ if (plugin.content.translationManifest)
+ plugin.content.translationManifest.pages =
+ plugin.content.translationManifest.pages.filter(p =>
+ rendered.has(key(p.permalink)),
+ );
+ }
+ const manifest = path.join(root, '.i18n-build/manifests', locale +
'.json');
+ if (fs.existsSync(manifest)) {
+ const data = JSON.parse(fs.readFileSync(manifest));
+ data.scope = {mode: 'sample', nativePages: data.pages.length};
+ data.pages = data.pages.filter(p =>
rendered.has(key(p.permalink)));
+ fs.writeFileSync(manifest, JSON.stringify(data, null, 2) + '\n');
+ }
+ await
require('@docusaurus/core/lib/server/codegen/codegenRoutes').generateRouteFiles({
+ generatedFilesDir: props.generatedFilesDir,
+ routes: pruned,
+ baseUrl: props.baseUrl,
+ });
+ scope = {
+ schemaVersion: 1,
+ mode: 'sample',
+ status: 'planned',
+ publicationAllowed: false,
+ sourceSha: plan.sourceSha,
+ locale,
+ metadataDocs: rows.length,
+ renderedDocs: selected.size,
+ metadataLoadMs: Date.now() - started,
+ deferredTimestampReads: deferredDates,
+ metadataRoutes: known.size,
+ compiledRoutes: props.routesPaths,
+ selected: [...selected.values()].map(({row, reasons}) => ({...row,
reasons})),
+ knownRoutes: [...known],
+ anchorChecksSkipped: 0,
+ };
+ props.plugins.push({
+ name: 'ci-sample-provenance',
+ configureWebpack(_config, isServer) {
+ return {
+ plugins: [
+ {
+ apply(compiler) {
+ compiler.hooks.done.tap('CICompiledModules',
stats => {
+ const files = new Set(),
+ seen = new Set();
+ const visit = module => {
+ if (seen.has(module)) return;
+ seen.add(module);
+ const resource =
module.resource?.split('?')[0];
+ if (
+ resource?.startsWith(root +
path.sep) &&
+ /\.mdx?$/i.test(resource)
+ )
+ files.add(gitPath(root, resource));
+ for (const child of module.modules ??
[]) visit(child);
+ };
+ for (const module of
stats.compilation.modules) visit(module);
+ scope.compiledMdxModules ??= {};
+ scope.compiledMdxModules[isServer ?
'server' : 'client'] = [
+ ...files,
+ ].sort();
+ save();
+ });
+ },
+ },
+ ],
+ };
+ },
+ async postBuild() {
+ scope.status = 'compiled';
+ save();
+ fs.writeFileSync(
+ path.join(props.outDir, '.ci-validation.json'),
+ JSON.stringify({
+ schemaVersion: 1,
+ mode: 'sample',
+ publicationAllowed: false,
+ sourceSha: plan.sourceSha,
+ }) + '\n',
+ );
+ },
+ });
+ save();
+ return site;
+ };
+ linksModule.handleBrokenLinks = async params => {
+ const sanitized = stripUnbuiltAnchors(params.collectedLinks, known,
rendered, aliases);
+ scope.anchorChecksSkipped = sanitized.skipped;
+ scope.linksValidatedFromMetadata = sanitized.metadataValidated;
+ save();
+ return originalLinks({...params, collectedLinks: sanitized.collected,
routes: fullRoutes});
+ };
+ return () => {
+ siteModule.loadSite = originalSite;
+ linksModule.handleBrokenLinks = originalLinks;
+ updates.readLastUpdateData = readUpdate;
+ };
+}
+module.exports = {
+ install,
+ selectDocs,
+ chooseRepresentatives,
+ rowsFor,
+ prune,
+ dependencies,
+ stripUnbuiltAnchors,
+};
diff --git a/scripts/i18n/pr-preflight.cjs b/scripts/i18n/pr-preflight.cjs
index 732f22ba310f..8d663d37c808 100644
--- a/scripts/i18n/pr-preflight.cjs
+++ b/scripts/i18n/pr-preflight.cjs
@@ -11,7 +11,8 @@ async function preflight({root, base, locale =
pipelineLocale(root)}) {
{cwd: root, stdio: 'inherit', env: {...process.env, I18N_LOCALE:
locale}});
const files = execFileSync('git', ['diff', '--name-only',
'--diff-filter=AM', '-z', base, 'HEAD'],
{cwd: root, encoding: 'utf8'}).split('\0').filter(Boolean)
- .filter(file =>
/^i18n\/[^/]+\/docusaurus-plugin-content-docs[^/]*\/.+\.(mdx?|tsx?)$/.test(file));
+ .filter(file => file.startsWith(`i18n/${locale}/`)
+ &&
/^i18n\/[^/]+\/docusaurus-plugin-content-docs[^/]*\/.+\.(mdx?|tsx?)$/.test(file));
for (const targetPath of files) {
const target = fs.readFileSync(path.join(root, targetPath), 'utf8');
await checkSyntax(target, path.extname(targetPath));
diff --git a/scripts/i18n/verify-ci-sample.cjs
b/scripts/i18n/verify-ci-sample.cjs
new file mode 100644
index 000000000000..f989aeaa12c0
--- /dev/null
+++ b/scripts/i18n/verify-ci-sample.cjs
@@ -0,0 +1,115 @@
+const fs = require('node:fs');
+const path = require('node:path');
+const assert = require('node:assert/strict');
+const cheerio = require('cheerio');
+const {verify: verifyStatic} = require('./verify-static.cjs');
+const {createServer} = require('../publish/serve.cjs');
+async function verify({root = process.cwd(), browser = true}) {
+ const site = path.join(root, '.i18n-build/ci-sample/site');
+ const plan = JSON.parse(fs.readFileSync(path.join(root,
'.i18n-build/ci-plan.json')));
+ const marker = JSON.parse(fs.readFileSync(path.join(site,
'.ci-validation.json')));
+ assert.equal(marker.publicationAllowed, false);
+ assert.equal(marker.source?.sha ?? marker.sourceSha, plan.sourceSha,
'Stale sample artifact');
+ const build = JSON.parse(fs.readFileSync(path.join(root,
'.i18n-build/build-result.json')));
+ assert.equal(build.success, true);
+ assert.equal(build.source.sha, plan.sourceSha);
+ assert.equal(build.validation?.mode, 'sample');
+ assert.equal(build.source.trackedDiffHash, plan.trackedDiffHash);
+ const results = [];
+ for (const locale of plan.locales) {
+ const scope = JSON.parse(
+ fs.readFileSync(path.join(root, '.i18n-build/ci-sample/scopes',
locale + '.json')),
+ );
+ assert.equal(scope.sourceSha, plan.sourceSha);
+ assert.equal(scope.status, 'compiled');
+ assert(scope.compiledMdxModules?.client?.length > 0);
+ assert(scope.renderedDocs > 0 && scope.renderedDocs <=
scope.metadataDocs);
+ for (const row of scope.selected) {
+ const filename = path.join(site, decodeURIComponent(row.route),
'index.html');
+ assert(fs.existsSync(filename), `Missing selected document:
${row.route}`);
+ const $ = cheerio.load(fs.readFileSync(filename));
+ assert.equal($('link[rel="canonical"]').length, 1, row.route);
+ assert($('main').text().trim(), row.route);
+ }
+ if (['ja', 'ko', 'ru'].includes(locale))
+ verifyStatic({
+ site,
+ manifest: path.join(root, '.i18n-build/manifests', locale +
'.json'),
+ partial: true,
+ });
+ results.push({
+ locale,
+ metadataDocs: scope.metadataDocs,
+ renderedDocs: scope.renderedDocs,
+ metadataRoutes: scope.metadataRoutes,
+ compiledRoutes: scope.compiledRoutes.length,
+ compiledMdxModules: new
Set(Object.values(scope.compiledMdxModules).flat()).size,
+ anchorChecksSkipped: scope.anchorChecksSkipped,
+ });
+ }
+ if (browser) {
+ const {chromium} = require('playwright'),
+ server = createServer(site);
+ await new Promise(resolve => server.listen(0, '127.0.0.1', resolve));
+ const origin = `http://127.0.0.1:${server.address().port}`;
+ let instance;
+ try {
+ instance = await chromium.launch({headless: true});
+ for (const locale of plan.locales) {
+ const context = await instance.newContext({serviceWorkers:
'block'}),
+ page = await context.newPage();
+ const errors = [];
+ page.on('pageerror', e => errors.push(e.message));
+ const scope = JSON.parse(
+ fs.readFileSync(
+ path.join(root, '.i18n-build/ci-sample/scopes', locale
+ '.json'),
+ ),
+ );
+ const row =
+ scope.selected.find(r => r.pluginId === 'default' &&
r.version === 'current') ??
+ scope.selected[0];
+ const response = await page.goto(origin + row.route);
+ assert.equal(response.status(), 200);
+ await page.locator('html[data-has-hydrated="true"]').waitFor();
+ assert.deepEqual(errors, [], locale);
+ await context.close();
+ }
+ } finally {
+ await instance?.close();
+ await new Promise(resolve => server.close(resolve));
+ }
+ }
+ const report = {
+ schemaVersion: 1,
+ mode: 'sample',
+ publicationAllowed: false,
+ sourceSha: plan.sourceSha,
+ completeDocumentCompilation: false,
+ results,
+ };
+ if (process.env.GITHUB_STEP_SUMMARY)
+ fs.appendFileSync(
+ process.env.GITHUB_STEP_SUMMARY,
+ '| Language | Native document metadata | Rendered documents |
Compiled MDX modules (including blog/imports) | Unverified destination anchors
|\n' +
+ '| --- | ---: | ---: | ---: | ---: |\n' +
+ results
+ .map(
+ r =>
+ `| ${r.locale} | ${r.metadataDocs} |
${r.renderedDocs} | ${r.compiledMdxModules} | ${r.anchorChecksSkipped} |`,
+ )
+ .join('\n') +
+ '\n\n',
+ );
+ fs.writeFileSync(
+ path.join(root, '.i18n-build/ci-sample/verification.json'),
+ JSON.stringify(report, null, 2) + '\n',
+ );
+ console.log(JSON.stringify(report));
+ return report;
+}
+if (require.main === module)
+ verify({browser: !process.argv.includes('--no-browser')}).catch(e => {
+ console.error(e);
+ process.exitCode = 1;
+ });
+module.exports = {verify};
diff --git a/scripts/i18n/verify-ci-sampling-fixture.cjs
b/scripts/i18n/verify-ci-sampling-fixture.cjs
new file mode 100644
index 000000000000..2839cc23ab04
--- /dev/null
+++ b/scripts/i18n/verify-ci-sampling-fixture.cjs
@@ -0,0 +1,120 @@
+const fs = require('node:fs'),
+ path = require('node:path'),
+ assert = require('node:assert/strict');
+const {execFileSync, spawnSync} = require('node:child_process');
+const {createFixture} = require('./__tests__/fixtures/multi-locale.cjs');
+const {plan} = require('./ci-plan.cjs');
+const {verify} = require('./verify-ci-sample.cjs');
+const root = path.resolve(__dirname, '../..');
+async function main() {
+ const site = path.join(root, '.i18n-build/ci-sampling-fixture');
+ fs.rmSync(site, {recursive: true, force: true});
+ const {write} = await createFixture(site);
+ write(
+ 'docs/02-rich.mdx',
+ '---\ntitle: Rich MDX\n---\n\nimport Widget from
"@site/src/components/FixtureRich";\n\n<Widget />\n\n$$ x + 1 $$\n',
+ );
+ write(
+ 'src/components/FixtureRich.tsx',
+ "import React from 'react'; export default function Widget(){return
<p>Rich module compiled</p>;}\n",
+ );
+ write(
+ 'docs/zz-unselected.mdx',
+ '---\ntitle: Unselected module\n---\n\nimport Missing from
"./does-not-exist";\n\n<Missing />\n',
+ );
+ fs.appendFileSync(
+ path.join(site, 'docs/02-rich.mdx'),
+ '\n[Known uncompiled route](./zz-unselected.mdx#not-checked)\n',
+ );
+ const git = (...args) =>
+ execFileSync('git', args, {cwd: site, encoding: 'utf8', stdio:
'pipe'}).trim();
+ const commit = () => {
+ git('add', '.');
+ git(
+ '-c',
+ 'user.name=Fixture',
+ '-c',
+ '[email protected]',
+ 'commit',
+ '-qm',
+ 'Sample inputs',
+ );
+ };
+ commit();
+ const base = git('rev-parse', 'HEAD'),
+ report = plan({root: site, base, forceSample: true});
+ fs.mkdirSync(path.join(site, '.i18n-build'), {recursive: true});
+ const file = path.join(site, '.i18n-build/ci-plan.json');
+ fs.writeFileSync(file, JSON.stringify(report, null, 2));
+ const run = (stdio = 'inherit') =>
+ spawnSync(process.execPath, [path.join(root,
'scripts/i18n/build-ci-sample.cjs'), file], {
+ cwd: site,
+ stdio,
+ encoding: 'utf8',
+ env: {
+ ...process.env,
+ NODE_OPTIONS: '--max-old-space-size=4096',
+ DORIS_OPTIMIZE_PUBLISH: 'false',
+ },
+ });
+ assert.equal(run().status, 0, 'Representative sample build');
+ const scopes = report.locales.map(locale =>
+ JSON.parse(
+ fs.readFileSync(path.join(site, '.i18n-build/ci-sample/scopes',
locale + '.json')),
+ ),
+ );
+ for (const scope of scopes) {
+ assert(scope.renderedDocs < scope.metadataDocs);
+ assert(!scope.selected.some(r => r.docId === 'zz-unselected'));
+ assert(scope.compiledMdxModules.client.length > 0);
+
assert(!scope.compiledMdxModules.client.includes('docs/zz-unselected.mdx'));
+ }
+ await verify({root: site, browser: process.argv.includes('--browser')});
+ assert(
+ scopes.find(s => s.locale === 'en').anchorChecksSkipped > 0,
+ 'Uncompiled anchors must be disclosed',
+ );
+ const proof = {
+ schemaVersion: 1,
+ implementation: gitRootSha(),
+ sourceSha: report.sourceSha,
+ locales: scopes.map(s => ({
+ locale: s.locale,
+ metadataDocs: s.metadataDocs,
+ renderedDocs: s.renderedDocs,
+ skippedAnchors: s.anchorChecksSkipped,
+ })),
+ excludedBrokenModuleWasNotCompiled: true,
+ };
+ // A real edit makes that same module mandatory. Its missing import must
now
+ // fail compilation instead of being silently dropped by the sampler.
+ fs.appendFileSync(path.join(site, 'docs/zz-unselected.mdx'),
'\nChanged.\n');
+ commit();
+ const changed = plan({root: site, base, forceSample: true});
+ fs.writeFileSync(file, JSON.stringify(changed, null, 2));
+ const failed = run('pipe');
+ fs.writeFileSync(
+ path.join(root, '.i18n-build/ci-sampling-expected-failure.log'),
+ failed.stdout + failed.stderr,
+ );
+ assert.notEqual(failed.status, 0, 'Changed module with a broken import
must fail');
+ assert.match(
+ failed.stdout + failed.stderr,
+ /does-not-exist/,
+ 'The changed document must reach the compiler',
+ );
+ proof.changedBrokenModuleFailed = true;
+ fs.writeFileSync(
+ path.join(root, '.i18n-build/ci-sampling-fixture-proof.json'),
+ JSON.stringify(proof, null, 2) + '\n',
+ );
+ console.log(JSON.stringify(proof));
+}
+function gitRootSha() {
+ return execFileSync('git', ['rev-parse', 'HEAD'], {cwd: root, encoding:
'utf8'}).trim();
+}
+if (require.main === module)
+ main().catch(e => {
+ console.error(e);
+ process.exitCode = 1;
+ });
diff --git a/scripts/publish/__tests__/apache-startup.test.cjs
b/scripts/publish/__tests__/apache-startup.test.cjs
new file mode 100644
index 000000000000..506555b02bf8
--- /dev/null
+++ b/scripts/publish/__tests__/apache-startup.test.cjs
@@ -0,0 +1,23 @@
+const test = require('node:test');
+const assert = require('node:assert/strict');
+const fs = require('node:fs');
+const path = require('node:path');
+const os = require('node:os');
+const {apache} = require('../apache.cjs');
+
+test('Apache startup failures expose the error log and clean up their runtime
directory', async t => {
+ const site = fs.mkdtempSync(path.join(os.tmpdir(),
'apache-startup-test-'));
+ t.after(() => fs.rmSync(site, {recursive: true, force: true}));
+ const binary = path.join(site, 'fake-apache');
+ const captured = path.join(site, 'runtime-directory');
+ fs.writeFileSync(binary, `#!${process.execPath}
+const fs = require('node:fs'), path = require('node:path');
+const config = fs.readFileSync(process.argv[process.argv.indexOf('-f') + 1],
'utf8');
+const log = config.match(/ErrorLog "([^"]+)"/)[1];
+fs.writeFileSync(${JSON.stringify(captured)}, path.dirname(log));
+fs.writeFileSync(log, 'AH01597: missing MIME configuration');
+process.exit(1);
+`, {mode: 0o755});
+ await assert.rejects(apache(site, {binary}), /AH01597: missing MIME
configuration/);
+ assert.equal(fs.existsSync(fs.readFileSync(captured, 'utf8')), false);
+});
diff --git a/scripts/publish/apache.cjs b/scripts/publish/apache.cjs
index 3d22354a7f73..7bd08d5cee34 100644
--- a/scripts/publish/apache.cjs
+++ b/scripts/publish/apache.cjs
@@ -5,7 +5,7 @@ const os = require('node:os');
const net = require('node:net');
const {spawn} = require('node:child_process');
-async function apache(site) {
+async function apache(site, {binary = process.platform === 'darwin' ?
'/usr/sbin/httpd' : '/usr/sbin/apache2'} = {}) {
const directory =
fs.mkdtempSync(path.join(os.tmpdir(),'doris-publish-apache-'));
const socket = net.createServer();
await new Promise(resolve => socket.listen(0,'127.0.0.1',resolve));
@@ -20,25 +20,42 @@ async function apache(site) {
.map(name => `LoadModule ${name}_module
"${moduleDir}/mod_${name}.so"`),
`User ${os.userInfo().username}`, `Group #${process.getgid()}`,
`ErrorLog "${directory}/error.log"`,
'LogLevel warn', 'StartServers 1','ServerLimit 1','ThreadsPerChild
8','MaxRequestWorkers 8',
- 'TypesConfig /etc/apache2/mime.types', `DocumentRoot
"${path.resolve(site)}"`,
+ `TypesConfig ${process.platform === 'darwin' ?
'/etc/apache2/mime.types' : '/etc/mime.types'}`, `DocumentRoot
"${path.resolve(site)}"`,
`<Directory "${path.resolve(site)}">`, 'AllowOverride All','Require
all granted','DirectoryIndex index.html','</Directory>',
].join('\n');
const configFile = path.join(directory,'httpd.conf');
fs.writeFileSync(configFile,config);
- const binary = process.platform === 'darwin' ? '/usr/sbin/httpd' :
'/usr/sbin/apache2';
const child =
spawn(binary,['-f',configFile,'-D','FOREGROUND'],{stdio:['ignore','ignore','pipe']});
- let stderr = '';
+ let stderr = '', spawnError;
+ const exited = new Promise(resolve => {
+ child.once('exit', resolve);
+ child.once('error', error => {spawnError = error; resolve();});
+ });
child.stderr.on('data',data => {stderr += data;});
const origin = `http://127.0.0.1:${port}`;
- for (let attempt = 0; attempt < 100; attempt++) {
- if (child.exitCode !== null) throw new Error(`Apache failed:
${stderr}`);
- try {await fetch(origin);break;} catch {}
- await new Promise(resolve => setTimeout(resolve,50));
- }
- return {origin,directory,async close() {
- child.kill('SIGTERM');
- if (child.exitCode === null) await new Promise(resolve =>
child.once('exit',resolve));
+ const close = async () => {
+ if (!spawnError && child.exitCode === null && child.signalCode ===
null) {
+ child.kill('SIGTERM');
+ await exited;
+ }
fs.rmSync(directory,{recursive:true,force:true});
- }};
+ };
+ try {
+ const deadline = Date.now() + 5000;
+ while (Date.now() < deadline) {
+ if (spawnError || child.exitCode !== null || child.signalCode !==
null) throw new Error('Apache failed to start');
+ try {
+ await fetch(origin, {method: 'HEAD', signal:
AbortSignal.timeout(250)});
+ return {origin, directory, close};
+ } catch {}
+ await new Promise(resolve => setTimeout(resolve,50));
+ }
+ throw new Error('Apache did not respond within 5 seconds');
+ } catch (error) {
+ const log = path.join(directory, 'error.log');
+ const diagnostic = fs.existsSync(log) ? fs.readFileSync(log,
'utf8').slice(-8192) : '';
+ await close();
+ throw new Error([error.message, spawnError?.message, stderr.trim(),
diagnostic.trim()].filter(Boolean).join('\n'));
+ }
}
module.exports = {apache};
diff --git a/scripts/publish/optimize.cjs b/scripts/publish/optimize.cjs
index 29d96c66207f..d593430ba478 100644
--- a/scripts/publish/optimize.cjs
+++ b/scripts/publish/optimize.cjs
@@ -24,6 +24,7 @@ function inventory(root) {
}
function optimize({site, locales, reportFile}) {
+ assert(!fs.existsSync(path.join(site,'.ci-validation.json')), 'Sample
validation artifacts must not be published');
site = path.resolve(site);
const before = inventory(site);
const prefixes = locales.filter(locale => locale.directory !==
'.').map(locale => locale.directory);
diff --git a/scripts/publish/verify-fixture.cjs
b/scripts/publish/verify-fixture.cjs
index 8b0b2bcdfd40..7b6b8197883c 100644
--- a/scripts/publish/verify-fixture.cjs
+++ b/scripts/publish/verify-fixture.cjs
@@ -10,6 +10,13 @@ const root = path.resolve(__dirname,'../..');
const fixture = path.join(root,'.i18n-build/publish-fixture');
const locales = ['en','zh-CN','ja','ko','ru'];
const hash = content => createHash('sha256').update(content).digest('hex');
+// Status/header probes still need to finish their response body so Undici can
+// reuse the connection. Otherwise a long Apache probe can exhaust its workers.
+async function probe(url, options) {
+ const response = await fetch(url, options);
+ await response.arrayBuffer();
+ return response;
+}
async function main() {
const verifyOnly = process.argv.includes('--verify-existing');
@@ -81,7 +88,7 @@
themes:[['@yang1666204/docusaurus-search-local',{hashed:true,language:['en'],doc
for (const origin of [previewOrigin,...(apache ? [apache.origin] :
[])]) {
const manifest =
JSON.parse(fs.readFileSync(path.join(site,'.publish/redirects.json'),'utf8'));
for (const [from,to,preserveQuery,status = 301] of
manifest.redirects) {
- const response = await
fetch(origin+from+'?probe=1',{redirect:'manual'});
+ const response = await
probe(origin+from+'?probe=1',{redirect:'manual'});
assert.equal(response.status,status,`${origin}${from}`);
const location = new
URL(response.headers.get('location'),origin);
const expected = new URL(to,origin);
@@ -95,10 +102,10 @@
themes:[['@yang1666204/docusaurus-search-local',{hashed:true,language:['en'],doc
const $ = require('cheerio').load(html);
const urls =
$('script[src],link[rel="stylesheet"],img[src]').map((_,node) =>
$(node).attr('src') ?? $(node).attr('href')).get();
for (const resource of urls.filter(value =>
!value.startsWith('data:'))) {
- const response = await fetch(new URL(resource,origin));
+ const response = await probe(new URL(resource,origin));
assert.equal(response.status,200,resource);
}
- assert.equal((await
fetch(origin+prefix+'/not-a-page')).status,404);
+ assert.equal((await
probe(origin+prefix+'/not-a-page')).status,404);
}
}
let browser;
diff --git a/scripts/publish/verify.cjs b/scripts/publish/verify.cjs
index b414bd6fe3f2..bf9f4712e318 100644
--- a/scripts/publish/verify.cjs
+++ b/scripts/publish/verify.cjs
@@ -5,6 +5,7 @@ const {inventory} = require('./optimize.cjs');
function verify(site) {
site = path.resolve(site);
+ assert(!fs.existsSync(path.join(site,'.ci-validation.json')), 'Sample
validation artifacts must not be published');
const manifest =
JSON.parse(fs.readFileSync(path.join(site,'.publish/redirects.json'),'utf8'));
const redirects = new Map(manifest.redirects.map(([from,to]) =>
[decodeURIComponent(from),to]));
const files = inventory(site);
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
