This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new cc7b7d281e7 [fix](i18n) De-index untranslated blog pages, publish
every locale sitemap, and fix localized heading IDs and search (#4325)
cc7b7d281e7 is described below
commit cc7b7d281e7eb168acadf35565e4119cfd2cfea6
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Fri Oct 16 03:44:06 2026 +0800
[fix](i18n) De-index untranslated blog pages, publish every locale sitemap,
and fix localized heading IDs and search (#4325)
## Summary
This PR combines #4325 and #4327, rebased on master `17fc8205f0`. #4327
is closed in favour of this one.
It also fixes the Build Check failure of #4325 ([run
38716649929](https://github.com/apache/doris-website/actions/runs/38716649929/job/116360867466)).
`verify-static` stopped at `Source heading IDs:
/ja/docs/4.x/faq/bi-faq/`: the Japanese page carried
`q2-ssl-connection-errorprotocol-version-mismatch-…`, while the English
page has `q2-ssl-connection-error-version-mismatch-…`. Docusaurus parses
`:protocol` in `error:protocol` as a text directive and leaves it out of
the anchor; the translation parser did not.
It also unblocks the deployment of master: [Cron Deploy
38730035203](https://github.com/apache/doris-website/actions/runs/38730035203)
built and verified `17fc8205f0`, but GitHub rejected the `asf-site` push
because `ja/search-index.json` was 110.22 MB, over its 100 MiB file
limit (section 5).
### 1. Localized blog pages of an untranslated blog
The blog has no translations: the Chinese ones were removed on purpose
in 2023 (#272) and Japanese has not started. The blog plugin falls back
per file, so `/zh-CN/blog/<post>/` and `/ja/blog/<post>/` served the
English article on an indexable, self-canonical URL, listed in that
locale's sitemap and advertised as an `hreflang` alternate. Every
article therefore existed two or three times as a page Google may pick.
`config/i18n.js` gains `hasBlogTranslations`: a locale counts as
translating the blog only when its `docusaurus-plugin-content-blog`
directory holds a post file (an empty directory, or one holding just
`tags.yml`, translates nothing). The render path reads the resulting
locale list through `defaultLocaleBlogLocales`.
- `SiteMetadata` marks every `/blog` route of such a locale `noindex,
follow`. It is the one component all blog routes pass through — the
archive page renders through the plain layout, not `BlogLayout` — and
the sitemap plugin drops noindex routes, so those URLs leave that
locale's sitemap too. The routes themselves stay browsable.
- `BlogPostPage/Metadata` points each of those posts at the default
locale's copy with a canonical link, in the URL form the `hreflang`
alternates already use.
Both decisions are derived from the filesystem, so translating the blog
lifts the gate by itself, and a test records the current intent so the
SEO decision is revisited rather than changed by accident.
### 2. Sitemap discovery and freshness
- `robots.txt` declares `/zh-CN/sitemap.xml` and `/ja/sitemap.xml`.
Google retired the sitemap ping endpoint, so robots.txt and Search
Console are the only ways a sitemap is found.
- The sitemap carries `lastmod` from each file's last commit.
`manual-deploy-website.yml` checks out full history for it;
`cron-deploy-website.yml` already did.
- The utility-route filter in `createSitemapItems` is locale-generic, so
`/ja/search` is excluded like its English and Chinese counterparts.
- The search page's robots directive uses `name=` instead of
`property=`, the form Google documents and the form the sitemap plugin
reads.
- The SEO lint looked for the literal `/zh-CN/search` in
`docusaurus.config.js` and therefore warned "Sitemap policy should
exclude search pages for all configured locales" on #4325. It now
accepts the locale-prefixed `${prefix}/search` form and still reports a
filter that excludes only `/search`.
### 3. Localized heading IDs
Master fixed the same parser bug in parallel (`88826df1fc`, #4322),
which is what conflicted with #4327. The resolution keeps a single
implementation:
- Heading IDs come from a directive-aware parse, as on master. Both
versions give identical IDs for all 3,014 recorded translations.
- `technicalHash` stays on the original parse, as in #4327. Every state
record was written with that fingerprint. Master's version derived it
from the directive-aware IDs, which left the BI FAQ records (dev and
4.x) with a `sourceTechnicalHash` that no longer matches their unchanged
source. Across all 6,482 source pages of the managed trees (`docs`,
`versioned_docs/version-4.x`, `community` and their zh-CN counterparts),
BI FAQ is the only page whose heading IDs change, so no other record is
affected.
- `remark-directive` is declared in `package.json`, since
`structure.cjs` imports it directly. `yarn.lock` already resolves
`^3.0.0` to 3.0.1.
- `heading-ids.test.cjs` compiles through Docusaurus itself and compares
the IDs for directives, duplicate headings, explicit IDs, headings
nested in admonitions and code blocks. The three-locale publishing
fixture gains the directive heading. Master's t04 cases pass unchanged.
### 4. Documentation search
- `searchInput.js` imports `language` from `proxiedGeneratedConstants`,
where the plugin exports it. Imported from `proxiedGenerated` it was
`undefined`, so the search dropdown and the full search page threw
`Cannot read properties of undefined (reading 'length')` after loading
their indexes, as reported on the [Dev Index Overview
page](https://doris.apache.org/docs/dev/table-design/index/index-overview/).
- `search-input.test.cjs` loads the installed plugin's generated
exports, so the wrong import fails the way it did online.
Both new suites run in Build Check.
### 5. Japanese search index over GitHub's file limit
The localized search writes one index per context (Dev, 4.x, community)
and, because `useAllContextsWithNoSearchContext` is on, a root
`ja/search-index.json` that held all of them again. It grew with every
translation batch — 27.9 MB in production this morning, 110.22 MB after
#4322 — until the push was rejected.
- The root index now holds only documents outside every context, as the
native plugin does without that option; for Japanese that is none. The
per-context files stay at 54.67 MB (Dev), 53.54 MB (4.x) and 3.12 MB
(community). Dev already has 1,634 of its 1,640 pages translated, so it
has little room left to grow.
- "Everywhere" keeps its results — the search page without a context,
the "See all results outside …" link, and the search bar on pages
outside the three contexts. `src/utils/searchContexts.js` queries each
context index and merges the results with the plugin's own ranking,
undoing the title boost each worker list already carries so it is
applied once.
- The search bar loaded its index on mount, before resolving the page's
context, and the search page fetched before reading `?ctx=`. Since an
empty context means "all", every Japanese docs page view and search page
load downloaded the whole union before the index it searched. Both now
wait for the context, so a page loads only the index it searches.
- Both deploy workflows check the build for files over 100 MiB before
the OSS upload, so a build like this one fails before anything is
published instead of at the `asf-site` push.
- `search-contexts.test.cjs` covers the fan-out, that a merge ranks like
one index over all hits, and that the root index stays free of context
documents; it runs in Build Check. `verify-search.cjs` gains the
"Everywhere" case and requires a single-context page to request only its
own index.
## Verification
Local run of the Build Check steps at the final head with Node 20.20.2,
the version CI uses:
- `check_move.py`, `check_move.js`: pass. `build-scope.cjs` selects
`DOCS_VERSIONS=4.x`, locales `en,zh-CN,ja`, full (not partial)
verification — the same scope as the failed run.
- The nine unit suites of the workflow (`t04`, `heading-ids`,
`search-input`, `search-contexts`, `locale-registry`, `planner`,
`translation-engine`, `workflow`, `codex-adapter`): 70 pass.
- `build:locales --locales en,zh-CN,ja` with `DOCS_VERSIONS=4.x
DORIS_ENABLE_LOCALES=ja`: all three locales built (en 96 s, zh-CN 87 s,
ja 101 s); no output file exceeds 100 MiB.
- `verify-static.cjs` on the Japanese manifest: 1,736 pages verified, 0
skipped (current 1,646, stale_technical 2, untranslated 89). The BI FAQ
anchors equal the English ones
(`q2-ssl-connection-error-version-mismatch-…`).
- Japanese translation gate: 0 errors.
- Docs link gate and docs governance report on the 25 changed files: 0
findings.
- Japanese build of Dev and 4.x (`DOCS_VERSIONS=current,4.x`, local
assets): `ja/search-index.json` is an empty index (was 110.22 MB), next
to Dev 54.67 MB, 4.x 53.54 MB and community 3.12 MB; the largest file is
a 57.80 MB Japanese PDF. `verify-static --partial` verifies 3,375 pages.
- `verify-search.cjs` in Chromium on that build, 12 checks: the seven
single-context search pages each request only their own index; two
"Everywhere" searches request exactly the three context indexes, never
the root, and return Dev and community results; the search bar on a Dev
page loads only the Dev index; mobile layout; the no-results link to the
English search. On 4.x pages the search bar loads nothing with the page
and only the 4.x index once the reader types.
- Also: the remaining i18n suites (`doc-navigation`, `inventory`,
`state`), `build-locales` and `poc/config` (12), docs governance (38,
including the new lint case), Python link tests (5): pass.
In the three-locale build output:
- Sitemaps: English has 1,976 entries including 214 blog routes; Chinese
(1,762) and Japanese (1,672) have no blog route; no locale lists its
`/search`; 99% of entries carry `lastmod`.
- The `/zh-CN/blog/` and `/ja/blog/` list, archive, tags and post pages
render `noindex, follow`, and their posts are canonical to the English
copy. The English post stays indexable and self-canonical;
`/ja/download/` and the localized docs pages are not affected.
- `robots.txt` lists the three sitemaps, and the three search pages
render `<meta name="robots" content="noindex, follow">`.
## Not in this change
- `/zh-CN/course/...`, `/zh-CN/why-doris/...` and `/zh-CN/use-cases/...`
still serve English pages on indexable Chinese URLs. Those are
untranslated pages rather than a fallback rule like the blog's, and
`config/i18n.js` declares Chinese ordinary pages translated, so deciding
their indexability is a product call.
- `htmlLang` and therefore `hreflang` stay `en-US` / `zh-Hans-CN`.
Narrowing those to `en` / `zh-CN` is a separate discussion.
## Versions
Shared build, runtime and SEO fixes apply to all documentation versions;
no document is edited.
- [x] dev
- [x] 4.x
- [x] 3.x
- [x] 2.1 or older (not covered by version/language sync gate)
## Languages
- [x] Chinese
- [x] English
## Docs Checklist
- [x] Checked by AI
- [x] Test Cases Built
- [x] Updated required version and language counterparts, or explained
why not: shared code and configuration; no document edits.
- [x] If only one language changed, confirmed whether source/translation
counterparts need sync: not applicable.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: morningman <[email protected]>
Co-authored-by: Claude Code <[email protected]>
---
.github/workflows/build-check.yml | 3 +
.github/workflows/cron-deploy-website.yml | 10 ++
.github/workflows/manual-deploy-website.yml | 13 +++
config/i18n.js | 22 ++++-
docusaurus.config.js | 35 +++++--
package.json | 1 +
.../__tests__/docs-governance.test.js | 29 ++++++
scripts/docs-governance/lint-seo.js | 5 +-
scripts/i18n/__tests__/heading-ids.test.cjs | 70 ++++++++++++++
scripts/i18n/__tests__/locale-registry.test.cjs | 26 ++++++
scripts/i18n/__tests__/search-contexts.test.cjs | 102 +++++++++++++++++++++
scripts/i18n/__tests__/search-input.test.cjs | 71 ++++++++++++++
scripts/i18n/search-plugin.cjs | 16 +++-
scripts/i18n/structure.cjs | 24 +++--
scripts/i18n/verify-fixture.cjs | 3 +-
scripts/i18n/verify-search.cjs | 28 +++++-
src/theme/BlogPostPage/Metadata/index.tsx | 12 +++
src/theme/SearchBar/SearchBar.jsx | 12 ++-
src/theme/SearchPage/SearchPage.jsx | 18 +++-
src/theme/SiteMetadata/index.tsx | 14 +++
src/utils/localization.ts | 33 +++++++
src/utils/searchContexts.js | 45 +++++++++
src/utils/searchInput.js | 2 +-
static/robots.txt | 5 +
website-quality-governance/seo-geo-baseline.md | 7 ++
25 files changed, 572 insertions(+), 34 deletions(-)
diff --git a/.github/workflows/build-check.yml
b/.github/workflows/build-check.yml
index 84c42a1e71e..0ab7003af0d 100644
--- a/.github/workflows/build-check.yml
+++ b/.github/workflows/build-check.yml
@@ -96,6 +96,9 @@ jobs:
yarn
yarn key-features:generate
node --test scripts/i18n/__tests__/t04.test.cjs
+ node --test scripts/i18n/__tests__/heading-ids.test.cjs
+ node --test scripts/i18n/__tests__/search-input.test.cjs
+ node --test scripts/i18n/__tests__/search-contexts.test.cjs
node --test scripts/i18n/__tests__/locale-registry.test.cjs
node --test scripts/i18n/__tests__/planner.test.cjs
node --test
scripts/i18n/__tests__/translation-engine.test.cjs
diff --git a/.github/workflows/cron-deploy-website.yml
b/.github/workflows/cron-deploy-website.yml
index f0ad0b0f235..b07af9f0552 100644
--- a/.github/workflows/cron-deploy-website.yml
+++ b/.github/workflows/cron-deploy-website.yml
@@ -103,6 +103,16 @@ jobs:
--site build --manifest
".i18n-build/manifests/${LOCALE}.json"
done
+ - name: Check file sizes for asf-site
+ # GitHub rejects a push holding any file over 100 MiB. Checking
+ # before the OSS upload stops such a build before anything is
+ # published. The PDFs under assets/files are removed before the
push.
+ run: |
+ if find build -type f -size +100M ! -path
'build/assets/files/*.pdf' -exec ls -l {} + | grep .; then
+ echo "::error::The files above exceed GitHub's 100 MiB
file limit; the asf-site push would be rejected."
+ exit 1
+ fi
+
# - name: Generate PDF
# run: |
# npx vitpress-generate-pdf
--initialDocURLs="https://doris.apache.org/docs/dev/get-starting/";
--paginationSelector=".pagination-nav__link--next" --contentSelector="article"
--coverImage="https://cdn.selectdb.com/static/doris_logo_512_4903556647.png";
--coverTitle="Apache Doris Docs (English)" --outputPDFFilename="Apache Doris
Docs (English).pdf" --tocOnlyH1=true
diff --git a/.github/workflows/manual-deploy-website.yml
b/.github/workflows/manual-deploy-website.yml
index 5a3e7879616..beb28d85393 100644
--- a/.github/workflows/manual-deploy-website.yml
+++ b/.github/workflows/manual-deploy-website.yml
@@ -28,6 +28,9 @@ jobs:
persist-credentials: false
submodules: recursive
ref: ${{ inputs.branch }}
+ # Sitemap lastmod comes from each file's last commit, so a
+ # shallow clone would date every page to the same commit.
+ fetch-depth: 0
- name: Add Chinese Fonts
run: |
@@ -82,6 +85,16 @@ jobs:
--site build --manifest
".i18n-build/manifests/${LOCALE}.json"
done
+ - name: Check file sizes for asf-site
+ # GitHub rejects a push holding any file over 100 MiB. Checking
+ # before the OSS upload stops such a build before anything is
+ # published.
+ run: |
+ if find build -type f -size +100M -exec ls -l {} + | grep .;
then
+ echo "::error::The files above exceed GitHub's 100 MiB
file limit; the asf-site push would be rejected."
+ exit 1
+ fi
+
- name: Upload files to OSS
id: oss-upload
# Same retry as cron-deploy-website.yml: the action has no
diff --git a/config/i18n.js b/config/i18n.js
index d7aec59ab99..dc417ae08d2 100644
--- a/config/i18n.js
+++ b/config/i18n.js
@@ -8,6 +8,9 @@
// accident. `ordinaryPages` records whether the locale's non-docs pages are
// translated; when they are not, a link to one has to target the default
// locale instead of a locale-prefixed page that would only repeat English.
+const fs = require('node:fs');
+const path = require('node:path');
+
const localeRegistry = [
{code: 'en', label: 'English', htmlLang: 'en-US', optional: false,
ordinaryPages: true},
{code: 'zh-CN', label: '中文', htmlLang: 'zh-Hans-CN', optional: false,
ordinaryPages: true},
@@ -23,4 +26,21 @@ const isEnabled = locale => !locale.optional
|| selectedLocales().includes(locale.code) || process.env[flagFor(locale)]
=== 'true';
const enabledLocaleRecords = () => localeRegistry.filter(isEnabled);
const enabledLocales = () => enabledLocaleRecords().map(locale => locale.code);
-module.exports = {locales, localeRegistry, enabledLocaleRecords,
enabledLocales};
+
+// Blog posts are the one content type without per-document translation state:
+// the blog plugin falls back per file, and a locale that holds no translated
+// post serves the default locale's articles at its own blog URLs. Those routes
+// are a second copy of every article, so the render path keeps them out of the
+// index, and this decides which locale that applies to. A locale leaves the
+// list as soon as its blog directory holds a translated post file.
+const blogTranslationDir = (root, locale) => path.join(root, 'i18n', locale,
'docusaurus-plugin-content-blog');
+function hasBlogTranslations(root, locale) {
+ try {
+ return fs.readdirSync(blogTranslationDir(root, locale), {recursive:
true, withFileTypes: true})
+ .some(entry => entry.isFile() && /\.mdx?$/i.test(entry.name));
+ } catch {
+ return false;
+ }
+}
+
+module.exports = {locales, localeRegistry, enabledLocaleRecords,
enabledLocales, hasBlogTranslations};
diff --git a/docusaurus.config.js b/docusaurus.config.js
index d6154869592..497c33b551c 100644
--- a/docusaurus.config.js
+++ b/docusaurus.config.js
@@ -11,13 +11,29 @@ const path = require('path');
const translationDocsPlugin = require('./scripts/i18n/docs-plugin.cjs');
const translationStatusPlugin = require('./scripts/i18n/status-plugin.cjs');
const searchPlugin = require('./scripts/i18n/search-plugin.cjs');
-const {enabledLocales, enabledLocaleRecords} = require('./config/i18n');
+const {enabledLocales, enabledLocaleRecords, hasBlogTranslations} =
require('./config/i18n');
const {managedLocales} = require('./scripts/i18n/state.cjs');
+const DEFAULT_LOCALE = 'en';
const ENABLED_LOCALES = enabledLocales();
// Locales with translation state that are enabled in this build. Empty
whenever
// no localized content is being served, which disables the whole render path.
const MANAGED_LOCALES = managedLocales(__dirname, ENABLED_LOCALES);
+// Locales whose blog is entirely the default locale's articles, see
+// hasBlogTranslations in config/i18n.js. The render path keeps those routes
+// out of the index.
+const DEFAULT_LOCALE_BLOG_LOCALES = ENABLED_LOCALES
+ .filter(locale => locale !== DEFAULT_LOCALE &&
!hasBlogTranslations(__dirname, locale));
+
+// Utility routes that are never content: /search renders a noindex search page
+// in every locale, and /slack is a tracking redirect. The sitemap filter below
+// drops them by exact path, so the set follows the enabled locales instead of
+// naming a language.
+const UTILITY_PATHS = new Set(ENABLED_LOCALES.flatMap(locale => {
+ const prefix = locale === DEFAULT_LOCALE ? '' : `/${locale}`;
+ return [`${prefix}/search`, `${prefix}/slack`];
+}));
+
const BRAND_THEME_BOOTSTRAP = `(function () {
var theme = 'doris';
try {
@@ -288,7 +304,7 @@ const config = {
},
trailingSlash: true,
i18n: {
- defaultLocale: 'en',
+ defaultLocale: DEFAULT_LOCALE,
locales: ENABLED_LOCALES,
localeConfigs: Object.fromEntries(enabledLocaleRecords()
.map(({code, label, htmlLang}) => [code, {label, htmlLang}])),
@@ -349,10 +365,12 @@ const config = {
// than naming a language. `translatedLocales` carry per-document
// translation state and render its notices; `defaultLocalePageLocales`
// show ordinary pages from the default locale, so a link to one must
- // target it directly.
+ // target it directly; `defaultLocaleBlogLocales` show the default
+ // locale's blog, so those routes must not be indexed on their own.
translatedLocales: MANAGED_LOCALES,
defaultLocalePageLocales: enabledLocaleRecords()
.filter(locale => locale.ordinaryPages === false).map(locale =>
locale.code),
+ defaultLocaleBlogLocales: DEFAULT_LOCALE_BLOG_LOCALES,
},
future: {
experimental_faster: true,
@@ -499,16 +517,17 @@ const config = {
changefreq: 'weekly',
priority: 0.5,
filename: 'sitemap.xml',
+ // Last modification is the only field Google uses from a
+ // sitemap for crawl scheduling; it reads the date from the
+ // file's last commit. Deployments check out full history
+ // (fetch-depth: 0) so the dates are real.
+ lastmod: 'date',
createSitemapItems: async params => {
const { defaultCreateSitemapItems, ...rest } = params;
const items = await defaultCreateSitemapItems(rest);
const filteredItems = items.filter(item => {
const pathname = new
URL(item.url).pathname.replace(/\/+$/, '');
- // /slack is a tracking redirect, not content.
- if (['/search', '/zh-CN/search', '/slack',
'/zh-CN/slack'].includes(pathname)) {
- return false;
- }
- return true;
+ return !UTILITY_PATHS.has(pathname);
});
for (let item of filteredItems) {
if (item.url.includes('docs')) {
diff --git a/package.json b/package.json
index e5e42a44cd9..4cbd0d79f91 100644
--- a/package.json
+++ b/package.json
@@ -71,6 +71,7 @@
"react-github-button": "^0.1.11",
"react-markdown": "^10.1.0",
"rehype-katex": "7",
+ "remark-directive": "^3.0.0",
"remark-gfm": "^4.0.1",
"remark-math": "6",
"sass": "^1.63.2",
diff --git a/scripts/docs-governance/__tests__/docs-governance.test.js
b/scripts/docs-governance/__tests__/docs-governance.test.js
index 4dc131e7464..367e2115114 100644
--- a/scripts/docs-governance/__tests__/docs-governance.test.js
+++ b/scripts/docs-governance/__tests__/docs-governance.test.js
@@ -407,6 +407,35 @@ test('lintSeo checks static robots, llms, and sitemap
policy drafts', () => {
assert.ok(!findings.some((finding) => finding.rule ===
'seo-sitemap-search-exclusion'));
});
+test('lintSeo accepts search exclusions derived from the enabled locales', ()
=> {
+ const searchExclusionFindings = (configLines) =>
+ withTempFixture(
+ (rootDir) => {
+ fs.writeFileSync(path.join(rootDir, 'docusaurus.config.js'),
[...configLines, ''].join('\n'), 'utf8');
+ },
+ (rootDir) => lintSeo({ rootDir, manifest: buildManifest({ rootDir }) }),
+ ).filter((finding) => finding.rule === 'seo-sitemap-search-exclusion');
+
+ assert.deepEqual(
+ searchExclusionFindings([
+ 'const UTILITY_PATHS = new Set(ENABLED_LOCALES.flatMap(locale => {',
+ " const prefix = locale === DEFAULT_LOCALE ? '' : `/${locale}`;",
+ ' return [`${prefix}/search`, `${prefix}/slack`];',
+ '}));',
+ "module.exports = { presets: [[ 'classic', { sitemap: { filename:
'sitemap.xml' } } ]] };",
+ ]),
+ [],
+ );
+ // Excluding only the default locale's search page leaves the localized ones.
+ assert.equal(
+ searchExclusionFindings([
+ "const UTILITY_PATHS = new Set(['/search', '/slack']);",
+ "module.exports = { presets: [[ 'classic', { sitemap: { filename:
'sitemap.xml' } } ]] };",
+ ]).length,
+ 1,
+ );
+});
+
test('filterSeoFindings keeps changed page findings without full-site SEO
noise', () => {
const findings = [
{ rule: 'seo-description-length', path: 'docs/changed.md', message:
'Changed page.' },
diff --git a/scripts/docs-governance/lint-seo.js
b/scripts/docs-governance/lint-seo.js
index 07701302d2a..5067d5a55ef 100644
--- a/scripts/docs-governance/lint-seo.js
+++ b/scripts/docs-governance/lint-seo.js
@@ -288,7 +288,10 @@ function lintSitemap(rootDir, manifest) {
makeFinding('warning', 'seo-sitemap-filename', sourcePath, 1,
'Docusaurus sitemap should emit sitemap.xml.'),
);
}
- if (!/\/search/.test(raw) || !/\/zh-CN\/search/.test(raw)) {
+ // Localized search routes are either listed one by one or derived from the
+ // enabled locales through a locale prefix, as in `${prefix}/search`.
+ const excludesLocalizedSearch = /\/zh-CN\/search/.test(raw) ||
/\$\{[^}]+\}\/search/.test(raw);
+ if (!/\/search/.test(raw) || !excludesLocalizedSearch) {
findings.push(
makeFinding(
'warning',
diff --git a/scripts/i18n/__tests__/heading-ids.test.cjs
b/scripts/i18n/__tests__/heading-ids.test.cjs
new file mode 100644
index 00000000000..1c19881c88f
--- /dev/null
+++ b/scripts/i18n/__tests__/heading-ids.test.cjs
@@ -0,0 +1,70 @@
+const test = require('node:test');
+const assert = require('node:assert/strict');
+const path = require('node:path');
+const matter = require('gray-matter');
+const {createProcessorUncached} =
require('@docusaurus/mdx-loader/lib/processor');
+const {getFormat} = require('@docusaurus/mdx-loader/lib/format');
+const {escapeMarkdownHeadingIds} = require('@docusaurus/utils');
+const {structure, alignHeadings} = require('../structure.cjs');
+const {protectionSignature} = require('../protection.cjs');
+
+const root = path.resolve(__dirname, '../../..');
+
+// Compile through Docusaurus itself: comparing two calls to structure() would
+// miss a parser difference shared by source and translation, as the deploy
did.
+async function nativeHeadingIds(raw, extension) {
+ const {visit} = await import('unist-util-visit');
+ const {data, content} = matter(raw);
+ const format = getFormat({filePath: `doc${extension}`, frontMatterFormat:
data.mdx?.format,
+ markdownConfigFormat: 'detect'});
+ const ids = [];
+ const processor = await createProcessorUncached({format, options: {
+ siteDir: root, staticDirs: [path.join(root, 'static')], admonitions:
true,
+ markdownConfig: {format: 'detect', anchors: {maintainCase: false},
mermaid: false,
+ mdx1Compat: {comments: true}},
+ rehypePlugins: [() => tree => {
+ visit(tree, 'element', node => {
+ if (/^h[1-6]$/.test(node.tagName)) ids.push(node.properties.id
?? '');
+ });
+ }],
+ }});
+ await processor.process({content: format === 'mdx' ?
escapeMarkdownHeadingIds(content) : content,
+ filePath: path.join(root, `docs/heading-fixture${extension}`),
frontMatter: data, compilerName: 'server'});
+ return ids;
+}
+
+for (const extension of ['.md', '.mdx']) {
+ test(`${extension} heading alignment matches native directives, duplicate
IDs and explicit IDs`, async () => {
+ const source = '## SSL connection error:protocol version mismatch\n\n'
+ + '## SSL connection error:protocol version mismatch\n\n'
+ + '## See :abbr[SQL] {#CaseID}\n\n'
+ + ':::note\n## Nested heading\n:::\n\n```md\n## Not a
heading\n```\n';
+ const target = '## SSL 接続エラー: protocol version mismatch\n\n'
+ + '## SSL 接続エラー: protocol version mismatch\n\n'
+ + '## :abbr[SQL] を参照\n\n'
+ + ':::note\n## 入れ子の見出し\n:::\n\n```md\n## Not a heading\n```\n';
+ const parsed = await structure(source, extension);
+ const expected = await nativeHeadingIds(source, extension);
+ assert.deepEqual(expected, ['ssl-connection-error-version-mismatch',
+ 'ssl-connection-error-version-mismatch-1', 'CaseID',
'nested-heading']);
+ assert.deepEqual(parsed.headings.map(heading => heading.id), expected);
+ const aligned = await alignHeadings(parsed, target, extension);
+ assert.deepEqual(await nativeHeadingIds(aligned, extension), expected);
+ assert.equal(protectionSignature(await structure(aligned, extension)),
protectionSignature(parsed));
+ assert(aligned.includes('```md\n## Not a heading\n```'));
+ });
+}
+
+test('BI FAQ reproducer preserves its legacy fingerprint and native English
anchor', async () => {
+ const source = '### Q2. SSL connection error:protocol version mismatch
Failed to connect to the MySQL server\n';
+ const target = '### Q2. SSL 接続エラー: protocol version mismatch Failed to
connect to the MySQL server\n';
+ const parsed = await structure(source, '.md');
+ assert.equal(parsed.technicalHash,
'09f2b78fdc50b79811dc2e8c696e4e38ceb2fb060f36c45219ac232eb0f076b9');
+ assert.equal(parsed.semanticHash,
'0cf300eea389c68ece856262d11a140d0213d6d6111cf79497785c4e6a858b76');
+ const expected = await nativeHeadingIds(source, '.md');
+ assert.deepEqual(expected,
['q2-ssl-connection-error-version-mismatch-failed-to-connect-to-the-mysql-server']);
+ assert.deepEqual(parsed.headings.map(heading => heading.id), expected);
+ const aligned = await alignHeadings(parsed, target, '.md');
+ assert.deepEqual(await nativeHeadingIds(aligned, '.md'), expected);
+ assert.equal(protectionSignature(await structure(aligned, '.md')),
protectionSignature(parsed));
+});
diff --git a/scripts/i18n/__tests__/locale-registry.test.cjs
b/scripts/i18n/__tests__/locale-registry.test.cjs
index 6942f34a0a2..3571711146b 100644
--- a/scripts/i18n/__tests__/locale-registry.test.cjs
+++ b/scripts/i18n/__tests__/locale-registry.test.cjs
@@ -82,3 +82,29 @@ test('route groups span every managed locale while a
manifest carries one', asyn
assert.equal(manifest.reports.filter(report => report.kind ===
'source_deleted').length, locale === 'ja' ? 1 : 0);
}
});
+
+test('a blog counts as translated only when the locale holds a post file', t
=> {
+ const {hasBlogTranslations} = load();
+ const siteDir = fs.mkdtempSync(path.join(os.tmpdir(),
'doris-blog-translations-'));
+ t.after(() => fs.rmSync(siteDir, {recursive: true, force: true}));
+ const blogDir = path.join(siteDir, 'i18n', 'ja',
'docusaurus-plugin-content-blog');
+ // No directory at all: the plugin serves the default locale's posts.
+ assert.equal(hasBlogTranslations(siteDir, 'ja'), false);
+ fs.mkdirSync(blogDir, {recursive: true});
+ // An empty directory, and one holding only the tag registry, translate
nothing.
+ assert.equal(hasBlogTranslations(siteDir, 'ja'), false);
+ fs.writeFileSync(path.join(blogDir, 'tags.yml'), 'release-notes:\n');
+ assert.equal(hasBlogTranslations(siteDir, 'ja'), false);
+ fs.writeFileSync(path.join(blogDir, 'some-post.md'), '# 記事\n');
+ assert.equal(hasBlogTranslations(siteDir, 'ja'), true);
+});
+
+test('neither shipped locale translates the blog yet', () => {
+ // Guards the intent behind the noindex on localized blog routes: the
+ // Chinese translations were removed on purpose, and Japanese has not
+ // started. Translating a post flips that locale over to indexing, so this
+ // test fails on purpose and asks for the SEO decision to be revisited.
+ const {hasBlogTranslations} = load();
+ assert.equal(hasBlogTranslations(root, 'zh-CN'), false);
+ assert.equal(hasBlogTranslations(root, 'ja'), false);
+});
diff --git a/scripts/i18n/__tests__/search-contexts.test.cjs
b/scripts/i18n/__tests__/search-contexts.test.cjs
new file mode 100644
index 00000000000..c887fba3859
--- /dev/null
+++ b/scripts/i18n/__tests__/search-contexts.test.cjs
@@ -0,0 +1,102 @@
+const test = require('node:test');
+const assert = require('node:assert/strict');
+const fs = require('node:fs');
+const path = require('node:path');
+const ts = require('typescript');
+const {indexGroups} = require('../search-plugin.cjs');
+
+const root = path.resolve(__dirname, '../../..');
+const pluginClient =
path.dirname(require.resolve('@yang1666204/docusaurus-search-local/dist/client/client/utils/sortSearchResults.js'));
+
+// Transpile the ES module sources the browser bundle uses, including the
+// installed plugin's own sort, so the merge is checked against the worker's.
+function load(file, imports = {}) {
+ const {outputText} = ts.transpileModule(fs.readFileSync(file, 'utf8'), {
+ compilerOptions: {module: ts.ModuleKind.CommonJS, target:
ts.ScriptTarget.ES2020, esModuleInterop: true},
+ });
+ const module = {exports: {}};
+ const resolve = name => {
+ if (Object.hasOwn(imports, name)) return imports[name];
+ if (name.startsWith('.')) return
load(require.resolve(path.resolve(path.dirname(file), name)));
+ const resolved = require.resolve(name);
+ return resolved.startsWith(path.dirname(path.dirname(pluginClient))) ?
load(resolved) : require(name);
+ };
+ new Function('require', 'module', 'exports', outputText)(resolve, module,
module.exports);
+ return module.exports;
+}
+
+const plugin = name => load(path.join(pluginClient, `${name}.js`));
+const japanese = {
+ searchContextByPaths: ['docs/dev', 'docs/4.x', 'community'].map(value =>
({path: value, label: {ja: value}})),
+ useAllContextsWithNoSearchContext: true,
+};
+const english = {searchContextByPaths: ['docs', 'course'],
useAllContextsWithNoSearchContext: false};
+const searchContexts = generated => load(path.join(root,
'src/utils/searchContexts.js'), {'./proxiedGenerated': generated});
+
+// Raw worker hits for one context: a page title and two sections of each page.
+function hits(prefix, scores) {
+ return scores.flatMap(([title, ...sections], i) => {
+ const page = {i: `${prefix}${i}`, t: `${prefix} page ${i}`, u:
`/ja/${prefix}/${i}/`};
+ return [{document: page, type: 0, score: title, tokens: ['sql']},
+ ...sections.map((score, j) => ({document: {i:
`${prefix}${i}-${j}`, p: page.i, u: page.u},
+ type: 4, page, score, tokens: ['sql']}))];
+ });
+}
+// What the worker returns: its own ranking applied to fresh result objects.
+function ranked(results) {
+ const copy = results.map(result => ({...result}));
+ plugin('sortSearchResults').sortSearchResults(copy);
+
plugin('processTreeStatusOfSearchResults').processTreeStatusOfSearchResults(copy);
+ return copy;
+}
+const shape = results => results.map(({document, type, score, isInterOfTree,
isLastOfTree}) =>
+ ({id: document.i, type, score, tree: isInterOfTree ? 'inter' :
isLastOfTree ? 'last' : ''}));
+
+test('a page outside every context searches each context index, not a union',
async () => {
+ const {indexContexts, fetchContextIndexes, searchAcrossContexts} =
searchContexts(japanese);
+ assert.deepEqual(indexContexts(''), ['docs/dev', 'docs/4.x', 'community']);
+ assert.deepEqual(indexContexts('docs/4.x'), ['docs/4.x']);
+ const fetched = [];
+ await fetchContextIndexes(async (...args) => {fetched.push(args);},
'/ja/', '');
+ assert.deepEqual(fetched, [['/ja/', 'docs/dev'], ['/ja/', 'docs/4.x'],
['/ja/', 'community']]);
+ const calls = [];
+ const search = searchAcrossContexts(async (...args) => {calls.push(args);
return [];});
+ await search('/ja/', 'community', 'sql', 10);
+ assert.deepEqual(calls, [['/ja/', 'community', 'sql', 10]]);
+ calls.length = 0;
+ await search('/ja/', '', 'sql', 10);
+ assert.deepEqual(calls.map(call => call[1]), ['docs/dev', 'docs/4.x',
'community']);
+});
+
+test('without useAllContextsWithNoSearchContext the root index stays the only
one', async () => {
+ const {indexContexts, searchAcrossContexts} = searchContexts(english);
+ assert.deepEqual(indexContexts(''), ['']);
+ const calls = [];
+ await searchAcrossContexts(async (...args) => {calls.push(args); return
[];})('/', '', 'sql', 10);
+ assert.deepEqual(calls, [['/', '', 'sql', 10]]);
+});
+
+test('merged context results rank like one index over all of their hits', ()
=> {
+ const {mergeSearchResults} = searchContexts(japanese);
+ const dev = hits('dev', [[3, 9, 1], [8, 2]]);
+ const community = hits('community', [[6, 7], [1, 10, 4]]);
+ const merged = mergeSearchResults([ranked(dev), ranked(community)], 100);
+ // Titles keep the single boost the worker applies, and every section still
+ // follows its own page title with correct tree markers.
+ assert.deepEqual(shape(merged), shape(ranked([...dev, ...community])));
+ assert.equal(merged.find(result => result.document.i === 'dev1').score,
12);
+ const limited = mergeSearchResults([ranked(dev), ranked(community)], 4);
+ assert.deepEqual(shape(limited), shape(ranked([...dev,
...community]).slice(0, 4)).map((item, i, all) =>
+ i === all.length - 1 && item.tree === 'inter' ? {...item, tree:
'last'} : item));
+});
+
+test('the localized root index holds only documents outside every context', ()
=> {
+ const doc = (u, i) => ({i, u, t: u});
+ const all = [[doc('/ja/docs/dev/a/', 1), doc('/ja/docs/4.x/a/', 2),
doc('/ja/community/a/', 3)], []];
+ const selected = new Map([['/ja/docs/dev/a', 'docs/dev'],
['/ja/docs/4.x/a', 'docs/4.x'], ['/ja/community/a', 'community']]);
+ const groups = indexGroups(all, selected);
+ assert.deepEqual([...groups.keys()], ['docs/dev', 'docs/4.x', 'community',
'']);
+ assert.deepEqual(groups.get('docs/4.x'), [[doc('/ja/docs/4.x/a/', 2)],
[]]);
+ assert.deepEqual(groups.get(''), [[], []]);
+ assert.deepEqual(indexGroups([[doc('/ja/other/', 4)]], selected).get(''),
[[doc('/ja/other/', 4)]]);
+});
diff --git a/scripts/i18n/__tests__/search-input.test.cjs
b/scripts/i18n/__tests__/search-input.test.cjs
new file mode 100644
index 00000000000..0e9c4717331
--- /dev/null
+++ b/scripts/i18n/__tests__/search-input.test.cjs
@@ -0,0 +1,71 @@
+const test = require('node:test');
+const assert = require('node:assert/strict');
+const fs = require('node:fs');
+const os = require('node:os');
+const path = require('node:path');
+const ts = require('typescript');
+const {generate} =
require('@yang1666204/docusaurus-search-local/dist/server/server/utils/generate');
+
+const root = path.resolve(__dirname, '../../..');
+
+function load(file, imports = {}) {
+ const {outputText} = ts.transpileModule(fs.readFileSync(file, 'utf8'), {
+ compilerOptions: {module: ts.ModuleKind.CommonJS, target:
ts.ScriptTarget.ES2020, esModuleInterop: true},
+ });
+ const module = {exports: {}};
+ const resolve = name => {
+ if (Object.hasOwn(imports, name)) return imports[name];
+ if (name.startsWith('.')) return
load(require.resolve(path.resolve(path.dirname(file), name)));
+ if (name.endsWith('/client/shared/lunrLanguageZh.js')) return
load(name);
+ return require(name);
+ };
+ new Function('require', 'module', 'exports', outputText)(resolve, module,
module.exports);
+ return module.exports;
+}
+
+function searchInput(t, language) {
+ const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'doris-search-input-'));
+ t.after(() => fs.rmSync(dir, {recursive: true, force: true}));
+ // Use the installed plugin's real exports so importing a constant from
+ // generated.js instead of generated-constants.js fails as it does online.
+ generate({language, hashed: false}, dir);
+ const generatedPath =
'@generated/@yang1666204/docusaurus-search-local/default/';
+ const generated = load(path.join(root, 'src/utils/proxiedGenerated.ts'), {
+ [`${generatedPath}generated.js`]: load(path.join(dir, 'generated.js')),
+ });
+ const constants = load(path.join(root,
'src/utils/proxiedGeneratedConstants.ts'), {
+ [`${generatedPath}generated-constants.js`]: load(path.join(dir,
'generated-constants.js')),
+ });
+ return load(path.join(root, 'src/utils/searchInput.js'), {
+ './tokenize': load(path.join(root, 'src/utils/tokenize.ts')),
+ './proxiedGenerated': generated,
+ './proxiedGeneratedConstants': constants,
+ }).searchWithLocaleFallback;
+}
+
+test('English and Chinese search use the generated language constants', async
t => {
+ const search = searchInput(t, ['en', 'zh']);
+ const calls = [], results = [{document: {t: 'Inverted Index'}}];
+ const worker = async (...args) => {calls.push(args); return results;};
+ assert.equal(await search(worker, '/docs/dev/', 'docs', 'inverted_index',
100), results);
+ assert.deepEqual(calls, [['/docs/dev/', 'docs', 'inverted_index', 100]]);
+});
+
+test('Japanese search retries SQL identifier parts only after an empty
normalized query', async t => {
+ const search = searchInput(t, ['ja']);
+ const calls = [], results = [{document: {t: 'query_vector'}}];
+ const worker = async (...args) => {
+ calls.push(args);
+ return args[2] === 'vector' ? results : [];
+ };
+ assert.equal(await search(worker, '/ja/', 'docs/dev', 'query_vector',
100), results);
+ assert.deepEqual(calls, [['/ja/', 'docs/dev', 'query vector', 100],
['/ja/', 'docs/dev', 'vector', 100]]);
+});
+
+test('Japanese search keeps a successful normalized result without retrying',
async t => {
+ const search = searchInput(t, ['ja']);
+ const calls = [], results = [{document: {t: 'query_vector'}}];
+ const worker = async (...args) => {calls.push(args); return results;};
+ assert.equal(await search(worker, '/ja/', 'docs/4.x', 'query-vector', 10),
results);
+ assert.deepEqual(calls, [['/ja/', 'docs/4.x', 'query vector', 10]]);
+});
diff --git a/scripts/i18n/search-plugin.cjs b/scripts/i18n/search-plugin.cjs
index 0cb0014e9d0..1b3a8e3faa1 100644
--- a/scripts/i18n/search-plugin.cjs
+++ b/scripts/i18n/search-plugin.cjs
@@ -26,8 +26,20 @@ function localizedOptions(options, locale, labels) {
docsRouteBasePath:['docs','community',`${locale}/docs`,`${locale}/community`],
// A locale without configured labels falls back to the context path.
searchContextByPaths:contexts.map(value =>
({path:value,label:{[locale]:labels?.[value] ?? value}})),
+ // Outside every context, search all of them. The browser queries each
+ // context index for that (src/utils/searchContexts.js); see
indexGroups.
useAllContextsWithNoSearchContext:true, ignoreFiles:[]};
}
+// One index per context, plus the root index for documents outside every
+// context. The root does not repeat the contexts the way the native plugin's
+// union does: that copy held every context a second time, and the Japanese one
+// outgrew GitHub's 100 MiB file limit, which rejected the deployment.
+function indexGroups(allDocuments, selected) {
+ const groups = new Map(contexts.map(group => [group,
+ allDocuments.map(documents => documents.filter(doc =>
selected.get(normalize(doc.u)) === group))]));
+ groups.set('', allDocuments.map(documents => documents.filter(doc =>
!selected.has(normalize(doc.u)))));
+ return groups;
+}
function targetFingerprint(root, locale) {
const digester = createHash('sha256');
function scan(dir) {
@@ -76,8 +88,7 @@ async function buildLocalizedIndex(buildData, config,
filename, locale) {
const name = filename.replace('{dir}', group ?
`-${group.replace(/\//g,'-')}` : '');
fs.writeFileSync(path.join(buildData.outDir,name),
JSON.stringify(buildIndex(documents,config)));
};
- for (const group of contexts) write(group, allDocuments.map(documents =>
documents.filter(doc => selected.get(normalize(doc.u)) === group)));
- write('',allDocuments);
+ for (const [group, documents] of indexGroups(allDocuments, selected))
write(group, documents);
fs.writeFileSync(path.join(buildData.outDir,`${locale}-search-summary.json`),
JSON.stringify({schemaVersion:1,locale,
indexedPages:selected.size,
contexts:Object.fromEntries(contexts.map(group =>
[group,[...selected.values()].filter(value => value ===
group).length]))})+'\n');
}
@@ -105,4 +116,5 @@ module.exports.validateOptions =
nativeModule.validateOptions;
module.exports.localizedOptions = localizedOptions;
module.exports.groupFor = groupFor;
module.exports.localizedPages = localizedPages;
+module.exports.indexGroups = indexGroups;
module.exports.buildLocalizedIndex = buildLocalizedIndex;
diff --git a/scripts/i18n/structure.cjs b/scripts/i18n/structure.cjs
index 198ec2ce516..d6c3dfeb832 100644
--- a/scripts/i18n/structure.cjs
+++ b/scripts/i18n/structure.cjs
@@ -32,18 +32,22 @@ async function structure(raw, extension,
anchorsMaintainCase = false) {
}
offsets[parseContent.length] = content.length;
const tree = processor.parse(parseContent);
- // Keep the raw tree for protected tokens: directive-like prose such as
- // `:s3` must retain its number even though native directive parsing hides
it.
+ // Existing state records fingerprint this original parse. Keep its IDs and
+ // protected nodes stable so a rendering fix does not invalidate
translations;
+ // directive-like prose such as `:s3` also keeps its protected number here.
+ const fingerprintHeadings = [];
await headingsPlugin({anchorsMaintainCase})(tree);
- // Docusaurus parses directives before generating heading IDs. For example,
- // the directive name in `error:protocol version` is absent from its slug.
- const headingProcessor = unified().use(parse).use(directive).use(gfm);
- if (format === 'mdx') headingProcessor.use(mdx);
- const headingTree = headingProcessor.parse(parseContent);
- const headings = [];
- // Use the installed framework's heading transform, including its duplicate
+ visit(tree, 'heading', node => {
+ fingerprintHeadings.push({depth: node.depth, id: node.data.id});
+ });
+ // Docusaurus parses directives before assigning heading IDs. In
particular,
+ // `error:protocol` contains a textDirective whose name is absent from the
ID.
+ // Parse the original content again to preserve its raw heading offsets,
and
+ // use the installed framework's heading transform, including its duplicate
// slug order and explicit-ID behavior, instead of a second slug algorithm.
+ const headingTree = processor().use(directive).parse(parseContent);
await headingsPlugin({anchorsMaintainCase})(headingTree);
+ const headings = [];
visit(headingTree, 'heading', node => {
headings.push({depth: node.depth, id: node.data.id, position: {
start: {...node.position.start, offset:
offsets[node.position.start.offset]},
@@ -71,7 +75,7 @@ async function structure(raw, extension, anchorsMaintainCase
= false) {
.sort(([a], [b]) => a.localeCompare(b)));
return {
content, headings, format, protectedNodes, technicalMetadata,
- technicalHash: hash(JSON.stringify([technicalMetadata, protectedNodes,
headings.map(({depth, id}) => ({depth, id}))])),
+ technicalHash: hash(JSON.stringify([technicalMetadata, protectedNodes,
fingerprintHeadings])),
// A secondary comparison for planning only. The state sourceHash stays
// byte-sensitive, so a formatting edit still needs an explicit review.
semanticHash:
hash(JSON.stringify([Object.fromEntries(Object.entries(data).sort(([a], [b]) =>
a.localeCompare(b))),
diff --git a/scripts/i18n/verify-fixture.cjs b/scripts/i18n/verify-fixture.cjs
index 36ab00d0da5..6646e51454d 100644
--- a/scripts/i18n/verify-fixture.cjs
+++ b/scripts/i18n/verify-fixture.cjs
@@ -9,8 +9,9 @@ const {structure} = require('./structure.cjs');
const {verify} = require('./verify-static.cjs');
const root = path.resolve(__dirname, '../..');
const fixture = path.join(root, '.i18n-build/fixture');
-const sourceBase = '---\ntitle: Example\nlast_update:\n date:
2026-01-01\n---\n## Usage\n\nUse Doris.\n\n```sql\nSELECT 1;\n```\n\n[Current
translation](./machine.md)\n';
+const sourceBase = '---\ntitle: Example\nlast_update:\n date:
2026-01-01\n---\n## Usage\n\nUse Doris.\n\n### SSL error:protocol version
mismatch\n\n```sql\nSELECT 1;\n```\n\n[Current translation](./machine.md)\n';
const japanese = sourceBase.replace('title: Example', 'title: 例').replace('##
Usage', '## 使用方法')
+ .replace('### SSL error:protocol version mismatch', '### SSL 接続エラー:
protocol version mismatch')
.replace('Use Doris.', 'Doris を使用します。').replace('date: 2026-01-01', 'date:
2026-10-11');
async function write(file, content) {
diff --git a/scripts/i18n/verify-search.cjs b/scripts/i18n/verify-search.cjs
index 216a2927a5b..3f168e6e826 100644
--- a/scripts/i18n/verify-search.cjs
+++ b/scripts/i18n/verify-search.cjs
@@ -10,6 +10,9 @@ async function verify({site,devOnly = false}) {
assert.equal(manifest.indexedPages,Object.values(manifest.contexts).reduce((a,b)=>a+b,0));
const indexFiles =
['search-index.json','search-index-docs-dev.json','search-index-docs-4.x.json','search-index-community.json'];
const indexes = Object.fromEntries(indexFiles.map(file =>
[file,JSON.parse(fs.readFileSync(path.join(site,'ja',file),'utf8'))]));
+ // "Everywhere" searches each context index in the browser; a union copy at
+ // the root outgrew GitHub's 100 MiB file limit.
+ assert(indexes['search-index.json'].every(section =>
section.documents.length === 0),'Root index repeats the contexts');
for (const [file,index] of Object.entries(indexes)) {
for (const section of index) for (const doc of section.documents) {
assert(doc.u.startsWith('/ja/'),`Cross-locale result in ${file}:
${doc.u}`);
@@ -52,11 +55,28 @@ async function verify({site,devOnly = false}) {
catch (error) {throw new Error(`${item.ctx}: ${error.message};
requests=${requests}; errors=${errors}; body=${(await
page.locator('body').innerText()).slice(0,500)}`);}
const links = await page.locator('section article
a').evaluateAll(nodes => nodes.map(node => node.getAttribute('href')));
assert(links.length && links.every(link =>
link.startsWith(item.path)),`${item.ctx} leaked a result: ${links.join(',')}`);
- assert(requests.some(request =>
request.includes(`search-index-${item.ctx.replace(/\//g,'-')}`)),`Wrong worker
index: ${requests}`);
+ // A page searching one context loads only that context's index.
+ assert(requests.length && requests.every(request =>
request.includes(`search-index-${item.ctx.replace(/\//g,'-')}`)),`Wrong worker
index: ${requests}`);
assert.deepEqual(errors,[],`Search page errors: ${item.ctx}`);
results.push({...item,indexRequests:requests,results:links.length});
await page.close();
}
+ // Without ctx the page searches "Everywhere": every context index,
merged.
+ for (const item of
[{query:'SQL',path:'/ja/docs/dev/'},{query:'コミュニティ',path:'/ja/community/'}]) {
+ const everywhere = await browser.newPage();
+ const requests = [];
+ everywhere.on('request',request =>
{if(request.url().includes('search-index')) requests.push(new
URL(request.url()).pathname);});
+ await everywhere.goto(`${origin}/ja/search/?${new
URLSearchParams({q:item.query})}`);
+ await everywhere.locator('section article
a').first().waitFor({timeout:30000});
+ const links = await everywhere.locator('section article
a').evaluateAll(nodes => nodes.map(node => node.getAttribute('href')));
+ const expected = ['docs-dev','docs-4.x','community'].map(ctx =>
`/ja/search-index-${ctx}.json`);
+ assert(expected.every(file => requests.includes(file)),`Everywhere
skipped a context index: ${requests}`);
+ assert(!requests.includes('/ja/search-index.json'),`Everywhere
fetched the root index: ${requests}`);
+ assert(links.some(link => link.startsWith(item.path)),`Everywhere
missed ${item.path}: ${links.slice(0,10)}`);
+
results.push({everywhereQuery:item.query,results:links.length,indexRequests:requests});
+ await everywhere.close();
+ }
+
const bar = await browser.newPage();
const barRequests = [];
bar.on('request',request =>
{if(request.url().includes('search-index')) barRequests.push(new
URL(request.url()).pathname);});
@@ -67,7 +87,7 @@ async function verify({site,devOnly = false}) {
await input.fill('query_vector');
const suggestion = bar.locator('[class*="suggestion_"]');
await suggestion.first().waitFor({timeout:30000});
- assert(barRequests.some(request =>
request.includes('search-index-docs-dev')),`SearchBar used wrong index:
${barRequests}`);
+ assert(barRequests.length && barRequests.every(request =>
request.includes('search-index-docs-dev')),`SearchBar used wrong index:
${barRequests}`);
results.push({searchBarQuery:'query_vector',suggestions:await
suggestion.count(),indexRequests:barRequests});
await bar.close();
@@ -80,7 +100,9 @@ async function verify({site,devOnly = false}) {
await mobile.close();
const page = await browser.newPage();
- await
page.goto(`${origin}/ja/search/?q=zzzzzzzz_t04_nomatch_98765&ctx=docs/dev`);
+ // An identifier with no results is retried part by part, and every
part
+ // is a prefix query: "nomatch" from an earlier query found
NoMatchingRule.
+ await
page.goto(`${origin}/ja/search/?q=zzzzqqqq_xxxxwwww&ctx=docs/dev`);
try {await
page.getByRole('link',{name:'英語の原文を検索'}).waitFor({timeout:30000});}
catch (error) {throw new Error(`Missing original-language search link:
${(await page.locator('body').innerText()).slice(-600)}; ${error.message}`);}
const original = await
page.getByRole('link',{name:'英語の原文を検索'}).getAttribute('href');
diff --git a/src/theme/BlogPostPage/Metadata/index.tsx
b/src/theme/BlogPostPage/Metadata/index.tsx
index 9f751597bf5..d1690a5e639 100644
--- a/src/theme/BlogPostPage/Metadata/index.tsx
+++ b/src/theme/BlogPostPage/Metadata/index.tsx
@@ -1,19 +1,31 @@
import React from 'react';
import {PageMetadata} from '@docusaurus/theme-common';
import {useBlogPost} from '@docusaurus/plugin-content-blog/client';
+import useDocusaurusContext from '@docusaurus/useDocusaurusContext';
+import {defaultLocalePath} from '@site/src/utils/localization';
export default function BlogPostPageMetadata(): JSX.Element {
const {assets, metadata} = useBlogPost();
const {title, description, date, tags, authors, frontMatter} = metadata;
+ const {
+ i18n: {currentLocale, defaultLocale},
+ siteConfig: {url},
+ } = useDocusaurusContext();
const {keywords} = frontMatter;
const image = assets.image ?? frontMatter.image;
+ // The blog plugin falls back per file: a post without a translated
+ // counterpart is the default locale's article. It keeps the localized URL,
+ // but the default locale's copy is the one worth ranking.
+ const isDefaultLocalePost =
+ currentLocale !== defaultLocale && !(metadata.source ??
'').startsWith(`@site/i18n/${currentLocale}/`);
return (
<PageMetadata
title={frontMatter.title_meta ?? title}
description={description}
keywords={keywords}
image={image}>
+ {isDefaultLocalePost && <link rel="canonical" href={url +
defaultLocalePath(metadata.permalink)} />}
<meta property="og:type" content="article" />
<meta property="article:published_time" content={date} />
{/* TODO double check those article meta array syntaxes, see
https://ogp.me/#array */}
diff --git a/src/theme/SearchBar/SearchBar.jsx
b/src/theme/SearchBar/SearchBar.jsx
index 2a8d9c5acfe..0a3a5f2685e 100644
--- a/src/theme/SearchBar/SearchBar.jsx
+++ b/src/theme/SearchBar/SearchBar.jsx
@@ -29,8 +29,11 @@ import useIsDocPage from '@site/src/hooks/use-is-doc';
import { debounce } from '@site/src/utils/debounce';
import { getLocalePrefix, normalizePathname } from '@site/src/utils/locale';
import {searchWithLocaleFallback} from '@site/src/utils/searchInput';
+import {fetchContextIndexes, searchAcrossContexts} from
'@site/src/utils/searchContexts';
import {isTranslatedLocale} from '@site/src/utils/localization';
+const searchContextsByWorker = searchAcrossContexts(searchByWorker);
+
async function fetchAutoCompleteJS() {
const autoCompleteModule = await import('@easyops-cn/autocomplete.js');
const autoComplete = autoCompleteModule.default;
@@ -168,7 +171,7 @@ export default function SearchBar({ handleSearchBarToggle
}) {
setLoading(true);
const [autoComplete] = await Promise.all([
fetchAutoCompleteJS(),
- fetchIndexesByWorker(versionUrl, searchContext),
+ fetchContextIndexes(fetchIndexesByWorker, versionUrl,
searchContext),
]);
if (myToken !== loadToken.current) {
// Superseded by a newer loadIndex; don't mount a second
autocomplete on the same input.
@@ -268,7 +271,7 @@ export default function SearchBar({ handleSearchBarToggle
}) {
[
{
source: debounce(async (input, callback) => {
- const result = await
searchWithLocaleFallback(searchByWorker, versionUrl, searchContext, input,
searchResultLimits);
+ const result = await
searchWithLocaleFallback(searchContextsByWorker, versionUrl, searchContext,
input, searchResultLimits);
callback(result);
}, 300),
templates: {
@@ -385,6 +388,11 @@ export default function SearchBar({ handleSearchBarToggle
}) {
}
}, [location.pathname]);
useEffect(() => {
+ // On mount the page's search context is not resolved yet, and an empty
+ // one would load every context index. Leave that to focus or hover.
+ if (!searchContext && useAllContextsWithNoSearchContext) {
+ return;
+ }
loadIndex(true);
}, [curVersion]);
useEffect(() => {
diff --git a/src/theme/SearchPage/SearchPage.jsx
b/src/theme/SearchPage/SearchPage.jsx
index 5aa3561a56a..97b568dd025 100644
--- a/src/theme/SearchPage/SearchPage.jsx
+++ b/src/theme/SearchPage/SearchPage.jsx
@@ -1,5 +1,6 @@
import React, { useCallback, useEffect, useMemo, useState } from "react";
import useDocusaurusContext from "@docusaurus/useDocusaurusContext";
+import useIsBrowser from "@docusaurus/useIsBrowser";
import Layout from "@theme/Layout";
import Head from "@docusaurus/Head";
import Link from "@docusaurus/Link";
@@ -19,7 +20,9 @@ import { searchResultLimits } from
"../../utils/proxiedGeneratedConstants";
import styles from "./SearchPage.module.css";
import { normalizeContextByPath } from "../../utils/normalizeContextByPath";
import {searchWithLocaleFallback} from "../../utils/searchInput";
+import {fetchContextIndexes, searchAcrossContexts} from
"../../utils/searchContexts";
import {isTranslatedLocale} from "../../utils/localization";
+const searchContextsByWorker = searchAcrossContexts(searchByWorker);
export default function SearchPage() {
return (<Layout>
<SearchPageContent />
@@ -27,6 +30,7 @@ export default function SearchPage() {
}
function SearchPageContent() {
const { siteConfig: { baseUrl }, i18n: { currentLocale }, } =
useDocusaurusContext();
+ const isBrowser = useIsBrowser();
const { selectMessage } = usePluralForm();
const { searchValue, searchContext, searchVersion, updateSearchPath,
updateSearchContext, } = useSearchQuery();
const [searchQuery, setSearchQuery] = useState(searchValue);
@@ -49,7 +53,7 @@ function SearchPageContent() {
updateSearchPath(searchQuery);
if (searchQuery) {
(async () => {
- const results = await searchWithLocaleFallback(searchByWorker,
versionUrl, searchContext, searchQuery, searchResultLimits);
+ const results = await
searchWithLocaleFallback(searchContextsByWorker, versionUrl, searchContext,
searchQuery, searchResultLimits);
setSearchResults(results);
})();
}
@@ -80,19 +84,23 @@ function SearchPageContent() {
if (!Array.isArray(searchContextByPaths) ||
searchContext ||
useAllContextsWithNoSearchContext) {
- await fetchIndexesByWorker(versionUrl, searchContext);
+ await fetchContextIndexes(fetchIndexesByWorker, versionUrl,
searchContext);
}
setSearchWorkerReady(true);
}
- doFetchIndexes();
- }, [searchContext, versionUrl]);
+ // The context comes from the query string, which is only read in the
+ // browser; before that an empty one would load every context index.
+ if (isBrowser) {
+ doFetchIndexes();
+ }
+ }, [isBrowser, searchContext, versionUrl]);
return (<React.Fragment>
<Head>
{/*
We should not index search pages
See https://github.com/facebook/docusaurus/pull/3233
*/}
- <meta property="robots" content="noindex, follow"/>
+ <meta name="robots" content="noindex, follow"/>
<title>{pageTitle}</title>
</Head>
diff --git a/src/theme/SiteMetadata/index.tsx b/src/theme/SiteMetadata/index.tsx
index 148bf59d96e..e34214e5a03 100644
--- a/src/theme/SiteMetadata/index.tsx
+++ b/src/theme/SiteMetadata/index.tsx
@@ -8,6 +8,7 @@
import React, {type JSX} from 'react';
import Head from '@docusaurus/Head';
import {useTranslationRoutes} from '@site/src/utils/translation';
+import {isDefaultLocaleBlogRoute} from '@site/src/utils/localization';
import useDocusaurusContext from '@docusaurus/useDocusaurusContext';
import useBaseUrl from '@docusaurus/useBaseUrl';
import {PageMetadata, useThemeConfig} from '@docusaurus/theme-common';
@@ -124,6 +125,7 @@ export default function SiteMetadata(): JSX.Element {
const {
i18n: {currentLocale},
} = useDocusaurusContext();
+ const {pathname} = useLocation();
// TODO maybe move these 2 themeConfig to siteConfig?
// These seems useful for other themes as well
@@ -142,6 +144,18 @@ export default function SiteMetadata(): JSX.Element {
<CanonicalUrlHeaders />
+ {/* A locale without a translated blog serves the default locale's
+ articles under its own /blog routes. The section stays browsable, but
+ it is a second copy of /blog, so it stays out of the index — and out
+ of that locale's sitemap, which skips noindex routes. The rule lives
+ here rather than in a blog component because the archive renders
+ through the plain layout. */}
+ {isDefaultLocaleBlogRoute(pathname) && (
+ <Head>
+ <meta name="robots" content="noindex, follow" />
+ </Head>
+ )}
+
<AlternateLangHeaders />
<SearchMetadata tag={DEFAULT_SEARCH_TAG} locale={currentLocale} />
diff --git a/src/utils/localization.ts b/src/utils/localization.ts
index 0bcb5e5587b..76fad7ba0f7 100644
--- a/src/utils/localization.ts
+++ b/src/utils/localization.ts
@@ -1,15 +1,19 @@
import i18n from '@generated/i18n';
import siteConfig from '@generated/docusaurus.config';
import { translate } from '@docusaurus/Translate';
+import { applyTrailingSlash } from '@docusaurus/utils-common';
+import { normalizePathname } from './locale';
// The build writes locale facts into the site configuration, so this module
// reads configuration instead of naming a language. See config/i18n.js.
const fields = (siteConfig.customFields ?? {}) as {
translatedLocales?: string[];
defaultLocalePageLocales?: string[];
+ defaultLocaleBlogLocales?: string[];
};
const translatedLocales = fields.translatedLocales ?? [];
const defaultLocalePageLocales = fields.defaultLocalePageLocales ?? [];
+const defaultLocaleBlogLocales = fields.defaultLocaleBlogLocales ?? [];
/** A locale with per-document translation state, which renders its notices. */
export function isTranslatedLocale(locale: string): boolean {
@@ -26,6 +30,35 @@ export function usesDefaultLocalePages(): boolean {
return defaultLocalePageLocales.includes(i18n.currentLocale);
}
+/**
+ * A locale whose blog has no translated posts, so every blog route under it
+ * repeats the default locale's posts at its own URL. Those routes stay
+ * reachable but must not be indexed as a second copy of the articles.
+ */
+export function usesDefaultLocaleBlog(): boolean {
+ return defaultLocaleBlogLocales.includes(i18n.currentLocale);
+}
+
+/** Whether a route is one of those blog routes. */
+export function isDefaultLocaleBlogRoute(pathname: string): boolean {
+ if (!usesDefaultLocaleBlog()) return false;
+ // Blog routes live under the default locale's /blog, so its own prefix is
+ // what a localized route has to drop before the section can be told apart.
+ const path = normalizePathname(pathname,
[i18n.currentLocale]).replace(/\/+$/, '');
+ return path === '/blog' || path.startsWith('/blog/');
+}
+
+/**
+ * The default locale's path for a page the current locale served as a
fallback,
+ * so the page can point search engines at the copy worth ranking.
+ */
+export function defaultLocalePath(permalink: string): string {
+ const prefix = `/${i18n.currentLocale}/`;
+ const path = permalink.startsWith(prefix) ?
`/${permalink.slice(prefix.length)}` : permalink;
+ // Permalinks carry no trailing slash; the served URLs do.
+ return applyTrailingSlash(path, { trailingSlash: siteConfig.trailingSlash,
baseUrl: siteConfig.baseUrl });
+}
+
export function localizedHomeLink(href: string): string {
if (!usesDefaultLocalePages() || !href.startsWith('/')) return href;
const prefix = siteConfig.baseUrl.replace(/\/$/, '');
diff --git a/src/utils/searchContexts.js b/src/utils/searchContexts.js
new file mode 100644
index 00000000000..eeb53392d7b
--- /dev/null
+++ b/src/utils/searchContexts.js
@@ -0,0 +1,45 @@
+import {sortSearchResults} from
'@yang1666204/docusaurus-search-local/dist/client/client/utils/sortSearchResults';
+import {processTreeStatusOfSearchResults} from
'@yang1666204/docusaurus-search-local/dist/client/client/utils/processTreeStatusOfSearchResults';
+import {SearchDocumentType} from '../../shared/interfaces';
+import {searchContextByPaths, useAllContextsWithNoSearchContext} from
'./proxiedGenerated';
+
+// With useAllContextsWithNoSearchContext, a page outside every search context
+// searches all of them. The localized build writes no union of the context
+// indexes for that (see indexGroups in scripts/i18n/search-plugin.cjs), so the
+// browser queries each context index and merges the results.
+export function indexContexts(searchContext) {
+ if (searchContext || !useAllContextsWithNoSearchContext ||
!Array.isArray(searchContextByPaths)) {
+ return [searchContext];
+ }
+ return searchContextByPaths.map(item => (typeof item === 'string' ? item :
item.path));
+}
+
+export async function fetchContextIndexes(fetchIndexesByWorker, versionUrl,
searchContext) {
+ await Promise.all(indexContexts(searchContext).map(context =>
fetchIndexesByWorker(versionUrl, context)));
+}
+
+// Takes and returns the plugin's searchByWorker signature.
+export function searchAcrossContexts(searchByWorker) {
+ return async (versionUrl, searchContext, input, limit) => {
+ const contexts = indexContexts(searchContext);
+ if (contexts.length === 1) return searchByWorker(versionUrl,
contexts[0], input, limit);
+ const lists = await Promise.all(contexts.map(context =>
searchByWorker(versionUrl, context, input, limit)));
+ return mergeSearchResults(lists, limit);
+ };
+}
+
+// Each list arrives ranked and grouped by the worker, whose sort also raises
+// title scores by half. Undo that so the same sort ranks all lists together
and
+// applies it once, keeping each page's sections after its title.
+export function mergeSearchResults(lists, limit) {
+ const results = lists.flat().map(result => ({
+ ...result,
+ isInterOfTree: undefined,
+ isLastOfTree: undefined,
+ score: result.type === SearchDocumentType.Title && result.score ?
result.score / 1.5 : result.score,
+ }));
+ sortSearchResults(results);
+ const merged = results.slice(0, limit);
+ processTreeStatusOfSearchResults(merged);
+ return merged;
+}
diff --git a/src/utils/searchInput.js b/src/utils/searchInput.js
index b36408d1ea3..2f4b41313f9 100644
--- a/src/utils/searchInput.js
+++ b/src/utils/searchInput.js
@@ -1,5 +1,5 @@
import {usesSegmenterTokenizer} from './tokenize';
-import {language} from './proxiedGenerated';
+import {language} from './proxiedGeneratedConstants';
// The segmenter behind some languages emits standalone '_' and '-' tokens for
// SQL identifiers, but lunr removes those punctuation tokens from its index.
diff --git a/static/robots.txt b/static/robots.txt
index e4a626ee00a..2bd5218204d 100644
--- a/static/robots.txt
+++ b/static/robots.txt
@@ -27,4 +27,9 @@ Allow: /
User-agent: *
Allow: /
+# Every locale has its own subdirectory and its own sitemap (see
+# scripts/build-locales.cjs). Search engines only discover a sitemap through
+# this file or Search Console, so all of them are declared here.
Sitemap: https://doris.apache.org/sitemap.xml
+Sitemap: https://doris.apache.org/zh-CN/sitemap.xml
+Sitemap: https://doris.apache.org/ja/sitemap.xml
diff --git a/website-quality-governance/seo-geo-baseline.md
b/website-quality-governance/seo-geo-baseline.md
index 0dcb804e6c9..83d4e7d86a3 100644
--- a/website-quality-governance/seo-geo-baseline.md
+++ b/website-quality-governance/seo-geo-baseline.md
@@ -10,6 +10,13 @@ This runbook records the standing policy for Apache Doris
website SEO and genera
- `static/llms.txt` lists curated Overview, Getting Started, SQL Manual, Load,
Lakehouse, AI, Admin, Release Notes, and Community entry points.
- `static/llms-full-docs.txt` is intentionally a compact topic index, not a
full content dump.
+## Localized Indexing
+
+- A section that only repeats the default locale's content is kept out of the
index. A locale's blog routes are `noindex` until that locale holds a
translated post (`hasBlogTranslations` in `config/i18n.js`), and an
untranslated docs page is `noindex` with a canonical to the default locale
(`src/theme/DocItem/Metadata`).
+- The sitemap plugin skips noindex routes, so those URLs leave the locale's
sitemap as well. A localized page that should rank needs a real translation,
not an indexable English copy.
+- `static/robots.txt` declares every locale's sitemap (`/zh-CN/sitemap.xml`,
`/ja/sitemap.xml`). Each locale builds its own sitemap, and Google retired the
sitemap ping endpoint, so a new locale also needs its sitemap submitted in
Search Console.
+- Sitemaps carry `lastmod` from each file's last commit, so a deploy has to
check out full history (`fetch-depth: 0`).
+
## Structured Data Plan
- Blog pages should use `BlogPosting` schema with headline, description, date,
author, and canonical URL.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
![https://cdn.selectdb.com/static/doris_logo_512_4903556647.png"](https://cdn.selectdb.com/static/doris_logo_512_4903556647.png"){kind=link}