This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new 9a969fc33c1f [improvement](docs) Replace scheduled PDF export with a
local versioned CLI (#4558)
9a969fc33c1f is described below
commit 9a969fc33c1fa48ba09267331512f7620dd23084
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Sat Oct 31 02:31:43 2026 +0000
[improvement](docs) Replace scheduled PDF export with a local versioned CLI
(#4558)
Retire the scheduled PDF job that fails while launching Chromium and
provide a local command for exporting a selected documentation version
from the working tree. `Cron Generate Pdf` has already been disabled in
GitHub Actions; this PR removes its scheduled workflow definition.
## Changes and review
- **Goal:** Replace automatic Dev-only online crawling with an explicit
local `yarn docs:pdf --version <version>` command. Support Dev/current,
active releases, and historical source directories, with English and
Chinese selection.
- **Scope:** Retire the cron workflow, update its existing security-test
coverage, and add the CLI, executable shell wrapper, renderer, verifier,
setup guide, and shared documentation inventory. PDF binaries remain
local artifacts.
- **Information architecture:** Read the selected version's source root
and sidebar. Preserve sidebar order, append other source pages, and use
the native Docusaurus slug rules. Documentation content and
active-version configuration are preserved.
- **Paths and links:** Produce stable version/language filenames, a
manifest, and SHA-256 checksums. Restore document/heading links across
print batches and add contents/bookmarks.
- **Runtime:** Use local Pandoc, Python/pypdf, curl, and an installed
Chrome/Chromium with its normal sandbox. Expand tabs/details and
navigation cards; retain animation URLs. Fail on unsupported components
and missing images, with an explicit placeholder option for the latter.
No site UI changes are included.
- **Validation:** The command, source inventory, content conversion,
workflow security checks, and real PDF samples were tested as listed
below.
- **Other findings:** No additional blocking issues found. Runtime
dependencies and the working-tree snapshot behavior are documented. The
separate manual GitHub PDF workflow remains available.
## Usage
```sh
python3 -m venv tmp/pdfs/.venv
tmp/pdfs/.venv/bin/python -m pip install -r
scripts/docs-pdf/requirements.txt
yarn docs:pdf --list-versions
yarn docs:pdf --version 4.x
./scripts/docs-pdf/generate.sh 4.x zh-CN
yarn docs:pdf --version dev --locale zh-CN
yarn docs:pdf --version 2.1 --locale en --output ./output/pdf/2.1-english
```
The default output directory is `output/pdf/docs/<version>/`. Chrome and
the repository's Python virtual environment are detected automatically;
`--chrome` and `--python` provide overrides.
## Validation details
- 8 Node tests passed for selection/validation, current-version aliases,
native slugs, sidebars, MDX cards/tabs/details, custom heading
destinations, and real inventories for all six versions in both
languages.
- 18 existing workflow security/authorization tests passed after
removing the retired cron job from their expected workflow list.
- Generated four sample PDFs: 4.x English/Chinese, Dev Chinese, and 2.1
English. All passed hash, page/document count, bookmark, internal-link,
and extracted-text verification with zero missing images. Rendered
representative pages were visually inspected.
- Verified that `--resume` retains the other locale, checksum corruption
is rejected, and a missing Chrome executable fails before output
creation.
- Shell wrapper passed Bash syntax and ShellCheck, positional/flag
argument forwarding, spaces in arguments, invocation outside the
repository, help, and child exit-code checks.
- `git diff --check` passed.
The local export tooling has been split out of archive-page PR #4555
into this PR. Both PRs use the identical shared inventory helper.
## Versions
- [x] dev
- [x] 4.x
- [x] 3.x
- [x] 2.1 or older (not covered by version/language sync gate)
## Languages
- [x] Chinese
- [x] English
## Docs Checklist
- [x] Checked by AI
- [x] Test Cases Built
- [x] Updated required version and language counterparts, or explained
why not
- [x] If only one language changed, confirmed whether source/translation
counterparts need sync
---------
Co-authored-by: morningman <[email protected]>
---
.github/scripts/test_review.py | 2 +-
.github/workflows/cron-generate-pdf.yml | 98 ------------
package.json | 1 +
scripts/docs-pdf/README.md | 163 +++++++++++++++++++
scripts/docs-pdf/cli.cjs | 55 +++++++
scripts/docs-pdf/docs-pdf.test.cjs | 109 +++++++++++++
scripts/docs-pdf/generate.cjs | 272 ++++++++++++++++++++++++++++++++
scripts/docs-pdf/generate.sh | 48 ++++++
scripts/docs-pdf/merge.py | 72 +++++++++
scripts/docs-pdf/print.css | 27 ++++
scripts/docs-pdf/requirements.txt | 1 +
scripts/docs-pdf/verify.py | 37 +++++
12 files changed, 786 insertions(+), 99 deletions(-)
diff --git a/.github/scripts/test_review.py b/.github/scripts/test_review.py
index 8e0af1ba5834..45308ca22cd9 100644
--- a/.github/scripts/test_review.py
+++ b/.github/scripts/test_review.py
@@ -294,7 +294,7 @@ class WorkflowTests(unittest.TestCase):
if name == "opencode-review.yml":
self.assertEqual(step["with"]["fetch-depth"], 0)
for name in ("cron-deploy-website.yml", "manual-deploy-website.yml",
- "cron-generate-pdf.yml", "manual-generate-pdf.yml"):
+ "manual-generate-pdf.yml"):
workflow = workflows[name]
build = workflow["jobs"]["build"]
self.assertNotIn("environment", build)
diff --git a/.github/workflows/cron-generate-pdf.yml
b/.github/workflows/cron-generate-pdf.yml
deleted file mode 100644
index 835748931cf7..000000000000
--- a/.github/workflows/cron-generate-pdf.yml
+++ /dev/null
@@ -1,98 +0,0 @@
-name: Cron Generate Pdf
-
-on:
- schedule:
- - cron: '0 12,18,1 * * *'
-
-
-permissions:
- contents: read
-
-concurrency:
- group: production-pdf
- cancel-in-progress: false
-
-jobs:
- source:
- if: github.ref == 'refs/heads/master'
- runs-on: ubuntu-latest
- timeout-minutes: 5
- outputs:
- source_sha: ${{ steps.source.outputs.source_sha }}
- branch: ${{ steps.source.outputs.branch }}
- steps:
- - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
- with:
- ref: ${{ github.sha }}
- persist-credentials: false
- - name: Resolve a protected branch to an immutable commit
- id: source
- env:
- GH_TOKEN: ${{ github.token }}
- DEPLOY_BRANCH: master
- run: python3 .github/scripts/review.py deploy-ref
-
- build:
- needs: source
- runs-on: ubuntu-latest
- timeout-minutes: 90
- steps:
- - name: Checkout
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
- with:
- persist-credentials: false
- ref: ${{ needs.source.outputs.source_sha }}
-
- - name: Use Node.js
- uses: actions/setup-node@3235b876344d2a9aa001b8d1453c930bba69e610 # v3
- with:
- node-version: 20
-
- - name: Generate PDF
- run: |
- npx --yes [email protected]
--initialDocURLs="https://doris.apache.org/docs/dev/getting-started/intro";
--paginationSelector=".pagination-nav__link--next" --contentSelector="article"
--coverImage="https://cdn.selectdb.com/static/doris_logo_512_4903556647.png";
--coverTitle="Apache Doris Docs (English)" --outputPDFFilename="Apache Doris
Docs (English).pdf" --tocOnlyH1=true
- npx --yes [email protected]
--initialDocURLs="https://doris.apache.org/zh-CN/docs/dev/getting-started/intro";
--paginationSelector=".pagination-nav__link--next" --contentSelector="article"
--coverImage="https://cdn.selectdb.com/static/doris_logo_512_4903556647.png";
--coverTitle="Apache Doris Docs (中文)" --outputPDFFilename="Apache Doris Docs
(中文).pdf" --tocOnlyH1=true
- mkdir -p ./build-pdf/assets/files/
- cp -- ./*.pdf ./build-pdf/assets/files/
- ls ./build-pdf/assets/files/
- - name: Upload the site artifact for the isolated deployment job
- uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 #
v4
- with:
- name: site-build-${{ github.run_id }}-${{ github.run_attempt }}
- path: build-pdf
- include-hidden-files: true
- compression-level: 0
- if-no-files-found: error
- retention-days: 1
-
- deploy:
- needs: [source, build]
- runs-on: ubuntu-latest
- timeout-minutes: 180
- environment: Production
- permissions:
- contents: read
- env:
- DORIS_COMMIT: ${{ needs.source.outputs.source_sha }}
- steps:
- - name: Checkout trusted deployment code
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
- with:
- ref: ${{ github.sha }}
- persist-credentials: false
- submodules: recursive
- - uses:
actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4
- with:
- name: site-build-${{ github.run_id }}-${{ github.run_attempt }}
- path: build-pdf
- - name: Upload files to OSS
- uses: ./.github/actions/aliyun-oss-website-action
- with:
- accessKeyId: ${{ secrets.ALIYUN_ACCESS_KEY_ID }}
- accessKeySecret: ${{ secrets.ALIYUN_ACCESS_KEY_SECRET }}
- bucket: ${{ secrets.ALIYUN_OSS_BUCKET }}
- # Transfer-acceleration endpoint, same as the website deploys.
- endpoint: oss-accelerate.aliyuncs.com
- folder: build-pdf
- onlyUpload: true
- skipSetting: true
diff --git a/package.json b/package.json
index 7d6dc92a94e5..ccd3271d25e9 100644
--- a/package.json
+++ b/package.json
@@ -25,6 +25,7 @@
"generatePDF": "npx vitpress-generate-pdf -buildDirPath='./build/'
--firstDocPath='/docs/summary/basic-summary/'
--paginationSelector='.pagination-nav__link--next' --contentSelector='article'
--coverImage='https://cdn.selectdb.com/images/doris-logo-512.png'
--coverTitle='Apache Doris 1.1 (English)' --outputPDFFilename='Apache Doris 1.1
(English).pdf'",
"write-translations": "docusaurus write-translations --locale zh-CN",
"write-heading-ids": "docusaurus write-heading-ids",
+ "docs:pdf": "node scripts/docs-pdf/cli.cjs",
"docs:manifest": "node scripts/docs-governance/manifest.js --output
website-quality-governance/generated/docs-manifest.json",
"docs:links": "node scripts/docs-governance/lint-links.js",
"docs:links:changed": "node scripts/docs-governance/lint-links.js
--changed",
diff --git a/scripts/docs-pdf/README.md b/scripts/docs-pdf/README.md
new file mode 100644
index 000000000000..a140536137f9
--- /dev/null
+++ b/scripts/docs-pdf/README.md
@@ -0,0 +1,163 @@
+# Export documentation to PDF locally
+
+Generate the documentation for a selected version from this checkout. The
+command reads local Markdown/MDX and version sidebars, so you can export Dev,
+active releases, or historical versions without building or deploying the site.
+Remote images are downloaded and embedded; uploading the finished PDF is a
+separate step. The working tree is exported, including uncommitted
documentation
+edits; the recorded source commit identifies the checkout base.
+
+## Quick start
+
+Requirements: Node.js 20+, Python 3.10+, Pandoc, curl, and a recent Chrome or
+Chromium. Install a Chinese font such as Noto Sans CJK when exporting Chinese
+on Linux. macOS can use its system Chinese fonts.
+
+Run these setup and export commands from the repository root:
+
+```sh
+yarn install --frozen-lockfile
+python3 -m venv tmp/pdfs/.venv
+tmp/pdfs/.venv/bin/python -m pip install -r scripts/docs-pdf/requirements.txt
+
+# Check available versions, then export the Chinese 4.x documentation.
+./scripts/docs-pdf/generate.sh --list-versions
+./scripts/docs-pdf/generate.sh 4.x zh-CN
+```
+
+The PDF is written to `output/pdf/docs/4.x/apache-doris-4.x-zh-CN.pdf`.
+Omit `zh-CN` to export both English and Chinese.
+
+The command detects this virtual environment and common Chrome/Chromium paths.
+Use `--python` and `--chrome` to select other executables. Chrome's normal
+sandbox remains enabled; a Linux sandbox error needs a working local browser
+installation and OS sandbox configuration.
+
+## Export a version
+
+The shell wrapper locates the repository root and accepts the version and
+optional language as positional arguments:
+
+```sh
+./scripts/docs-pdf/generate.sh 4.x
+./scripts/docs-pdf/generate.sh dev zh-CN
+./scripts/docs-pdf/generate.sh 2.1 en --output ./output/pdf/2.1-english
+./scripts/docs-pdf/generate.sh --list-versions
+```
+
+It also accepts the original `--version` and `--locale` flags and forwards
+additional options unchanged. You can invoke it by its path from another
+directory; relative option paths are resolved from the repository root.
+
+The Yarn command is also available directly:
+
+```sh
+# Discover the versions present in this checkout.
+yarn docs:pdf --list-versions
+
+# Export both English and Chinese for 4.x.
+yarn docs:pdf --version 4.x
+
+# Export only Chinese for Dev (current is an alias for dev).
+yarn docs:pdf --version dev --locale zh-CN
+
+# Export a historical version to a chosen directory.
+yarn docs:pdf --version 2.1 --locale en --output ./output/pdf/2.1-english
+
+# Explicit browser/interpreter selection, if needed.
+yarn docs:pdf --version 3.x --locale zh-CN \
+ --chrome '/Applications/Google Chrome.app/Contents/MacOS/Google Chrome' \
+ --python tmp/pdfs/.venv/bin/python
+```
+
+## Parameters
+
+`generate.sh <version> [locale] [options]` maps the first two positional
+arguments to `--version` and `--locale`. The remaining options are forwarded
+to the CLI.
+
+| Parameter | Default | Meaning |
+| --- | --- | --- |
+| `--version <version>` | Required for an export | `dev` (`current` is an
alias), or a version directory present in `versioned_docs/`. Use
`--list-versions` for the names available in this checkout. |
+| `--locale <locales>` | `en,zh-CN` | Export `en`, `zh-CN`, or both as
`en,zh-CN`. The selected source directories must exist. |
+| `--output <directory>` | `output/pdf/docs/<version>/` | Destination for the
selected PDFs and their manifest/checksums. With the shell wrapper, relative
paths start at the repository root. |
+| `--chrome <executable>` | Auto-detected | Chrome/Chromium executable. Paths
containing spaces must be quoted. |
+| `--python <executable>` | Repository virtual environment, then `python3` |
Python interpreter with the dependencies from `requirements.txt` installed. |
+| `--limit <count>` | All source pages | Export the first `<count>` source
documents as a sample. The count must be a positive integer and requires an
explicit `--output`; it does not specify the number of printed PDF pages. |
+| `--resume` | Off | Keep manifest entries for other locales while
regenerating the selected locale. An existing manifest must have the same
source commit. |
+| `--allow-missing-images` | Off | Replace unreachable images with links to
their original URLs and record warnings in the manifest. |
+| `--list-versions` | Off | Print the versions available locally and exit
without exporting. |
+| `--help` | Off | Print usage and exit. The shell wrapper also accepts `-h`. |
+
+`--chrome` takes precedence over `DOCS_PDF_CHROME` and
+`PUPPETEER_EXECUTABLE_PATH`. `--python` takes precedence over
+`DOCS_PDF_PYTHON`. Without an explicit interpreter, the command checks
+`tmp/pdfs/.venv/bin/python` (`Scripts/python.exe` on Windows), then uses
+`python3`.
+
+## Output
+
+By default, output goes to `output/pdf/docs/<version>/`:
+
+- `apache-doris-<version>-en.pdf` and/or `apache-doris-<version>-zh-CN.pdf`
+- `manifest.json`: source commit, selected source files, document/page counts,
+ sample status, image warnings, file sizes, and SHA-256 hashes
+- `SHA256SUMS`: checksums for the PDFs listed in the manifest
+
+The PDF has a cover, clickable contents, document bookmarks, internal links,
+and page numbers. Source pages follow sidebar order, with pages outside the
+sidebar appended. All Tab content and collapsed details are expanded.
+Navigation cards retain their titles, descriptions, and links; animations are
+represented by links to the online resource. Unsupported MDX components fail
+with their source filename instead of silently dropping content.
+
+The selected locale source directory must exist. Supported languages are
+`en` and `zh-CN`. Unknown versions, locales, and options fail before creating
+output. The tool requires no OSS credentials and performs no upload.
+
+## Samples, retries, and verification
+
+```sh
+# A small sample: the explicit output directory prevents replacing a full PDF.
+./scripts/docs-pdf/generate.sh 4.x zh-CN --limit 3 \
+ --output ./tmp/pdfs/4.x-sample
+
+# Preserve other locales in the manifest when regenerating a selection.
+./scripts/docs-pdf/generate.sh 4.x en --resume
+
+# Check hashes, counts, bookmarks, internal links, and sample extracted text.
+tmp/pdfs/.venv/bin/python scripts/docs-pdf/verify.py output/pdf/docs/4.x
+
+# Run the command/inventory/content conversion tests.
+node --test scripts/docs-pdf/docs-pdf.test.cjs
+```
+
+`--limit` produces a partial sample and records `sample: true` in the manifest.
+`--resume` requires the same source commit as the existing manifest. Temporary
+render batches and cached images live under `tmp/pdfs/docs/<version>/`.
+
+Missing images fail the export by default. Retry after fixing the network or
+source image. `--allow-missing-images` explicitly permits linked placeholders
+and records them in the manifest; review those warnings before distributing
+the result. The verifier also requires `--allow-missing-images` to accept them.
+PDF files are replaced only after their render batches merge successfully.
+
+## Troubleshooting
+
+| Error or symptom | Action |
+| --- | --- |
+| Yarn is missing, or a Node module cannot be found | Make Yarn available in
`PATH` and run `yarn install --frozen-lockfile` from the repository root. |
+| `Unknown documentation version` | Run `./scripts/docs-pdf/generate.sh
--list-versions`. Select a documentation directory name such as `4.x`, rather
than a Doris patch release number such as `4.1.0`. |
+| Pandoc or curl is missing | Install the named tool and make it available in
`PATH`. |
+| Python cannot import `pypdf` | Repeat the virtual-environment setup above,
or pass its interpreter with `--python`. |
+| Chrome is missing or cannot render MathML | Install a recent Chrome/Chromium
and pass the full executable path with `--chrome`. |
+| `No usable sandbox!` on Linux | Check the local Chrome installation and OS
sandbox configuration. The exporter keeps Chrome's normal sandbox enabled. |
+| A remote image cannot be retrieved | Check the image URL and network
connection, then retry. Use `--allow-missing-images` only when a PDF with
linked placeholders is acceptable. |
+| `Cannot resume PDFs from a different source commit` | Choose a new output
directory, or regenerate the desired languages without `--resume`. |
+
+Check the command's exit status and run the verifier before uploading a PDF.
+When reusing an output directory, a failed export can leave the previous
+successful PDF in place. Review `manifest.json` to identify the files and
+source commit for the completed export.
+
+The retired scheduled workflow was `.github/workflows/cron-generate-pdf.yml`.
diff --git a/scripts/docs-pdf/cli.cjs b/scripts/docs-pdf/cli.cjs
new file mode 100644
index 000000000000..46199caaf9ab
--- /dev/null
+++ b/scripts/docs-pdf/cli.cjs
@@ -0,0 +1,55 @@
+#!/usr/bin/env node
+const fs = require('node:fs');
+const path = require('node:path');
+const {parseArgs} = require('node:util');
+const {availableVersions, normalizeVersion, contentDirectory} =
require('../docs-inventory.cjs');
+const root = path.resolve(__dirname, '../..');
+const help = `Export a local Apache Doris documentation version to PDF.
+
+Usage: yarn docs:pdf --version <version> [options]
+
+ --version <version> Required: dev (alias: current), or a
versioned_docs version
+ --locale <locales> en, zh-CN, or en,zh-CN (default: en,zh-CN)
+ --output <directory> Default: output/pdf/docs/<version>/
+ --chrome <executable> Installed Chrome/Chromium; auto-detected when
omitted
+ --python <executable> Python with pypdf (default: repo venv, then
python3)
+ --limit <count> Sample export; requires an explicit --output
+ --allow-missing-images Use linked placeholders for unreachable images
+ --resume Retain PDFs for other locales from the same commit
+ --list-versions List versions available in this checkout
+ --help Show this help
+
+Outputs: PDF files, manifest.json, and SHA256SUMS. Upload is a separate manual
step.
+`;
+function parseOptions(args, siteDir = root) {
+ const {values} = parseArgs({args, options: {
+ version: {type: 'string'}, locale: {type: 'string', default:
'en,zh-CN'}, output: {type: 'string'},
+ chrome: {type: 'string'}, python: {type: 'string'},
+ 'allow-missing-images': {type: 'boolean'}, limit: {type: 'string'},
resume: {type: 'boolean'}, 'list-versions': {type: 'boolean'}, help: {type:
'boolean'},
+ }});
+ if (values.help || values['list-versions']) return values;
+ if (!values.version?.trim()) throw new Error('Specify --version. Use
--list-versions to see available versions.');
+ const version = normalizeVersion(values.version.trim());
+ if (!availableVersions(siteDir).includes(version)) throw new
Error(`Unknown documentation version: ${values.version}. Use --list-versions.`);
+ const locales = values.locale.split(',').map(locale => locale.trim());
+ if (new Set(locales).size !== locales.length || locales.some(locale =>
!['en', 'zh-CN'].includes(locale)))
+ throw new Error('--locale must be en, zh-CN, or en,zh-CN.');
+ for (const locale of locales) if (!fs.existsSync(contentDirectory(siteDir,
version, locale)))
+ throw new Error(`Missing ${locale} source for ${version}.`);
+ const limit = values.limit === undefined ? 0 : Number(values.limit);
+ if (values.limit !== undefined && (!Number.isSafeInteger(limit) || limit <
1)) throw new Error('--limit must be a positive integer.');
+ if (limit && !values.output) throw new Error('Sample exports require an
explicit --output directory.');
+ const venvPython = path.join(siteDir, 'tmp/pdfs/.venv', process.platform
=== 'win32' ? 'Scripts/python.exe' : 'bin/python');
+ const python = values.python || process.env.DOCS_PDF_PYTHON ||
(fs.existsSync(venvPython) ? venvPython : 'python3');
+ return {version, locales, limit, allowMissingImages:
values['allow-missing-images'] ?? false, resume: values.resume ?? false, python,
+ chrome: values.chrome || process.env.DOCS_PDF_CHROME ||
process.env.PUPPETEER_EXECUTABLE_PATH,
+ output: values.output ? path.resolve(values.output) :
path.join(siteDir, 'output/pdf/docs', version)};
+}
+async function main() {
+ const options = parseOptions(process.argv.slice(2));
+ if (options.help) return process.stdout.write(help);
+ if (options['list-versions']) return
process.stdout.write(availableVersions(root).join('\n') + '\n');
+ await require('./generate.cjs').generate(options);
+}
+if (require.main === module) main().catch(error => {console.error(`PDF export
failed: ${error.message}`); process.exitCode = 1;});
+module.exports = {parseOptions};
diff --git a/scripts/docs-pdf/docs-pdf.test.cjs
b/scripts/docs-pdf/docs-pdf.test.cjs
new file mode 100644
index 000000000000..b93b9fcde21e
--- /dev/null
+++ b/scripts/docs-pdf/docs-pdf.test.cjs
@@ -0,0 +1,109 @@
+const assert = require('node:assert/strict');
+const fs = require('node:fs');
+const os = require('node:os');
+const path = require('node:path');
+const {execFileSync, spawnSync} = require('node:child_process');
+const test = require('node:test');
+const {parseOptions} = require('./cli.cjs');
+const {normalize, addHeadingTargets} = require('./generate.cjs');
+const cheerio = require('cheerio');
+const {availableVersions, inventory, filename} =
require('../docs-inventory.cjs');
+const root = path.resolve(__dirname, '../..');
+function fixture(t) {
+ const directory = fs.mkdtempSync(path.join(os.tmpdir(),
'doris-pdf-test-'));
+ t.after(() => fs.rmSync(directory, {recursive: true, force: true}));
+ const write = (file, text) => {const full = path.join(directory, file);
fs.mkdirSync(path.dirname(full), {recursive: true}); fs.writeFileSync(full,
text);};
+ for (const locale of ['en', 'zh-CN']) {
+ const current = locale === 'en' ? 'docs' :
'i18n/zh-CN/docusaurus-plugin-content-docs/current';
+ const versioned = locale === 'en' ? 'versioned_docs/version-4.x' :
'i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x';
+ for (const base of [current, versioned]) {
+ write(`${base}/intro.md`, '---\ntitle: Intro\n---\nWelcome');
+ write(`${base}/01-topic.md`, '---\ntitle: Topic\nslug:
/custom-topic\n---\nTopic');
+ write(`${base}/99-extra.md`, '# Extra\nOutside the sidebar');
+ }
+ }
+ write('sidebars.ts', "const sidebars = {docs: [{type: 'category', label:
'Start', link: {type: 'doc', id: 'intro'}, items: ['topic']}]}; export default
sidebars;");
+ write('versioned_sidebars/version-4.x-sidebars.json',
JSON.stringify({docs: ['intro', 'topic']}));
+ return {directory, write};
+}
+
+test('version discovery and current alias use the actual local source roots',
t => {
+ const {directory} = fixture(t);
+ assert.deepEqual(availableVersions(directory), ['dev', '4.x']);
+ for (const version of ['dev', 'current', '4.x']) for (const locale of
['en', 'zh-CN']) {
+ const docs = inventory(directory, version, locale);
+ assert.deepEqual(docs.map(doc => doc.id), ['intro', 'topic', 'extra']);
+ assert.equal(docs[1].route, `${locale === 'en' ? '' :
'/zh-CN'}/docs/${version === 'current' ? 'dev' : version}/custom-topic`);
+ }
+ assert.equal(filename('current', 'zh-CN'), 'apache-doris-dev-zh-CN.pdf');
+});
+
+test('explicit CLI selection, locale subsets, output paths, and samples are
retained', t => {
+ const {directory} = fixture(t);
+ const options = parseOptions(['--version', 'current', '--locale', 'zh-CN',
'--output', 'folder with spaces', '--limit', '2', '--resume'], directory);
+ assert.equal(options.version, 'dev');
+ assert.deepEqual(options.locales, ['zh-CN']);
+ assert.equal(options.limit, 2);
+ assert.equal(options.resume, true);
+ assert.equal(options.output, path.resolve('folder with spaces'));
+ assert.deepEqual(parseOptions(['--version', '4.x'], directory).locales,
['en', 'zh-CN']);
+});
+
+test('invalid selections fail without creating files', t => {
+ const {directory} = fixture(t);
+ for (const args of [[], ['--version', '9.x'], ['--version', '../4.x'],
['--version', '4.x', '--locale', 'ja'],
+ ['--version', '4.x', '--locale', 'en,en'], ['--version', '4.x',
'--limit', '2'],
+ ['--version', '4.x', '--limit', '-1', '--output', 'sample'],
['--version', '4.x', '--typo']]) assert.throws(() => parseOptions(args,
directory));
+ fs.rmSync(path.join(directory,
'i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x'), {recursive: true});
+ assert.throws(() => parseOptions(['--version', '4.x'], directory),
/Missing zh-CN source/);
+ assert(!fs.existsSync(path.join(directory, 'output')));
+ assert(!fs.existsSync(path.join(directory, 'tmp')));
+});
+
+test('CLI help and version listing do not load the renderer or create
outputs', () => {
+ const cli = path.join(__dirname, 'cli.cjs');
+ const help = execFileSync(process.execPath, [cli, '--help'], {encoding:
'utf8'});
+ assert(help.includes('--version'));
+ const versions = execFileSync(process.execPath, [cli, '--list-versions'],
{encoding: 'utf8'}).trim().split('\n');
+ assert(versions.includes('dev') && versions.includes('4.x') &&
versions.includes('2.1'));
+ const invalid = spawnSync(process.execPath, [cli, '--version', 'missing'],
{encoding: 'utf8'});
+ assert.equal(invalid.status, 1);
+ assert(invalid.stderr.includes('Unknown documentation version'));
+});
+
+test('printing retains every tab/detail body and preserves fenced and inline
examples', () => {
+ const code = '```sql\nSELECT "<Tabs>";\n```\n\n```sh\nexport
JAVA_TOOL_OPTIONS="x"\n```';
+ const result = normalize('import Tabs from
"@theme/Tabs";\n<Tabs>\n<TabItem label="S3">\nAWS\n</TabItem>\n<TabItem
label="OSS">\nAliyun\n</TabItem>\n</Tabs>\n<details><summary>More</summary>\nHidden\n</details>\n'
+ code);
+ assert(result.includes('### S3') && result.includes('### OSS') &&
result.includes('Hidden'));
+ assert(result.includes(code));
+ assert(!result.includes('import Tabs') && !result.includes('<TabItem'));
+ assert.equal(normalize('Type: <INT>; literal: `<INT>`'), 'Type:
<INT>; literal: `<INT>`');
+});
+
+test('current-version cards preserve titles/descriptions/links; animations
remain usable links', () => {
+ const result = normalize(`import GettingStartedCard from
'@site/src/components/card';\n<GettingStartedCard title="What's Doris"
description="Core features" link="quick-start" />\n<iframe
src="/animations/intro.html" title="Doris animation"></iframe>`);
+ assert(result.includes('What's Doris') && result.includes('Core
features') && result.includes('(quick-start)'));
+ assert(result.includes('[Doris animation](/animations/intro.html)'));
+ assert(!result.includes('<GettingStartedCard') &&
!result.includes('<iframe'));
+ assert.throws(() => normalize('import Widget from "./widget";\n<Widget
/>'), /Unsupported MDX component Widget/);
+});
+
+test('real current/active/historical inventories keep sidebar entries and
outside-sidebar sources', () => {
+ for (const version of availableVersions(root)) for (const locale of ['en',
'zh-CN']) {
+ const docs = inventory(root, version, locale);
+ assert(docs.length > 700);
+ assert.equal(docs.length, new Set(docs.map(doc => doc.source)).size);
+ assert(docs[0].id.includes('intro') ||
docs[0].id.includes('what-is-apache-doris'));
+ }
+});
+
+test('custom heading IDs remain destinations and card links are not nested
inside marker links', () => {
+ const $ = cheerio.load('<article><h1>Title</h1><h2 id="stable-anchor">New
heading</h2><h3 id="card"><a href="quick-start">Quick
Start</a></h3></article>');
+ const aliases = {};
+ addHeadingTargets($, $('article'), 'doc-1', aliases);
+ assert.equal(aliases['doc-1#stable-anchor'], 'doc-1#new-heading');
+ assert.equal(aliases['doc-1#card'], 'doc-1#quick-start');
+ assert.equal($('a[href="quick-start"]').length, 1);
+ assert.equal($('a a').length, 0);
+ assert.equal($('a[href^="https://archive.invalid/start/";]').length, 3);
+});
diff --git a/scripts/docs-pdf/generate.cjs b/scripts/docs-pdf/generate.cjs
new file mode 100644
index 000000000000..32397ec12788
--- /dev/null
+++ b/scripts/docs-pdf/generate.cjs
@@ -0,0 +1,272 @@
+#!/usr/bin/env node
+// Local-only export: never uploads files or requires a production build.
+const fs = require('node:fs');
+const path = require('node:path');
+const {execFile, execFileSync} = require('node:child_process');
+const {createHash} = require('node:crypto');
+const {pathToFileURL} = require('node:url');
+const cheerio = require('cheerio');
+const GithubSlugger = require('github-slugger');
+const puppeteer = require('puppeteer');
+const {inventory, filename} = require('../docs-inventory.cjs');
+const root = path.resolve(__dirname, '../..');
+const escape = text => String(text).replace(/[&<>"']/g, c => ({'&': '&',
'<': '<', '>': '>', '"': '"', "'": '''}[c]));
+const ts = require('typescript');
+
+function staticProps(tag) {
+ const source = ts.createSourceFile('component.tsx', `const element =
(${tag});`, ts.ScriptTarget.Latest, true, ts.ScriptKind.TSX);
+ let opening;
+ function visit(node) {
+ if (ts.isJsxSelfClosingElement(node) || ts.isJsxOpeningElement(node))
opening = node;
+ if (!opening) ts.forEachChild(node, visit);
+ }
+ visit(source);
+ const props = {};
+ for (const attr of opening?.attributes.properties ?? []) {
+ if (!ts.isJsxAttribute(attr) || !attr.initializer) continue;
+ const value = ts.isJsxExpression(attr.initializer) ?
attr.initializer.expression : attr.initializer;
+ if (value && (ts.isStringLiteral(value) ||
ts.isNoSubstitutionTemplateLiteral(value))) props[attr.name.getText(source)] =
value.text;
+ }
+ return props;
+}
+function addHeadingTargets($, article, docKey, aliases) {
+ const slugger = new GithubSlugger();
+ article.find('h1,h2,h3,h4,h5,h6').each((index, heading) => {
+ const original = $(heading).attr('id');
+ const headingKey = docKey + (index ? '#' +
slugger.slug($(heading).text()) : '');
+ if (index && original && `${docKey}#${original}` !== headingKey)
aliases[`${docKey}#${original}`] = headingKey;
+ const inner = $(heading).html();
+ $(heading).attr('id', headingKey);
+ if ($(heading).find('a[href]').length) $(heading).append(` <a
class="section-link"
href="https://archive.invalid/start/${escape(headingKey)}">§</a>`);
+ else $(heading).html(`<a
href="https://archive.invalid/start/${escape(headingKey)}">${inner}</a>`);
+ });
+}
+function findChrome(explicit) {
+ const candidates = explicit ? [explicit] : [
+ ...(process.platform === 'darwin' ? ['/Applications/Google
Chrome.app/Contents/MacOS/Google Chrome',
+ '/Applications/Chromium.app/Contents/MacOS/Chromium'] : []),
+ ...(process.env.PATH ?? '').split(path.delimiter).flatMap(dir =>
['google-chrome-stable', 'google-chrome', 'chromium',
+ 'chromium-browser', 'chrome.exe'].map(name => path.join(dir,
name))),
+ puppeteer.executablePath(),
+ ];
+ const executable = candidates.find(file => {try {fs.accessSync(file,
fs.constants.X_OK); return fs.statSync(file).isFile();} catch {return false;}});
+ if (!executable) throw new Error('No Chrome/Chromium executable found.
Install Chrome and pass --chrome <executable>.');
+ return executable;
+}
+function checkCommand(executable, args, message) {
+ try {execFileSync(executable, args, {stdio: 'pipe'});} catch {throw new
Error(message);}
+}
+
+function checkComponents(content) {
+ for (const match of
content.matchAll(/^import\s+(\w+)\s+from\s+['"][^'"]+['"]/gm)) {
+ if (!['Tabs', 'TabItem', 'GettingStartedCard'].includes(match[1]) &&
new RegExp(`<${match[1]}\\b`).test(content))
+ throw new Error(`Unsupported MDX component ${match[1]}; provide a
static printable representation before exporting.`);
+ }
+ return content;
+}
+function normalize(content) {
+ // Transform presentation components outside code fences only. Every tab
+ // and collapsed details body is included in the printed document.
+ return content.split(/(^[ \t]*(`{3,}|~{3,})[^\n]*\n[\s\S]*?^[ \t]*\2[
\t]*$)/gm)
+ .filter(part => part && !/^(`{3,}|~{3,})$/.test(part))
+ .map(part => /^[ \t]*(`{3,}|~{3,})/.test(part) ? part :
checkComponents(part)
+ .replace(/<GettingStartedCard\b[\s\S]*?\/>/g, tag => {
+ const props = staticProps(tag);
+ if (!props.title || !props.link) throw new
Error('GettingStartedCard requires static title and link properties.');
+ return `\n\n###
[${escape(props.title)}](${props.link})\n\n${props.description ?? ''}\n\n`;
+ })
+ .replace(/<iframe\b[\s\S]*?<\/iframe>/g, tag => {
+ const props = staticProps(tag);
+ if (!props.src) throw new Error('Iframe requires a static src
property.');
+ return `\n\n[${escape(props.title || 'Interactive
content')}](${props.src})\n\n`;
+ })
+ .replace(/^import\s+\w+\s+from\s+['"][^'"]+['"];?\s*$/gm, '')
+ .replace(/<Tabs\b[^>]*>|<\/Tabs>|<\/TabItem>/g, '\n\n')
+ .replace(/<TabItem\b[\s\S]*?>/g, tag => '\n\n### ' +
(tag.match(/label\s*=\s*["']([^"']+)/)?.[1] ?? 'Example') + '\n\n')
+ .replace(/^:::(\w+)\s*(.*?)\s*$/gm, (_, kind, title) => `\n<div
class="admonition">\n\n**${title || kind.toUpperCase()}**\n`)
+ .replace(/^:::\s*$/gm, '\n</div>\n')
+ .replace(/<details\b[^>]*>/g, '<div class="details">')
+ .replace(/<\/details>/g, '</div>')
+ .replace(/<summary>([\s\S]*?)<\/summary>/g, '\n\n**$1**\n\n')
+ .replace(/\sstyle=\{\{[\s\S]*?\}\}/g, '')
+ .replace(/^
+(?=<\/?(?:div|p|img|table|thead|tbody|tr|th|td|ul|ol|li)\b)/gm, '')
+ .replace(/(`+)[^\n]*?\1|<\/?[A-Z][A-Za-z0-9_]*\s*\/?>/g,
+ match => match.startsWith('`') ? match : escape(match))
+ .replace(/^(#{1,5})\s/gm, '#$1 ')).join('');
+}
+
+function markdownToHtml(input) {
+ return new Promise((resolve, reject) => {
+ const child = execFile('pandoc',
['--from=markdown+markdown_in_html_blocks+pipe_tables+raw_html', '--to=html5',
'--mathml', '--wrap=none'],
+ {encoding: 'utf8', maxBuffer: 64 * 1024 * 1024}, (error, stdout,
stderr) => error ? reject(new Error(stderr || error.message)) :
resolve(stdout));
+ child.stdin.end(input);
+ });
+}
+async function imageSource(src, doc, temporary, imageCache) {
+ const clean = decodeURI(src.split(/[?#]/)[0]);
+ let local;
+ if (/^https?:/.test(src)) {
+ const url = new URL(src);
+ local = path.join(root, 'static', url.pathname);
+ } else local = clean.startsWith('/') ? path.join(root, 'static', clean) :
path.resolve(path.dirname(doc.file), clean);
+ if (fs.existsSync(local) && fs.statSync(local).isFile()) return
pathToFileURL(local).href;
+ // Older Markdown sometimes addresses shared assets through a localized
+ // route or several ../ segments. They still live under static/images.
+ if ((!/^https?:/.test(src) || new URL(src).hostname ===
'doris.apache.org') && clean.includes('/images/')) {
+ const shared = path.join(root, 'static',
clean.slice(clean.indexOf('/images/')));
+ if (fs.existsSync(shared) && fs.statSync(shared).isFile()) return
pathToFileURL(shared).href;
+ }
+ const url = /^https?:/.test(src) ? src : new URL(src,
`https://doris.apache.org${doc.route}/`).href;
+ if (!imageCache.has(url)) imageCache.set(url, (async () => {
+ const target = path.join(temporary, 'images',
createHash('sha256').update(url).digest('hex') + (path.extname(clean) ||
'.png'));
+ if (fs.existsSync(target)) return pathToFileURL(target).href;
+ try {
+ fs.mkdirSync(path.dirname(target), {recursive: true});
+ const partial = target + '.download';
+ try {
+ const type = await new Promise((resolve, reject) =>
execFile('curl', ['--fail', '--location', '--silent', '--show-error',
+ '--connect-timeout', '15', '--max-time', '120',
'--output', partial, '--write-out', '%{content_type}', url],
+ {encoding: 'utf8', maxBuffer: 1024 * 1024}, (error,
stdout, stderr) => error ? reject(new Error(stderr || error.message)) :
resolve(stdout)));
+ if (!type.startsWith('image/')) throw new Error(`Expected an
image, received ${type}`);
+ fs.renameSync(partial, target);
+ } finally {fs.rmSync(partial, {force: true});}
+ return pathToFileURL(target).href;
+ } catch (error) {return {url, error: error.message};}
+ })());
+ return imageCache.get(url);
+}
+
+async function generate({version, locales, output, python = 'python3', chrome,
limit = 0, resume = false, allowMissingImages = false}) {
+ const versions = [version];
+ const temporary = path.join(root, 'tmp/pdfs/docs', version);
+ const commit = execFileSync('git', ['rev-parse', 'HEAD'], {cwd: root,
encoding: 'utf8'}).trim();
+ const date = new Date().toISOString().slice(0, 10);
+ const css = fs.readFileSync(path.join(__dirname, 'print.css'), 'utf8');
+ const imageCache = new Map();
+ checkCommand('pandoc', ['--version'], 'Pandoc is required. Install it
before exporting.');
+ checkCommand('curl', ['--version'], 'curl is required to cache remote
documentation images.');
+ checkCommand(python, ['-c', 'import pypdf'], 'Python with pypdf is
required. Install pypdf and pass --python <executable>.');
+ const executablePath = findChrome(chrome);
+ // Validate the entire selection before creating output or starting Chrome.
+ const selections = new Map(locales.map(locale => [locale, inventory(root,
version, locale)]));
+ if ([...selections.values()].some(docs => !docs.length)) throw new
Error('No documentation pages found for this selection.');
+ const manifestPath = path.join(output, 'manifest.json');
+ const manifest = resume && fs.existsSync(manifestPath)
+ ? JSON.parse(fs.readFileSync(manifestPath, 'utf8'))
+ : {generatedAt: new Date().toISOString(), sourceCommit: commit, files:
[]};
+ if (manifest.sourceCommit !== commit) throw new Error('Cannot resume PDFs
from a different source commit.');
+ fs.mkdirSync(output, {recursive: true});
+ fs.mkdirSync(temporary, {recursive: true});
+ const browser = await puppeteer.launch({headless: true, executablePath,
+ args: ['--allow-file-access-from-files', '--disable-dev-shm-usage']});
+ try {
+ for (const version of versions) for (const locale of locales) {
+ let docs = selections.get(locale);
+ if (limit) docs = docs.slice(0, limit);
+ docs.forEach((doc, index) => {doc.key = `doc-${index + 1}`;});
+ const bySource = new Map(docs.map(doc =>
[doc.source.replace(/\.mdx?$/, ''), doc]));
+ const byRoute = new Map(docs.map(doc => [doc.route.replace(/\/$/,
''), doc]));
+ const cn = locale === 'zh-CN';
+ const title = `Apache Doris ${version === 'dev' ? 'Dev' : version}
${cn ? '中文手册' : 'Documentation (English)'}`;
+ const contentsTitle = cn ? '目录' : 'Contents';
+ const directory = path.join(temporary,
`${version}-${locale}${limit ? '-sample' : ''}`);
+ fs.mkdirSync(directory, {recursive: true});
+ const batches = [], warnings = [], headingAliases = {};
+ const cover = `<div class="cover"><img
src="${pathToFileURL(path.join(root,
'static/images/logo-doris.svg')).href}"><h1>${escape(title)}</h1><p>${cn ?
'归档版本文档' : 'Archived documentation'}</p><p>${date} · ${docs.length} ${cn ?
'篇文档' : 'documents'}</p><p>Source: ${commit.slice(0, 12)}</p></div>`;
+ const toc = `<nav
class="toc"><h1>${contentsTitle}</h1><ol>${docs.map(doc => `<li><a
href="https://archive.invalid/${doc.key}";>${escape(doc.title)}</a></li>`).join('')}</ol></nav>`;
+ const page = await browser.newPage();
+ try {
+ if (!await page.evaluate(() => typeof MathMLElement !==
'undefined'))
+ throw new Error('This browser cannot render MathML. Use an
installed modern Chrome/Chromium with --chrome.');
+ for (let start = 0; start < docs.length; start += 40) {
+ const selected = docs.slice(start, start + 40);
+ const fragments = new Array(selected.length);
+ let next = 0;
+ await Promise.all(Array.from({length: Math.min(4,
selected.length)}, async () => {
+ while (next < selected.length) {
+ const index = next++, doc = selected[index];
+ let normalized;
+ try {normalized = normalize(doc.content);} catch
(error) {throw new Error(`${doc.source}: ${error.message}`);}
+ const converted = await markdownToHtml(`#
${escape(doc.title)} {.doc-title}\n\n${normalized}`);
+ // Repair legacy raw HTML in isolation so an
unclosed
+ // tag cannot consume the following document.
+ const parsed = cheerio.load(converted);
+ fragments[index] = `<section class="archive-doc"
data-key="${doc.key}">${parsed('body').html()}</section>`;
+ }
+ }));
+ const $ = cheerio.load(fragments.join('\n'));
+ for (const doc of selected) {
+ const article =
$(`.archive-doc[data-key="${doc.key}"]`);
+ if (!article.length) throw new Error(`Missing rendered
document: ${doc.source}`);
+ addHeadingTargets($, article, doc.key, headingAliases);
+ article.find('a[href]').each((_, anchor) => {
+ const href = $(anchor).attr('href');
+ if (href.startsWith('https://archive.invalid/'))
return;
+ const [pathname, fragment] = href.split('#');
+ let target;
+ if (!pathname) target = doc;
+ else if (!/^(?:[a-z]+:|\/\/)/i.test(pathname) &&
!pathname.startsWith('/'))
+ target =
bySource.get(path.posix.normalize(path.posix.join(path.posix.dirname(doc.source),
decodeURI(pathname))).replace(/\.mdx?$/, ''));
+ else {
+ try {target = byRoute.get(decodeURI(new
URL(href, 'https://doris.apache.org').pathname).replace(/\/$/, ''));} catch {}
+ }
+ if (target) $(anchor).attr('href',
`https://archive.invalid/${target.key}${fragment ? '#' + fragment : ''}`);
+ else if (!/^(?:[a-z]+:|\/\/)/i.test(href))
$(anchor).attr('href', new URL(href,
`https://doris.apache.org${doc.route}`).href);
+ });
+ const images = article.find('img').toArray();
+ for (let offset = 0; offset < images.length; offset +=
8) {
+ await Promise.all(images.slice(offset, offset +
8).map(async element => {
+ const img = $(element), src = img.attr('src');
+ if (!src || src.startsWith('data:')) return;
+ const resolved = await imageSource(src, doc,
temporary, imageCache);
+ if (typeof resolved === 'string')
img.attr('src',
resolved).removeAttr('loading').removeAttr('width').removeAttr('height');
+ else {
+ if (!allowMissingImages) throw new
Error(`${doc.source}: could not retrieve ${resolved.url}: ${resolved.error}.
Retry, or use --allow-missing-images for placeholders.`);
+ warnings.push({source: doc.source,
...resolved});
+ img.replaceWith(`<p
class="missing-image">${escape(img.attr('alt') || (cn ? '图片' : 'Image'))}: <a
href="${escape(resolved.url)}">${escape(resolved.url)}</a> (${cn ? '源图片不可用' :
'source image unavailable'})</p>`);
+ }
+ }));
+ }
+ }
+ const htmlPath = path.join(directory, `${start}.html`);
+ const pdfPath = path.join(directory, `${start}.pdf`);
+ const html = `<!doctype html><html
lang="${locale}"><head><meta
charset="utf-8"><title>${escape(title)}</title><style>${css}</style></head><body>${start
=== 0 ? cover + toc : ''}${$('body').html()}</body></html>`;
+ fs.writeFileSync(htmlPath, html);
+ const digest =
createHash('sha256').update(html).digest('hex');
+ if (fs.existsSync(pdfPath) && fs.existsSync(pdfPath +
'.sha256') && fs.readFileSync(pdfPath + '.sha256', 'utf8') === digest) {
+ batches.push(pdfPath);
+ console.log(`${version} ${locale}: ${Math.min(start +
40, docs.length)}/${docs.length} (cached)`);
+ continue;
+ }
+ await page.goto(pathToFileURL(htmlPath).href, {waitUntil:
'load', timeout: 120000});
+ await page.evaluate(async () => {await
document.fonts.ready; await Promise.race([
+ Promise.all(Array.from(document.images, img =>
img.decode().catch(() => {}))),
+ new Promise(resolve => setTimeout(resolve, 30000)),
+ ]);});
+ const broken = await page.$$eval('img', images =>
images.filter(img => !img.complete || img.naturalWidth === 0).map(img =>
img.src));
+ if (broken.length) throw new Error(`Unrendered images:
${broken.join(', ')}`);
+ await page.pdf({path: pdfPath, format: 'A4',
preferCSSPageSize: true, printBackground: true, timeout: 0});
+ fs.writeFileSync(pdfPath + '.sha256', digest);
+ batches.push(pdfPath);
+ console.log(`${version} ${locale}: ${Math.min(start + 40,
docs.length)}/${docs.length}`);
+ }
+ } finally {await page.close();}
+ const outputPath = path.join(output, filename(version, locale));
+ const mergedPath = path.join(directory, 'merged.pdf');
+ const jobPath = path.join(directory, 'merge.json');
+ fs.writeFileSync(jobPath, JSON.stringify({title, contentsTitle,
version, locale, commit, docs: docs.map(({key, title}) => ({key, title})),
batches, headingAliases, output: mergedPath}));
+ const stats = JSON.parse(execFileSync(python,
[path.join(__dirname, 'merge.py'), jobPath], {encoding: 'utf8', maxBuffer: 10 *
1024 * 1024}));
+ fs.copyFileSync(mergedPath, outputPath + '.partial');
+ fs.renameSync(outputPath + '.partial', outputPath);
+ const bytes = fs.readFileSync(outputPath);
+ manifest.files = manifest.files.filter(file => file.version !==
version || file.locale !== locale);
+ manifest.files.push({version, locale, filename:
path.basename(outputPath),
+ sample: Boolean(limit), bytes: bytes.length,
+ sha256: createHash('sha256').update(bytes).digest('hex'),
...stats, sourceFiles: docs.map(doc => path.relative(root, doc.file)),
imageWarnings: warnings});
+ fs.writeFileSync(path.join(output, 'manifest.json'),
JSON.stringify(manifest, null, 2) + '\n');
+ fs.writeFileSync(path.join(output, 'SHA256SUMS'),
manifest.files.map(file => `${file.sha256} ${file.filename}`).join('\n') +
'\n');
+ console.log(`Saved ${outputPath} (${stats.pages} pages,
${warnings.length} image warnings)`);
+ }
+ } finally {await browser.close();}
+}
+module.exports = {generate, normalize, findChrome, addHeadingTargets};
diff --git a/scripts/docs-pdf/generate.sh b/scripts/docs-pdf/generate.sh
new file mode 100755
index 000000000000..2bc822f1e012
--- /dev/null
+++ b/scripts/docs-pdf/generate.sh
@@ -0,0 +1,48 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+usage() {
+ cat <<'EOF'
+Usage:
+ ./scripts/docs-pdf/generate.sh <version> [locale] [options]
+ ./scripts/docs-pdf/generate.sh --version <version> [options]
+ ./scripts/docs-pdf/generate.sh --list-versions
+
+Examples:
+ ./scripts/docs-pdf/generate.sh 4.x zh-CN
+ ./scripts/docs-pdf/generate.sh dev
+ ./scripts/docs-pdf/generate.sh 2.1 en --output ./output/pdf/2.1-english
+
+Locale: en, zh-CN, or en,zh-CN. Omit it to export both languages.
+Additional options are passed to yarn docs:pdf.
+Relative paths are resolved from the repository root.
+Run yarn docs:pdf --help for the complete option list.
+EOF
+}
+
+case "${1:-}" in
+ '') usage >&2; exit 2 ;;
+ -h|--help) usage; exit 0 ;;
+esac
+
+docs_pdf_repo_root="$(CDPATH='' cd -- "$(dirname --
"${BASH_SOURCE[0]}")/../.." && pwd -P)"
+
+if [[ "$1" != -* ]]; then
+ docs_pdf_version="$1"
+ shift
+ if (( $# > 0 )) && [[ "$1" != -* ]]; then
+ docs_pdf_locale="$1"
+ shift
+ set -- --version "$docs_pdf_version" --locale "$docs_pdf_locale" "$@"
+ else
+ set -- --version "$docs_pdf_version" "$@"
+ fi
+fi
+
+if ! command -v yarn >/dev/null 2>&1; then
+ printf 'yarn is required to run the documentation PDF exporter.\n' >&2
+ exit 127
+fi
+
+cd -- "$docs_pdf_repo_root"
+exec yarn docs:pdf "$@"
diff --git a/scripts/docs-pdf/merge.py b/scripts/docs-pdf/merge.py
new file mode 100644
index 000000000000..e9d566c9883e
--- /dev/null
+++ b/scripts/docs-pdf/merge.py
@@ -0,0 +1,72 @@
+"""Merge print batches, restore cross-batch links and add document
bookmarks."""
+import json
+import sys
+from pathlib import Path
+from pypdf import PdfReader, PdfWriter
+from pypdf.generic import (
+ ArrayObject, DecodedStreamObject, DictionaryObject, FloatObject,
+ NameObject, NumberObject,
+)
+
+job = json.loads(Path(sys.argv[1]).read_text())
+writer = PdfWriter()
+for batch in job['batches']:
+ writer.append(PdfReader(batch), import_outline=False)
+
+prefix = 'https://archive.invalid/'
+targets = {'contents': (1, 40, 800)}
+for index, page in enumerate(writer.pages):
+ for ref in page.get('/Annots', []):
+ link = ref.get_object()
+ uri = str(link.get('/A', {}).get('/URI', ''))
+ if uri.startswith(prefix + 'start/'):
+ rect = link['/Rect']
+ targets[uri[len(prefix + 'start/'):]] = (index, float(rect[0]),
float(rect[3]))
+
+for alias, canonical in job.get('headingAliases', {}).items():
+ if canonical in targets:
+ targets.setdefault(alias, targets[canonical])
+
+for page in writer.pages:
+ for ref in page.get('/Annots', []):
+ link = ref.get_object()
+ uri = str(link.get('/A', {}).get('/URI', ''))
+ if not uri.startswith(prefix):
+ continue
+ key = uri[len(prefix):].removeprefix('start/')
+ target = targets.get(key) or targets.get(key.split('#')[0])
+ if target is None:
+ raise RuntimeError(f'Unresolved internal PDF link: {key}')
+ destination = ArrayObject([writer.pages[target[0]].indirect_reference,
+ NameObject('/XYZ'), FloatObject(target[1]),
+ FloatObject(target[2]), NumberObject(0)])
+ link[NameObject('/A')] = DictionaryObject({NameObject('/S'):
NameObject('/GoTo'),
+ NameObject('/D'):
destination})
+
+missing = [doc['key'] for doc in job['docs'] if doc['key'] not in targets]
+if missing:
+ raise RuntimeError(f'Missing document headings: {missing}')
+writer.add_outline_item(job['contentsTitle'], 1)
+for doc in job['docs']:
+ writer.add_outline_item(doc['title'], targets[doc['key']][0])
+writer.add_metadata({'/Title': job['title'], '/Author': 'Apache Doris',
+ '/Subject': f"Archived documentation, source commit
{job['commit']}",
+ '/Keywords': f"Apache
Doris,{job['version']},{job['locale']},archive"})
+font = writer._add_object(DictionaryObject({NameObject('/Type'):
NameObject('/Font'),
+ NameObject('/Subtype'): NameObject('/Type1'), NameObject('/BaseFont'):
NameObject('/Helvetica')}))
+for index in range(1, len(writer.pages)):
+ page = writer.pages[index]
+ resources = page['/Resources'].get_object()
+ if '/Font' not in resources:
+ resources[NameObject('/Font')] = DictionaryObject()
+ resources['/Font'][NameObject('/ArchiveFooter')] = font
+ stream = DecodedStreamObject()
+ stream.set_data(f'\nQ\nq BT /ArchiveFooter 8 Tf 0.3 0.35 0.32 rg 1 0 0 1
495 22 Tm ({index + 1} / {len(writer.pages)}) Tj ET Q\n'.encode('ascii'))
+ old = page.get('/Contents')
+ contents = old.get_object() if old is not None else None
+ values = list(contents) if isinstance(contents, ArrayObject) else ([old]
if old is not None else [])
+ start = DecodedStreamObject()
+ start.set_data(b'q\n')
+ page[NameObject('/Contents')] = ArrayObject([writer._add_object(start)] +
values + [writer._add_object(stream)])
+writer.write(job['output'])
+print(json.dumps({'pages': len(writer.pages), 'documents': len(job['docs']),
'bookmarks': len(job['docs']) + 1}))
diff --git a/scripts/docs-pdf/print.css b/scripts/docs-pdf/print.css
new file mode 100644
index 000000000000..32fa09fa08b0
--- /dev/null
+++ b/scripts/docs-pdf/print.css
@@ -0,0 +1,27 @@
+@page { size: A4; margin: 18mm 16mm 20mm; }
+html { color: #172b25; background: white; }
+body { font: 10pt/1.55 Arial, 'PingFang SC', 'Hiragino Sans GB', 'Noto Sans
CJK SC', sans-serif; overflow-wrap: anywhere; }
+h1 { font-size: 23pt; line-height: 1.2; }
+h1.doc-title { break-before: page; border-bottom: 2pt solid #11a679;
padding-bottom: 10pt; }
+h2, h3, h4, h5, h6 { break-after: avoid; line-height: 1.3; margin-top: 16pt; }
+h2 { font-size: 16pt; } h3 { font-size: 13pt; } h4, h5, h6 { font-size: 11pt; }
+h1 a, h2 a, h3 a, h4 a, h5 a, h6 a { color: inherit; text-decoration: none; }
+a { color: #087352; text-decoration: underline; }
+pre { font: 8pt/1.4 Menlo, Consolas, 'PingFang SC', 'Noto Sans Mono CJK SC',
monospace; white-space: pre-wrap; overflow-wrap: anywhere; background: #f4f7f6;
padding: 8pt; border: .5pt solid #d9e5df; }
+code { font: .9em Menlo, Consolas, 'PingFang SC', 'Noto Sans Mono CJK SC',
monospace; }
+table { width: 100%; border-collapse: collapse; font-size: 8pt; table-layout:
fixed; margin: 10pt 0; }
+th, td { border: .5pt solid #c4d2cc; padding: 4pt; vertical-align: top;
overflow-wrap: anywhere; }
+th { background: #e8f4ee; } tr { break-inside: avoid; }
+img { max-width: 100%; height: auto; max-height: 220mm; object-fit: contain; }
+figure { margin: 10pt 0; break-inside: avoid; }
+blockquote, .admonition { border-left: 3pt solid #11a679; padding: 6pt 10pt;
margin: 10pt 0; background: #f1f8f5; }
+details > * { display: block; } summary { font-weight: bold; }
+.cover { padding-top: 50mm; text-align: center; break-after: page; }
+.cover img { width: 65mm; margin-bottom: 20mm; }
+.cover h1 { font-size: 32pt; }
+.cover p { color: #465c52; font-size: 12pt; }
+.toc { break-after: page; } .toc ol { padding-left: 2.6em; } .toc li { margin:
2pt 0; }
+.source { font-size: 8pt; color: #576b61; }
+.missing-image { border: 1pt solid #9d7630; padding: 8pt; font-size: 9pt; }
+
+.section-link { font-size: .75em; }
diff --git a/scripts/docs-pdf/requirements.txt
b/scripts/docs-pdf/requirements.txt
new file mode 100644
index 000000000000..fd8ffbb5aa4e
--- /dev/null
+++ b/scripts/docs-pdf/requirements.txt
@@ -0,0 +1 @@
+pypdf>=6.0,<7
diff --git a/scripts/docs-pdf/verify.py b/scripts/docs-pdf/verify.py
new file mode 100644
index 000000000000..19452946ffc7
--- /dev/null
+++ b/scripts/docs-pdf/verify.py
@@ -0,0 +1,37 @@
+"""Check an exported archive against its source-coverage manifest."""
+import argparse
+import hashlib
+import json
+from pathlib import Path
+from pypdf import PdfReader
+
+parser = argparse.ArgumentParser(description=__doc__)
+parser.add_argument('directory', type=Path)
+parser.add_argument('--allow-missing-images', action='store_true')
+args = parser.parse_args()
+directory = args.directory
+manifest = json.loads((directory / 'manifest.json').read_text())
+assert manifest['files'], 'No PDF files in manifest'
+for item in manifest['files']:
+ assert args.allow_missing_images or not item['imageWarnings'],
item['filename']
+ file = directory / item['filename']
+ data = file.read_bytes()
+ assert len(data) == item['bytes'], file
+ assert hashlib.sha256(data).hexdigest() == item['sha256'], file
+ reader = PdfReader(file)
+ assert len(reader.pages) == item['pages'], file
+ assert len(reader.outline) == item['documents'] + 1, file
+ assert len(set(item['sourceFiles'])) == item['documents'], file
+ assert reader.metadata.author == 'Apache Doris', file
+ for page in reader.pages:
+ for ref in page.get('/Annots', []):
+ uri = str(ref.get_object().get('/A', {}).get('/URI', ''))
+ assert 'archive.invalid' not in uri, (file, uri)
+ sample = sorted({index for index in [0, 1, 30, len(reader.pages) // 2,
len(reader.pages) - 1] if index < len(reader.pages)})
+ for index in sample:
+ text = reader.pages[index].extract_text()
+ assert text.strip(), (file, index)
+ assert '\ufffd' not in text, (file, index)
+ print(json.dumps({'file': item['filename'], 'pages': len(reader.pages),
+ 'documents': item['documents'], 'imageWarnings':
len(item['imageWarnings']),
+ 'status': 'pass'}))
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
![https://cdn.selectdb.com/static/doris_logo_512_4903556647.png"](https://cdn.selectdb.com/static/doris_logo_512_4903556647.png"){kind=link}
