This is an automated email from the ASF dual-hosted git repository.
morningman pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/doris-website.git
The following commit(s) were added to refs/heads/master by this push:
new ffb97b065fd ci(i18n): run serial twice-daily translation batches as
regular PRs (#4332)
ffb97b065fd is described below
commit ffb97b065fddd340a8c7d243e86f82620bd982a9
Author: Mingyu Chen (Rayner) <[email protected]>
AuthorDate: Fri Oct 16 17:02:03 2026 +0800
ci(i18n): run serial twice-daily translation batches as regular PRs (#4332)
## Problem and behavior
Translation progress currently waits for maintainers to merge the only
allowed robot batch, and generated PRs are drafts. Run automatic batches
at 04:23 and 16:23 UTC, plus manual dispatch that defaults to generate.
Remove the source-push trigger so automatic runs follow that twice-daily
schedule.
Each enabled run creates a regular PR with `translate-ja` and at most 10
target pages, for maintainers to review and merge. Multiple batches may
remain open. Reserve every document already covered by an open PR,
including stale candidates; invalid or edited target/state records block
their own documents before applying the batch limit. Recheck
reservations for primary and reused Dev/4.x targets. Preserve existing
PRs instead of closing or replacing them.
A stable workflow-level concurrency group covers every trigger, mode and
locale. `cancel-in-progress: false` preserves the active run, while
`queue: max` queues subsequent runs. Discovery checks out the latest
master when a queued run starts; generation reconciles again and
verifies source/target and master freshness before publication. The
local simulator now accommodates the full native scan and accounts for
multiple model chunks.
## Validation
- Node 20.20.2: all 107 i18n tests passed, including stale reservations,
close-and-retry, multiple open batches, blocked targets before the
limit, generation of ten other pages, and reserved Dev/4.x partner
protection.
- Full local Git/GitHub/Codex simulation passed: malformed response
rejected; multi-chunk candidate accepted; translation gate passed; only
the target and its page state committed; regular PR and translate-ja
label verified. No external model or GitHub writes in this simulation.
- Existing master concurrency API observed one run in progress and
another pending. Both read-only probes were then cancelled; the same
group is retained by this change. The new queue setting is checked
against current GitHub documentation and is pending deployment.
- Local translation gate, five Python link-resolver tests,
YAML/trusted-checkout checks and changed-file size checks passed.
- Final commit differs from the tested runtime commit only in README
wording. GitHub CI for the latest head is still waiting for runners.
Continuous generation remains disabled until this change is merged and
enabled. The real one-page model/PAT/PR/CI acceptance already passed in
#4330. No website rendering, English/Chinese source content, model
credentials or token scopes change.
---------
Co-authored-by: morningman <[email protected]>
---
.github/workflows/japanese-translation.yml | 37 +++++------------
scripts/i18n/__tests__/planner.test.cjs | 28 ++++++++++---
scripts/i18n/__tests__/workflow.test.cjs | 50 +++++++++++++++-------
scripts/i18n/plan-translations.cjs | 7 +++-
scripts/i18n/simulate-workflow.cjs | 14 ++++---
scripts/i18n/workflow-generate.cjs | 11 ++---
scripts/i18n/workflow-prs.cjs | 2 +-
scripts/i18n/workflow-publish.cjs | 13 ++----
scripts/i18n/workflow-scan.cjs | 13 +-----
translations/README.md | 67 +++++++++++++++++++-----------
10 files changed, 134 insertions(+), 108 deletions(-)
diff --git a/.github/workflows/japanese-translation.yml
b/.github/workflows/japanese-translation.yml
index 29b89b2005c..37c66792004 100644
--- a/.github/workflows/japanese-translation.yml
+++ b/.github/workflows/japanese-translation.yml
@@ -7,7 +7,7 @@ on:
description: Read-only dry run, one-page live validation, or enabled
batch
type: choice
options: [dry-run, validate, generate]
- default: dry-run
+ default: generate
required: true
plugin:
description: Optional plugin filter (default or community)
@@ -21,33 +21,19 @@ on:
locale:
description: Optional translation locale; required once several are
managed
type: string
- push:
- branches: [master]
- paths:
- - 'docs/**'
- - 'versioned_docs/version-4.x/**'
- - 'community/**'
- - 'i18n/zh-CN/docusaurus-plugin-content-docs/current/**'
- - 'i18n/zh-CN/docusaurus-plugin-content-docs/version-4.x/**'
- - 'i18n/zh-CN/docusaurus-plugin-content-docs-community/current/**'
- - 'translations/config.json'
- - 'translations/scope/**'
- - 'translations/glossary/**'
- - 'sidebars.ts'
- - 'sidebarsCommunity.json'
- - 'versioned_sidebars/version-4.x-sidebars.json'
- - 'docusaurus.config.js'
- - 'config/i18n.js'
schedule:
- - cron: '23 4 * * *'
+ - cron: '23 4,16 * * *'
permissions:
contents: read
pull-requests: read
concurrency:
+ # One entire run across scheduled/manual triggers, locales and modes.
+ # Keep this stable so an in-flight run also holds the same lock.
group: japanese-translation-ja
cancel-in-progress: false
+ queue: max
jobs:
discover:
@@ -62,6 +48,7 @@ jobs:
steps:
- uses: actions/checkout@v4
with:
+ ref: master
fetch-depth: 0
persist-credentials: false
- uses: actions/setup-node@v4
@@ -77,8 +64,7 @@ jobs:
- name: Plan all current and historical pending documents
id: scan
env:
- GITHUB_EVENT_BEFORE: ${{ github.event.before }}
- I18N_MODE: ${{ inputs.mode || 'dry-run' }}
+ I18N_MODE: ${{ inputs.mode || 'generate' }}
I18N_PLUGIN: ${{ inputs.plugin }}
I18N_VERSION: ${{ inputs.version }}
I18N_PATH: ${{ inputs.path }}
@@ -122,7 +108,7 @@ jobs:
APP_ID: ${{ secrets.I18N_GITHUB_APP_ID }}
APP_PRIVATE_KEY: ${{ secrets.I18N_GITHUB_APP_PRIVATE_KEY }}
I18N_GITHUB_PAT: ${{ secrets.I18N_GITHUB_PAT }}
- I18N_MODE: ${{ inputs.mode || 'dry-run' }}
+ I18N_MODE: ${{ inputs.mode || 'generate' }}
run: |
if [[ -n "$APP_ID" && -n "$APP_PRIVATE_KEY" ]]; then
echo 'auth_kind=app' >> "$GITHUB_OUTPUT"
@@ -152,8 +138,7 @@ jobs:
if: steps.config.outputs.ready == 'true'
id: fresh
env:
- GITHUB_EVENT_BEFORE: ${{ github.event.before }}
- I18N_MODE: ${{ inputs.mode || 'dry-run' }}
+ I18N_MODE: ${{ inputs.mode || 'generate' }}
I18N_PLUGIN: ${{ inputs.plugin }}
I18N_VERSION: ${{ inputs.version }}
I18N_PATH: ${{ inputs.path }}
@@ -216,7 +201,7 @@ jobs:
repositories: doris-website
permission-contents: write
permission-pull-requests: write
- - name: Open draft PR with the configured write identity
+ - name: Open reviewable PR with the configured write identity
if: steps.translation.outputs.created == 'true' &&
steps.config.outputs.auth_kind != 'none'
env:
GH_TOKEN: ${{ steps.app.outputs.token || secrets.I18N_GITHUB_PAT }}
@@ -224,7 +209,7 @@ jobs:
run: node scripts/i18n/workflow-publish.cjs
"$RUNNER_TEMP/japanese-translation"
- name: Report staged validation without GitHub write credentials
if: steps.translation.outputs.created == 'true' &&
steps.config.outputs.auth_kind == 'none'
- run: echo 'One-page Codex candidate is in the generation artifact. A
GitHub App or personal token is required for a draft PR.' >>
"$GITHUB_STEP_SUMMARY"
+ run: echo 'One-page Codex candidate is in the generation artifact. A
GitHub App or personal token is required for a translation PR.' >>
"$GITHUB_STEP_SUMMARY"
- name: Sync refreshed Codex auth without overwriting a newer snapshot
if: always() && steps.auth.outcome == 'success'
env:
diff --git a/scripts/i18n/__tests__/planner.test.cjs
b/scripts/i18n/__tests__/planner.test.cjs
index fbddcfd1b94..5b3baa377bd 100644
--- a/scripts/i18n/__tests__/planner.test.cjs
+++ b/scripts/i18n/__tests__/planner.test.cjs
@@ -57,22 +57,40 @@ test('modified completed source is an update and valid
active PR suppresses sele
assert.deepEqual(result.selected, []);
});
-test('stale active PR is reported and a later source revision reenters
selection', async () => {
+test('stale active PR stays excluded until closed, without adding merged
coverage', async () => {
const changed = page('edited', {state: 'stale_unknown'});
const active = {schemaVersion: 1, candidates: [{pluginId: 'default',
version: 'current', docId: 'edited',
locale: 'ja', sourcePath: changed.sourcePath, sourceHash:
hash('older'), pr: '#123'}]};
const result = await run([changed], {old: [{...changed, sourceHash:
hash('older'), sourceCommit: baselineSha}], active});
assert.equal(result.exceptions.some(item => item.kind ===
'stale_active_candidate'), true);
- assert.equal(result.selected.length, 1);
+ assert.equal(result.tasks[0].activePr, '#123');
+ assert.equal(result.counts.activeCandidates, 1);
+ assert.equal(result.counts.mergedCurrent, 0);
+ assert.deepEqual(result.selected, []);
+ const afterClose = await run([changed], {old: [{...changed, sourceHash:
hash('older'), sourceCommit: baselineSha}]});
+ assert.equal(afterClose.selected.length, 1);
});
-test('an active PR for an old path cannot suppress a moved page with the same
source hash', async () => {
+test('an active PR for an old path reserves the same document identity until
resolved', async () => {
const changed = page('new', {sourcePath: 'docs/new.md'});
const active = {schemaVersion: 1, candidates: [{pluginId: 'default',
version: 'current', docId: 'new',
locale: 'ja', sourcePath: 'docs/old.md', sourceHash:
changed.sourceHash, pr: '#123'}]};
const result = await run([changed], {active});
- assert.equal(result.tasks[0].activePr, null);
- assert.equal(result.selected.length, 1);
+ assert.equal(result.tasks[0].activePr, '#123');
+ assert.deepEqual(result.selected, []);
+ assert.equal(result.exceptions[0].kind, 'stale_active_candidate');
+});
+
+test('multiple open PRs reserve only their documents and blocked targets do
not consume the batch limit', async () => {
+ const pages = [page('a'), page('b'), page('c'), page('d')];
+ const active = {schemaVersion: 1, candidates: pages.slice(0, 2).map((p, i)
=> ({...p, locale: 'ja',
+ sourceHash: i === 0 ? p.sourceHash : hash('older'), pr: `#${i + 1}`})),
+ blockedTargets: [pages[2].targetPath]};
+ const result = await run(pages, {active, limit: 1});
+ assert.equal(result.counts.pending, 4);
+ assert.equal(result.counts.mergedCurrent, 0);
+ assert.equal(result.counts.activeCandidates, 2);
+ assert.deepEqual(result.selected, [result.tasks.find(task => task.docId
=== 'd').id]);
});
test('same identity path move includes cleanup and only reuses a verified
target', async () => {
diff --git a/scripts/i18n/__tests__/workflow.test.cjs
b/scripts/i18n/__tests__/workflow.test.cjs
index fc2825f1cf7..80f443034c3 100644
--- a/scripts/i18n/__tests__/workflow.test.cjs
+++ b/scripts/i18n/__tests__/workflow.test.cjs
@@ -8,7 +8,7 @@ const {hash, loadConfig, stateFileFor, loadRecords, identity} =
require('../stat
const {structure} = require('../structure.cjs');
const {readState, applyCandidates} = require('../translate-docs.cjs');
const {reconcileOpenPrs} = require('../workflow-prs.cjs');
-const {relevantPush, replaceableBotPr} = require('../workflow-scan.cjs');
+const {relevantPush} = require('../workflow-scan.cjs');
const {pairFor, machineOwned, generate, allowGeneration} =
require('../workflow-generate.cjs');
const {expectedFiles, authorFor} = require('../workflow-publish.cjs');
@@ -113,15 +113,6 @@ test('two open PRs for one source are flagged and robot
branch is identified', a
assert.equal(result.exceptions[0].kind, 'duplicate_open_pr_candidate');
});
-test('an untouched stale robot PR may be replaced, but a human edit blocks
replacement', () => {
- const sha = 'a'.repeat(40);
- const plan = {exceptions: [{kind: 'stale_active_candidate', pr: '#21'}]};
- const bot = {number: 21, headSha: sha, expectedHead: sha, untouched: true};
- assert.equal(replaceableBotPr(plan, {botDetails: [bot]}), bot);
- assert.equal(replaceableBotPr(plan, {botDetails: [{...bot, untouched:
false}]}), null);
- assert.equal(replaceableBotPr({exceptions: []}, {botDetails: [bot]}),
null);
-});
-
test('only exact current Dev/4.x identity, path and source hash are paired',
() => {
const dev = {pluginId: 'default', version: 'current', docId: 'guide',
kind: 'translate_backlog',
sourcePath: 'docs/guide.md', sourceHash};
@@ -282,6 +273,28 @@ async function updateFixture(t, {sharded = true, human =
false} = {}) {
calls: () => calls};
}
+test('a paired target reserved by an open PR is not copied even when the
supplied plan omits that reservation', async t => {
+ const fixture = await updateFixture(t);
+ const sourcePath = 'versioned_docs/version-4.x/guide.md';
+ fixture.put(sourcePath, fs.readFileSync(path.join(fixture.root,
fixture.task.sourcePath), 'utf8'));
+ fixture.git('add', '.');
+ fixture.git('-c', 'user.name=Test', '-c', '[email protected]',
'commit', '-qm', 'paired source');
+ const sourceCommit = fixture.git('rev-parse', 'HEAD');
+ const primary = {...fixture.task, sourceCommit};
+ const partner = {...primary, version: '4.x', groupId: '4X', id:
'T06-4X-000000000000002',
+ kind: 'translate_new', sourcePath, targetHash: null,
+ targetPath:
'i18n/ja/docusaurus-plugin-content-docs/version-4.x/guide.md'};
+ const manifest = await fixture.run({maxPages: 2, ensureFresh: () => {},
+ plan: {...fixture.plan, sourceCommit, tasks: [primary, partner]},
+ reconciliation: {botPrs: [22], blockedTargets: [],
+ active: {candidates: [{...partner, sourceHash: hash('older
source'), pr: '#22'}]}}});
+ assert.equal(manifest.candidates.length, 1);
+ assert.equal(manifest.candidates[0].version, 'current');
+ assert.equal(fixture.calls(), 1);
+ assert(!fs.existsSync(path.join(fixture.root, partner.targetPath)));
+ assert(!fs.existsSync(path.join(fixture.root, stateFileFor(partner))));
+});
+
test('an existing page shard is updated without rewriting or trusting its
obsolete legacy copy', async t => {
const fixture = await updateFixture(t);
const {root, legacyPath, shardPath, task} = fixture;
@@ -414,7 +427,7 @@ test('publication deduplicates shared state and rejects an
unrelated page shard'
[record.targetPath, second.targetPath,
'translations/state/ja/docs.json'].sort());
});
-test('normal generation defaults to ten target pages and leaves the next task
untouched', async t => {
+test('normal generation skips open PRs and blocked targets, then generates ten
other pages', async t => {
const root = fs.mkdtempSync(path.join(os.tmpdir(),
'translation-ten-page-batch-'));
t.after(() => fs.rmSync(root, {recursive: true, force: true}));
const put = (file, text) => {
@@ -424,7 +437,7 @@ test('normal generation defaults to ten target pages and
leaves the next task un
put('translations/scope/ja.json', JSON.stringify(scope));
put('translations/state/ja/docs.json', JSON.stringify({schemaVersion: 1,
pages: []}));
put('translations/state/ja/shards.schema.json',
JSON.stringify({schemaVersion: 1, format: 'page-record-v1'}));
- const tasks = Array.from({length: 11}, (_, index) => {
+ const tasks = Array.from({length: 14}, (_, index) => {
const source = `# Guide ${index}\n\nA guide.\n`;
const sourcePath = `docs/page-${index}.md`;
put(sourcePath, source);
@@ -438,6 +451,7 @@ test('normal generation defaults to ten target pages and
leaves the next task un
git('init', '-q'); git('add', '.');
git('-c', 'user.name=Test', '-c', '[email protected]', 'commit',
'-qm', 'eleven pending pages');
const sourceCommit = git('rev-parse', 'HEAD');
+ tasks[0].activePr = '#21';
let calls = 0;
const adapter = {model: 'fixture', usedTokens: 0, translate: async
({units}) => {
calls++;
@@ -445,12 +459,18 @@ test('normal generation defaults to ten target pages and
leaves the next task un
}};
const manifest = await generate({rootDir: root, target, output:
path.join(root, 'output'),
plan: {schemaVersion: 1, locale: 'ja', sourceCommit, selected:
tasks.map(task => task.id), tasks, exceptions: []},
- reconciliation: {botPrs: [], botDetails: [], blockedTargets: []},
ensureFresh: () => {},
+ reconciliation: {botPrs: [21, 22], botDetails: [{number: 21,
untouched: false}, {number: 22, untouched: true}],
+ active: {candidates: [{...tasks[1], sourceHash: hash('older
source'), pr: '#22'}]},
+ blockedTargets: [tasks[2].targetPath]}, ensureFresh: () => {},
adapter, glossary: require('../../../translations/glossary/ja.json')});
assert.equal(manifest.candidates.length, 10);
assert.equal(manifest.failures.length, 0);
assert.equal(calls, 10);
assert.equal(loadRecords(root, 'ja').size, 10);
- assert(!fs.existsSync(path.join(root, tasks[10].targetPath)));
- assert(!fs.existsSync(path.join(root, stateFileFor(tasks[10]))));
+ assert.deepEqual(manifest.candidates.map(item => item.id), tasks.slice(3,
13).map(item => item.id));
+ for (const task of [tasks[0], tasks[1], tasks[2], tasks[13]]) {
+ assert(!fs.existsSync(path.join(root, task.targetPath)));
+ assert(!fs.existsSync(path.join(root, stateFileFor(task))));
+ }
+ assert.equal(manifest.replacesPr, undefined);
});
diff --git a/scripts/i18n/plan-translations.cjs
b/scripts/i18n/plan-translations.cjs
index 1f2306b1151..03a3a6ee601 100644
--- a/scripts/i18n/plan-translations.cjs
+++ b/scripts/i18n/plan-translations.cjs
@@ -162,7 +162,9 @@ async function plan({report, scope, records, baseline = [],
active, readCurrent,
targetHash: page.targetHash, state: page.state, reviewStatus:
page.reviewStatus,
estimatedSourceTokens: estimate(source), requiresHumanReview:
protectedTask,
sourceChange: page.change,
- activePr: activeValid ? candidate.pr : null,
+ // An open PR owns this identity until it is merged or closed,
+ // even if its source revision or path has become stale.
+ activePr: candidate?.pr ?? null,
...(moveFrom ? {previousDocId: moveFrom.docId, reuseTarget:
moveFrom.sourceHash === page.sourceHash
&& !!moveFrom.targetHash && !!moveFrom.targetPath
&& readCurrent(moveFrom.targetPath) !== null &&
hash(readCurrent(moveFrom.targetPath)) === moveFrom.targetHash,
@@ -200,9 +202,10 @@ async function plan({report, scope, records, baseline =
[], active, readCurrent,
&& (!filters.path || task.sourcePath.startsWith(filters.path) ||
task.targetPath?.startsWith(filters.path)));
const selected = [];
let tokens = 0;
+ const blockedTargets = new Set(active?.blockedTargets ?? []);
for (const task of selectedTasks) {
if (selected.length >= limit) break;
- if (task.activePr || task.requiresHumanReview) continue;
+ if (task.activePr || task.requiresHumanReview ||
blockedTargets.has(task.targetPath)) continue;
if (tokenBudget !== null && tokens + task.estimatedSourceTokens >
tokenBudget) continue;
selected.push(task.id);
tokens += task.estimatedSourceTokens;
diff --git a/scripts/i18n/simulate-workflow.cjs
b/scripts/i18n/simulate-workflow.cjs
index 9422b3a3c70..359e14c1092 100644
--- a/scripts/i18n/simulate-workflow.cjs
+++ b/scripts/i18n/simulate-workflow.cjs
@@ -50,13 +50,13 @@ if (args[0] === 'api' && args.some(arg =>
arg.startsWith('repos/apache/doris-web
process.exit(0);
} else if (args[0] === 'pr' && args[1] === 'create') {
const value = flag => args[args.indexOf(flag) + 1];
- assert(args.includes('--draft'));
+ assert(!args.includes('--draft'));
assert.equal(value('--base'), 'master');
assert.equal(value('--repo'), 'apache/doris-website');
assert.equal(value('--label'), 'translate-ja');
const body = fs.readFileSync(value('--body-file'), 'utf8');
assert.match(body, /japanese-translation-automation:v1
bot-head=[a-f0-9]{40}/);
- const record = {draft: true, base: value('--base'), head: value('--head'),
labels: [value('--label')], body,
+ const record = {draft: false, base: value('--base'), head:
value('--head'), labels: [value('--label')], body,
url: 'https://example.invalid/apache/doris-website/pull/1'};
fs.writeFileSync(process.env.I18N_SIM_PR_RECORD, JSON.stringify(record,
null, 2) + '\\n');
process.stdout.write(record.url + '\\n');
@@ -68,7 +68,7 @@ if (args[0] === 'api' && args.some(arg =>
arg.startsWith('repos/apache/doris-web
function runNode(root, script, args, env, logDir, expectedStatus = 0) {
const result = spawnSync(process.execPath, [path.join(root, script),
...args], {
- cwd: root, env, encoding: 'utf8', timeout: 300000, maxBuffer: 16 *
1024 * 1024,
+ cwd: root, env, encoding: 'utf8', timeout: 1200000, maxBuffer: 16 *
1024 * 1024,
});
const name = path.basename(script, '.cjs') + (env.I18N_SIM_PHASE || '');
fs.writeFileSync(path.join(logDir, `${name}.log`), `${result.stdout ||
''}${result.stderr || ''}`);
@@ -110,7 +110,7 @@ function main() {
GITHUB_RUN_ID: runId, I18N_PR_AUTH_KIND: 'pat', I18N_MODE: 'validate',
I18N_VALIDATION_RUN: 'true', I18N_PATH: sourcePath, I18N_MAX_PAGES:
'1',
I18N_LIMIT: '20', DORIS_ENABLE_JA: 'true', CODEX_HOME: auth,
- I18N_SIM_PR_RECORD: path.join(output, 'draft-pr.json')};
+ I18N_SIM_PR_RECORD: path.join(output, 'pull-request.json')};
runNode(repo, 'scripts/i18n/workflow-prs.cjs', [scan], env, logs);
runNode(repo, 'scripts/i18n/workflow-scan.cjs', [scan], env, logs);
const plan = JSON.parse(fs.readFileSync(path.join(scan, 'plan.json'),
'utf8'));
@@ -128,7 +128,9 @@ function main() {
const generation = JSON.parse(fs.readFileSync(path.join(scan,
'generation.json'), 'utf8'));
assert.equal(generation.candidates.length, 1);
assert.equal(generation.failures.length, 0);
- assert.equal(generation.usedTokens, 100000);
+ const requests = generation.candidates.reduce((sum, candidate) => sum +
candidate.requests, 0);
+ assert(Number.isInteger(requests) && requests > 0);
+ assert.equal(generation.usedTokens, requests * 100000);
runNode(repo, 'scripts/i18n/workflow-publish.cjs', [scan], env, logs);
const pr = JSON.parse(fs.readFileSync(env.I18N_SIM_PR_RECORD, 'utf8'));
assert.equal(pr.head, `i18n/ja-automated-${runId}`);
@@ -146,7 +148,7 @@ function main() {
sourceCommit: plan.sourceCommit, selected: plan.selected, sourcePath,
invalidModel: {exitCode: 1, error: failure.failures[0].error,
usedTokens: failure.usedTokens},
generation: {candidates: generation.candidates.length, usedTokens:
generation.usedTokens},
- draftPr: {url: pr.url, branch: pr.head, commit: branch, files:
committedFiles, labels: pr.labels}};
+ pullRequest: {url: pr.url, draft: pr.draft, branch: pr.head, commit:
branch, files: committedFiles, labels: pr.labels}};
fs.writeFileSync(path.join(output, 'report.json'), JSON.stringify(report,
null, 2) + '\n');
process.stdout.write(JSON.stringify(report, null, 2) + '\n');
}
diff --git a/scripts/i18n/workflow-generate.cjs
b/scripts/i18n/workflow-generate.cjs
index 35276255640..7ccb6ec49d2 100644
--- a/scripts/i18n/workflow-generate.cjs
+++ b/scripts/i18n/workflow-generate.cjs
@@ -8,7 +8,6 @@ const {hash, identity, languageName, pipelineLocale} =
require('./state.cjs');
const {checkedTask, managedPath, safeFile} =
require('./translation-engine.cjs');
const {codexAdapter} = require('./codex-adapter.cjs');
const {prepare, applyCandidates, readState, statePathForRecord, targetFor} =
require('./translate-docs.cjs');
-const {replaceableBotPr} = require('./workflow-scan.cjs');
const {markerPrefixFor} = require('./automation-naming.cjs');
const root = path.resolve(__dirname, '../..');
@@ -86,24 +85,23 @@ async function generate({plan, reconciliation, maxPages =
10, target,
assert(Number.isInteger(maxPages) && maxPages > 0 && maxPages <= 20,
'Invalid page limit');
assert(plan.schemaVersion === 1 && plan.sourceCommit ===
execFileSync('git', ['rev-parse', 'HEAD'],
{cwd: rootDir, encoding: 'utf8'}).trim(), 'Plan does not match trusted
checkout');
- const replaces = replaceableBotPr(plan, reconciliation);
- assert(reconciliation.botPrs.length === 0 || replaces,
- 'An automated translation PR is already open or was edited after
generation');
const scope = JSON.parse(fs.readFileSync(path.join(rootDir,
`translations/scope/${target.locale}.json`), 'utf8'));
const state = readState(rootDir, target.locale);
const blocked = new Set(reconciliation.blockedTargets);
+ const active = new Set((reconciliation.active?.candidates ??
[]).map(identity));
const candidates = [], failures = [], consumed = new Set();
for (const id of plan.selected) {
if (candidates.length >= maxPages) break;
const task = plan.tasks.find(item => item.id === id);
- if (!task || consumed.has(id) || blocked.has(task.targetPath))
continue;
+ if (!task || consumed.has(id) || task.activePr ||
active.has(identity(task))
+ || blocked.has(task.targetPath)) continue;
consumed.add(id);
if (!machineOwned(task, state)) {
failures.push({id, sourcePath: task.sourcePath, error: 'Human
maintained target requires manual review'});
continue;
}
let partner = pairFor(task, plan.tasks, scope);
- if (partner && (blocked.has(partner.targetPath) ||
!machineOwned(partner, state)
+ if (partner && (active.has(identity(partner)) ||
blocked.has(partner.targetPath) || !machineOwned(partner, state)
|| candidates.length + 2 > maxPages)) partner = null;
try {
checkedTask(plan, id, scope, rootDir);
@@ -125,7 +123,6 @@ async function generate({plan, reconciliation, maxPages =
10, target,
locale: target.locale, language: target.language,
model: adapter.model, glossaryVersion: glossary.version, liveCall:
adapter.usedTokens > 0,
liveVerified: false, usedTokens: adapter.usedTokens,
- replacesPr: replaces ? {number: replaces.number, headSha:
replaces.headSha} : null,
candidates: candidates.map(item => ({id: item.task.id, kind:
item.task.kind,
pluginId: item.task.pluginId, version: item.task.version, docId:
item.task.docId,
sourcePath: item.task.sourcePath, sourceHash: item.task.sourceHash,
diff --git a/scripts/i18n/workflow-prs.cjs b/scripts/i18n/workflow-prs.cjs
index e874bec8ea7..8f90ceee1a7 100644
--- a/scripts/i18n/workflow-prs.cjs
+++ b/scripts/i18n/workflow-prs.cjs
@@ -124,7 +124,7 @@ async function reconcileOpenPrs({scope, language, pulls,
filesFor, readHead, rea
}
const candidates = [...active.values()].filter(candidate => !blocked.has(
managedPath(scope, {...candidate, targetPath: targetFor(scope,
candidate)})));
- return {schemaVersion: 1, active: {schemaVersion: 1, candidates},
blockedTargets: ordered(blocked),
+ return {schemaVersion: 1, active: {schemaVersion: 1, candidates,
blockedTargets: ordered(blocked)}, blockedTargets: ordered(blocked),
botPrs: botPrs.sort((a, b) => a - b), botDetails, openPrCount:
pulls.length,
relevantPrs: relevant.sort((a, b) => a - b), exceptions};
}
diff --git a/scripts/i18n/workflow-publish.cjs
b/scripts/i18n/workflow-publish.cjs
index 5b26f5b8fe2..69a7b54c43a 100644
--- a/scripts/i18n/workflow-publish.cjs
+++ b/scripts/i18n/workflow-publish.cjs
@@ -54,6 +54,7 @@ function main() {
const kind = process.env.I18N_PR_AUTH_KIND;
assert(kind === 'app' || kind === 'pat', 'Unsupported GitHub write
identity');
const manifest = JSON.parse(fs.readFileSync(path.join(output,
'generation.json'), 'utf8'));
+ assert(!manifest.replacesPr, 'Preserve existing translation PRs; rescan
with the current generator');
const locale = pipelineLocale(root);
const language = languageName(root, locale);
const scope = JSON.parse(fs.readFileSync(path.join(root,
`translations/scope/${locale}.json`), 'utf8'));
@@ -81,21 +82,13 @@ function main() {
// gh configures Git's credential helper from GH_TOKEN. The token is never
// placed in a remote URL, command argument, log or generated file.
execFileSync('gh', ['auth', 'setup-git'], {cwd: root, stdio: 'inherit'});
- if (manifest.replacesPr) {
- const old = JSON.parse(execFileSync('gh', ['pr', 'view',
String(manifest.replacesPr.number),
- '--repo', 'apache/doris-website', '--json', 'headRefOid,state'],
{cwd: root, encoding: 'utf8'}));
- assert(old.state === 'OPEN' && old.headRefOid ===
manifest.replacesPr.headSha,
- 'Previous robot PR was edited or closed; preserve it and rescan');
- execFileSync('gh', ['pr', 'close', String(manifest.replacesPr.number),
- '--repo', 'apache/doris-website'], {cwd: root, stdio: 'inherit'});
- }
git('push', 'origin', `HEAD:refs/heads/${branch}`);
const url = execFileSync('gh', ['pr', 'create', '--repo',
'apache/doris-website', '--base', 'master',
- '--head', branch, '--draft', '--label', `translate-${locale}`,
+ '--head', branch, '--label', `translate-${locale}`,
'--title', `docs(i18n): machine ${language} translation candidates`,
'--body-file', bodyFile], {cwd: root, encoding: 'utf8'}).trim();
if (process.env.GITHUB_STEP_SUMMARY)
fs.appendFileSync(process.env.GITHUB_STEP_SUMMARY,
- `\nOpened draft translation PR: ${url}; ${manifest.candidates.length}
candidate pages.\n`);
+ `\nOpened translation PR for review: ${url};
${manifest.candidates.length} candidate pages.\n`);
process.stdout.write(`${url}\n`);
}
diff --git a/scripts/i18n/workflow-scan.cjs b/scripts/i18n/workflow-scan.cjs
index 13b5b3f42a1..eea629a629d 100644
--- a/scripts/i18n/workflow-scan.cjs
+++ b/scripts/i18n/workflow-scan.cjs
@@ -47,7 +47,6 @@ function summarize(plan, reconciliation, eligible, mode,
language) {
`- Open PRs inspected: ${reconciliation.openPrCount}; relevant:
${reconciliation.relevantPrs.length}; active candidates:
${plan.counts.activeCandidates}`,
`- Selected for a possible next batch: ${plan.selected.length};
blocked targets: ${reconciliation.blockedTargets.length}`,
`- Robot batch PRs already open: ${reconciliation.botPrs.join(', ') ||
'none'}`,
- `- Untouched stale robot PR eligible for replacement:
${replaceableBotPr(plan, reconciliation)?.number || 'none'}`,
`- Exceptions: ${[...exceptionKinds].map(([kind, count]) =>
`${kind}=${count}`).join(', ') || 'none'}`,
'',
'This report is read only. Open PR candidates do not count as merged
translations.',
@@ -55,13 +54,6 @@ function summarize(plan, reconciliation, eligible, mode,
language) {
].join('\n');
}
-function replaceableBotPr(plan, reconciliation) {
- if (reconciliation.botDetails?.length !== 1 ||
!reconciliation.botDetails[0].untouched) return null;
- const bot = reconciliation.botDetails[0];
- return plan.exceptions.some(item => item.kind === 'stale_active_candidate'
&& item.pr === `#${bot.number}`)
- ? bot : null;
-}
-
function main() {
const output = process.argv[2];
assert(output && process.argv.length === 3, 'Usage: workflow-scan.cjs
<output-directory>');
@@ -103,12 +95,11 @@ function main() {
if (process.env.GITHUB_STEP_SUMMARY)
fs.appendFileSync(process.env.GITHUB_STEP_SUMMARY, summary);
else process.stdout.write(summary);
if (process.env.GITHUB_OUTPUT) fs.appendFileSync(process.env.GITHUB_OUTPUT,
- `can_generate=${eligible && plan.selected.length > 0
- && (reconciliation.botPrs.length === 0 || !!replaceableBotPr(plan,
reconciliation))}\n`
+ `can_generate=${eligible && plan.selected.length > 0}\n`
+ `source_commit=${plan.sourceCommit}\n`);
}
if (require.main === module) {
try {main();} catch (error) {console.error(error.message);
process.exitCode = 1;}
}
-module.exports = {relevantPush, changedPaths, summarize, replaceableBotPr,
rulesFor};
+module.exports = {relevantPush, changedPaths, summarize, rulesFor};
diff --git a/translations/README.md b/translations/README.md
index 668878fa167..252cc7f5517 100644
--- a/translations/README.md
+++ b/translations/README.md
@@ -21,7 +21,7 @@ shows the raw context path.
The translation workflow's `validate` mode selects one exact source page for a
live check. Normal `generate` runs produce at most 10 target pages per batch,
including any reused Dev/4.x partner; the planner can consider up to 20 tasks.
-Automated draft PRs receive the `translate-<locale>` repository label, such as
+Automated PRs receive the `translate-<locale>` repository label, such as
`translate-ja`. Create that label before enabling publication for a new locale.
Build Check installs dependencies and runs the i18n unit tests before the
@@ -200,11 +200,14 @@ file with the reviewed PR candidates:
}
```
-Use `--active-candidates /path/to/open-prs.json`. A matching source hash marks
-the task as active and removes it from `selected`, but does not add coverage.
-A source change makes the PR candidate stale and requeues the page. The planner
-does not query GitHub: a caller that wants complete PR deduplication must
supply
-all active candidate PRs, including manually opened T09 PRs.
+Use `--active-candidates /path/to/open-prs.json`. An open candidate reserves
its
+document identity and removes it from `selected`, but does not add coverage.
+A changed source hash or path reports a stale candidate and keeps the document
+reserved until the PR is merged or closed. The input may also include a
+`blockedTargets` array; those paths are excluded before the batch limit is
+applied. The planner does not query GitHub: a caller that wants complete PR
+deduplication must supply all active candidate PRs, including manually opened
+T09 PRs.
The plan also reports `zh_only_change` exceptions when a committed Chinese
counterpart changed since the T03 baseline while its English fact source did
@@ -304,29 +307,37 @@ review are required before T08 enables automatic
generation.
## Asynchronous Japanese discovery and generation
-`.github/workflows/japanese-translation.yml` runs a read-only scan after
-relevant master source pushes, each day, and on manual dispatch. It recounts
+`.github/workflows/japanese-translation.yml` runs at 04:23 and 16:23 UTC daily
+and on manual dispatch. It has no push trigger. Every run first recounts
the current native Dev, 4.x and community documents, reads every open master
PR's changed files, and checks Japanese target files against that PR's state
records. Each run uploads `plan.json`, `coverage.json`,
`active-candidates.json` and `open-pr-reconciliation.json` and writes a short
job summary. Open PR candidates stay pending until merged. A changed source
-hash makes an older PR candidate stale and queues the page again. A PR whose
+hash makes an older PR candidate stale, but that document remains reserved
+until its PR is merged or closed. The workflow reports it for maintainer
+attention and neither closes nor overwrites the existing PR. A PR whose
candidate cannot be verified blocks that target from robot generation and
appears in the exception report.
-Manual dispatch defaults to `dry-run` and accepts optional plugin, version and
+Manual dispatch defaults to `generate` and accepts optional plugin, version and
source path filters. `validate` mode requires one exact English source path
-ending in `.md` or `.mdx`; it can call the model and create a draft PR for one
+ending in `.md` or `.mdx`; it can call the model and create a regular PR for
one
page while ongoing automation remains disabled. It is the first live acceptance
path after OSS model access is available. With OSS access but no GitHub write
credential, `validate` still
writes the candidate to the generation artifact and makes no GitHub write.
-Pushes are checked against the current registered source
-identities before any generation. The daily full scan recovers missed or
-cancelled pushes. Japanese target or state-only commits do not trigger the
-push listener. The workflow does not participate in website builds or release
-deployment.
+`dry-run` remains available for read-only discovery. Full scans recover
+pending work from new or changed source documents and earlier failed runs.
+One workflow-level concurrency group covers every trigger, mode and locale.
+Only one complete run can execute at a time; `cancel-in-progress: false`
+preserves it when another run is requested. `queue: max` lets up to 100
+pending runs wait instead of replacing the previous pending run; see
+[GitHub's concurrency
documentation](https://docs.github.com/en/actions/how-tos/write-workflows/choose-when-workflows-run/control-workflow-concurrency).
+Each queued run takes a fresh master checkout and recounts pending documents.
+Translation PR
+merges do not trigger this workflow. It does not participate in website
+builds or release deployment.
Ongoing generation stays off unless both repository variables
`I18N_AUTO_TRANSLATE_ENABLED=true` and `I18N_LIVE_VERIFIED=true` are set. The
@@ -336,7 +347,7 @@ snapshots in OSS as the Apache Doris code-review runner.
The repository's
selected auth snapshot if Codex refreshes it. `I18N_CODEX_MODEL` is optional
and defaults to `gpt-6-sol`. A direct OpenAI API key is not used by this
workflow; the separate T07 manual tool still supports the Responses API.
-To open a draft PR, configure one GitHub write identity. The preferred App path
+To open a PR, configure one GitHub write identity. The preferred App path
uses `I18N_GITHUB_APP_ID` and `I18N_GITHUB_APP_PRIVATE_KEY`; install the App on
`apache/doris-website` with repository Contents and Pull requests write
permissions. If an App cannot be installed, a maintainer with repository write
@@ -347,11 +358,11 @@ must store the secret. The resulting commits and PR use
that person's identity.
When both credentials are present, the App takes precedence. The workflow
does not use `GITHUB_TOKEN` to create PRs because those PR checks can require
manual approval. Set the enablement variables only after a one-page `validate`
-run and its App- or PAT-created draft PR have passed the required checks.
+run and its App- or PAT-created PR have passed the required checks.
Missing write credentials leave manual validation as an artifact-only run;
missing OSS credentials leave discovery available without model calls.
-When enabled, one run produces at most two page candidates. The T08 planner
+When enabled, one run produces at most 10 target page candidates. The T08
planner
and Codex adapter do not reject work based on an estimated or accumulated
token allowance. The adapter still records actual usage for the run summary.
Document length, chunk size, call timeout and the model's own context limits
@@ -363,11 +374,15 @@ only if its OSS source has not changed since download.
The generator calls the model once for exact Dev/4.x pairs whose doc ID,
relative source path and normalized source hash match. It writes two separate
state records. Human-owned or changed targets, moves, deletions and other
-protected tasks stay in the manual queue. The robot opens a draft PR with
-machine review status and never merges it. It keeps one active robot batch PR;
-if its source becomes stale and its head still matches the robot's recorded
-commit, the next run closes that stale draft and opens a fresh batch. A human
-commit on the branch stops replacement and is reported for manual handling.
+protected tasks stay in the manual queue. The robot opens a regular PR with
+machine review status and the `translate-ja` label, and never merges it.
+Maintainers review and merge PRs when ready. Multiple batches can remain open:
+their documents are excluded, while unrelated pending documents can form the
+next batch. This exclusion includes stale candidates and manually edited
+branches; unresolved target/state records block their targets before the
+planner applies its batch limit. Generation rechecks PR reservations for both
+primary and reused Dev/4.x targets. Each batch writes separate page state
+shards where enabled, so different documents do not rewrite a shared ledger.
The PR body lists the source commit, source links, source hashes, model,
glossary version, failures and validation. The normal PR checks remain the
merge gate.
@@ -387,7 +402,9 @@ reconciliation and full planning against a GitHub CLI mock,
then runs the
translation engine with a schema-shaped Codex mock. It checks that an invalid
model unit fails validation, that a 100,000-token usage report does not stop a
valid candidate, and that publication pushes only managed Japanese files to a
-local bare Git repository and records a mock draft PR. `report.json` and step
+local bare Git repository and records a mock regular PR with its label. Each
+local step has a 20-minute timeout to accommodate the full native scan.
+`report.json` and step
logs remain in the output directory. No real model call or GitHub write occurs.
## Validation
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
