I've waited to respond to this email until the next release was performed. Looking at the latest rounds, our real problem is the queue is set to terminate. https://github.com/apache/grails-core/pull/16231/ will fix the doc publish for any future staged release. Please see the associated documentation.
I am still against changing the github action to retry instead of using a mutex to lock the values. Retrying will only let those actions run for longer periods of time, and the right solution is to have the mutex exist outside of the action of work. I'm happy to propose solutions for this mutex, but if we've fixed the queue, I don't see the need to proceed. On Fri, Aug 7, 2026 at 9:02 AM James Fredley <[email protected]> wrote: > > Technical Architecture: How Content Publishes to grails.apache.org > > Core Model > ---------- > > apache/grails-website (specifically the asf-site-production branch) is > the single aggregation and publication point for the live site. > > Apache Infrastructure watches this branch via the standard ASF site > publishing mechanism controlled by .asf.yaml: > > publish: > whoami: asf-site-production > > Any fast-forward commit that lands on asf-site-production is mirrored by > Apache Infra to https://grails.apache.org/ (with the usual CDN > soft-purge behaviour). The repository itself contains no build logic for > the site; it is purely the destination that multiple independent > producers push into. > > Content Producers > ----------------- > > There are currently two primary independent publishers (with more expected): > > 1. Main website + guides aggregate – apache/grails-static-website > > - Gradle-based static site generator (pages, blog, assets, > templates, guides registry, etc.). > - Workflow: .github/workflows/publish.yml > - Triggers: push to master, every 2 hours (cron: '0 */2 * * *'), > or workflow_dispatch. > - Has its own repository-scoped concurrency group (publish-${{ > github.ref }}, cancel-in-progress: false) so only one of its own > publishes runs at a time. > - Runs ./gradlew clean publishMainSite. > - The Gradle task uses environment variables > (GITHUB_SLUG=apache/grails-website, GH_BRANCH=asf-site-production, > GH_TOKEN) to push the generated content (root of the site) into the > shared branch. > - This is responsible for the majority of the top-level site content. > > 2. Versioned documentation – apache/grails-core (via the grails-doc build) > > - Snapshot and release documentation jobs in grails-core (and > related workflows) build the docs, then invoke the shared action: > uses: apache/grails-github-actions/deploy-github-pages@asf > - Key environment variables: > - TARGET_REPOSITORY: apache/grails-website > - DOCUMENTATION_BRANCH: asf-site-production > - TARGET_FOLDER: docs > - SOURCE_FOLDER: grails-doc/build/docs > - GRADLE_PUBLISH_RELEASE: true|false > - The action checks out the target branch, overlays the > documentation under /docs/ (creating versioned folders such as latest, > specific release versions, and snapshot), commits a single deployment > commit, and pushes. > > Legacy documentation already lives under docs-legacy-* directories in > the same branch. Guides are in the process of being fully consolidated > into the static-website publisher (previously split across several > locations). Grails Forge is expected to become another publisher > targeting the same destination. > > Shared Publishing Action > ------------------------ > > apache/grails-github-actions (branch asf) provides the reusable > deploy-github-pages action used by the documentation publishers. It is > deliberately a thin Git-based overlay: > > - Checkout of the target branch into a working tree. > - Copy of the generated content into the configured folder. > - Single commit + push. > > Because multiple independent workflows (from different repositories) can > target the same branch tip concurrently, non-fast-forward rejections are > a normal occurrence. The current work (PR #110) makes the action robust > to this by: > > - Detecting a genuine non-fast-forward rejection. > - Fetching the new tip into the existing shallow checkout. > - Confirming the fetched tip is a descendant of the originally observed tip. > - Rebasing the unpublished local deployment commit. > - Retrying a normal (non-force) push (maximum five attempts). > - Failing closed on true content conflicts or non-descendant history. > > No force-push is ever performed. > > Why Cross-Repository Coordination Is Required > --------------------------------------------- > > GitHub Actions concurrency groups are scoped to a single repository. A > group defined in grails-core (grails-docs-publish) cannot coordinate > with jobs running in grails-static-website (or future publishers such as > Forge). Serialising everything inside one repository simply creates > artificial queues and measurable release delays (observed 11–18 minute > waits, plus manual re-runs and runner contention). > > The fundamental problem is concurrent independent commits to a shared > branch. The correct coordination mechanism is ordinary Git fast-forward > semantics (fetch + rebase + push), which is exactly what the improved > action implements. > > Resulting Architecture > ---------------------- > > grails-static-website ──publishMainSite──┐ > │ > grails-core (docs jobs) ──deploy-github-pages──┼──► asf-site-production > ──ASF Infra──► grails.apache.org > │ > (future: Grails Forge, etc.) ──────────────┘ > > The live site is the pure aggregation of whatever the various generators > last successfully pushed. There is no central “build the whole site” > job; each producer owns its own content generation and is responsible > for landing a clean, rebaseable commit on the shared branch. > > This is the architecture that the current pair of PRs > (grails-github-actions#110 + grails-core#16110) is designed to make > reliable under concurrent load without introducing additional artifact > storage, coordinating workflows, or force-pushes. > > On 8/6/2026 4:12 PM, James Fredley wrote: > > We have an ongoing issue with concurrent publishing to the grails- > > website repository. This has affected multiple releases over the past > > several months in addition to grails-website publishing. Any time two > > publishing actions are running at the same time. > > > > I previously opened https://github.com/apache/grails-github-actions/ > > pull/98 to make concurrent documentation pushes safe. It was closed in > > favor of a job-level concurrency group (`grails-docs-publish`) added in > > grails-core#15988 and then closed again, after re-opening, when it > > became clear that did not work. That approach serializes independent > > documentation jobs within Grails Core. Because GitHub Actions > > concurrency groups are scoped to a single repository, it cannot > > coordinate publishers from other repositories (grails-static-website, > > and eventually Grails Forge) that also target the same destination branch. > > > > In practice making this serial has introduced measurable delays during > > release sequences (for example, an 18-minute wait observed on the each > > release documentation job, plus manual synchronization, plus manual re- > > run on grail-website action which failed, plus waiting on github runner > > availability). This burned 3 hours this morning. > > > > The current pair of pull requests restores the ability to publish in > > parallel while remaining conservative: > > > > > > - https://github.com/apache/grails-github-actions/pull/110 – Safely > > retry concurrent documentation pushes > > When another publisher advances the destination branch first, the > > action recognizes a genuine non-fast-forward rejection, fetches the new > > tip into the existing shallow checkout, confirms the fetched tip > > descends from the originally observed tip, rebases the unpublished local > > deployment commit, and retries a normal push (maximum five attempts). > > There is no force-push. Conflicts, non-descendant history, and unrelated > > failures fail closed without modifying the remote. > > > > - https://github.com/apache/grails-core/pull/16110 – Allow concurrent > > documentation publishing > > Removes the repository-local `grails-docs-publish` queue so > > independent documentation jobs can proceed once the action-level retry > > is available. Git’s normal fast-forward rules then coordinate updates to > > the shared branch. > > > > This is a deliberately narrow solution. It covers the cross-repository > > case, avoids serializing unrelated release work, and fails safely. I am > > open to any alternative that fully addresses the same constraints > > (multiple independent publishers, no force-push, safe handling of true > > content conflicts, and no artificial serialization of independent jobs). > > Until a better complete solution is available and verified, these > > changes allow the project to move forward without further release delays. > > > > An alternate solution has also been proposed of uploading artifacts and > > then having a separate workflow process them: > > That approach would introduce additional moving parts (artifact storage, > > a new coordinating workflow, hand-off between producers and a consumer, > > extra latency, and new failure modes) for what is fundamentally a > > concurrent commit problem on a shared branch. Multiple independent > > publishers are simply generating documentation and pushing it to the > > same destination. When another push lands first, the required response > > is the standard Git sequence: fetch the new tip, rebase the unpublished > > local commit, and push again. That is exactly what the retry logic in > > apache/grails-github-actions#110 does, using ordinary fast-forward rules > > and failing closed on true conflicts. > > > > Let's debate this and get it fixed before the next release cycle, so we > > can get back to faster releases. > > > > James >
