I understand the architecture, but I object to the retry methodology as it masks issues when there are failures. We have had this issue so many times - we add retry, which masks a problem, then it gets worse, and thus it just gets longer and longer.
I offered an alternative solution above - which is to move the mutex outside of that action into it's own action. On 2026/09/02 19:57:40 James Fredley wrote: > https://github.com/apache/grails-github-actions/pull/110 - PR solves it using > native git functionality for grails-core, grails-static-website and > grails-forge-ui committing to grails-website concurrently. If there are > conflicts it fails fast, you rerun the job and it works like today. Saves > 30-60 minutes for the 95% scenario. > > The entire architecture is described in the emails above, if the details need > reviewed. > > On 2026/09/01 14:46:46 James Daugherty wrote: > > My objection is only to having the publish process lock inside of the > > publishing. I have no problem adding a lock / unlock action that we can > > use in these workflows. We could even have the lock be a git committed > > object. Retrying leads to longer build times, which leads to problems not > > being found and is a reoccurring theme, which is why I'm so against it. > > > > -James > > > > On 2026/08/27 06:12:48 Mattias Reichel wrote: > > > Good find, that will certainly help with the most occurring problem. > > > However, I think the concurrency goups are scoped per repository, so > > > pushing from multiple repositories at the same time will still be > > > problematic? > > > > > > Den tors 27 aug. 2026 02:00James Daugherty via dev <[email protected]> > > > skrev: > > > > > > > I've waited to respond to this email until the next release was > > > > performed. Looking at the latest rounds, our real problem is the > > > > queue is set to terminate. > > > > https://github.com/apache/grails-core/pull/16231/ will fix the doc > > > > publish for any future staged release. Please see the associated > > > > documentation. > > > > > > > > I am still against changing the github action to retry instead of > > > > using a mutex to lock the values. Retrying will only let those > > > > actions run for longer periods of time, and the right solution is to > > > > have the mutex exist outside of the action of work. I'm happy to > > > > propose solutions for this mutex, but if we've fixed the queue, I > > > > don't see the need to proceed. > > > > > > > > On Fri, Aug 7, 2026 at 9:02 AM James Fredley <[email protected]> > > > > wrote: > > > > > > > > > > Technical Architecture: How Content Publishes to grails.apache.org > > > > > > > > > > Core Model > > > > > ---------- > > > > > > > > > > apache/grails-website (specifically the asf-site-production branch) is > > > > > the single aggregation and publication point for the live site. > > > > > > > > > > Apache Infrastructure watches this branch via the standard ASF site > > > > > publishing mechanism controlled by .asf.yaml: > > > > > > > > > > publish: > > > > > whoami: asf-site-production > > > > > > > > > > Any fast-forward commit that lands on asf-site-production is mirrored > > > > > by > > > > > Apache Infra to https://grails.apache.org/ (with the usual CDN > > > > > soft-purge behaviour). The repository itself contains no build logic > > > > > for > > > > > the site; it is purely the destination that multiple independent > > > > > producers push into. > > > > > > > > > > Content Producers > > > > > ----------------- > > > > > > > > > > There are currently two primary independent publishers (with more > > > > expected): > > > > > > > > > > 1. Main website + guides aggregate – apache/grails-static-website > > > > > > > > > > - Gradle-based static site generator (pages, blog, assets, > > > > > templates, guides registry, etc.). > > > > > - Workflow: .github/workflows/publish.yml > > > > > - Triggers: push to master, every 2 hours (cron: '0 */2 * * *'), > > > > > or workflow_dispatch. > > > > > - Has its own repository-scoped concurrency group (publish-${{ > > > > > github.ref }}, cancel-in-progress: false) so only one of its own > > > > > publishes runs at a time. > > > > > - Runs ./gradlew clean publishMainSite. > > > > > - The Gradle task uses environment variables > > > > > (GITHUB_SLUG=apache/grails-website, GH_BRANCH=asf-site-production, > > > > > GH_TOKEN) to push the generated content (root of the site) into the > > > > > shared branch. > > > > > - This is responsible for the majority of the top-level site > > > > > content. > > > > > > > > > > 2. Versioned documentation – apache/grails-core (via the grails-doc > > > > build) > > > > > > > > > > - Snapshot and release documentation jobs in grails-core (and > > > > > related workflows) build the docs, then invoke the shared action: > > > > > uses: apache/grails-github-actions/deploy-github-pages@asf > > > > > - Key environment variables: > > > > > - TARGET_REPOSITORY: apache/grails-website > > > > > - DOCUMENTATION_BRANCH: asf-site-production > > > > > - TARGET_FOLDER: docs > > > > > - SOURCE_FOLDER: grails-doc/build/docs > > > > > - GRADLE_PUBLISH_RELEASE: true|false > > > > > - The action checks out the target branch, overlays the > > > > > documentation under /docs/ (creating versioned folders such as latest, > > > > > specific release versions, and snapshot), commits a single deployment > > > > > commit, and pushes. > > > > > > > > > > Legacy documentation already lives under docs-legacy-* directories in > > > > > the same branch. Guides are in the process of being fully consolidated > > > > > into the static-website publisher (previously split across several > > > > > locations). Grails Forge is expected to become another publisher > > > > > targeting the same destination. > > > > > > > > > > Shared Publishing Action > > > > > ------------------------ > > > > > > > > > > apache/grails-github-actions (branch asf) provides the reusable > > > > > deploy-github-pages action used by the documentation publishers. It is > > > > > deliberately a thin Git-based overlay: > > > > > > > > > > - Checkout of the target branch into a working tree. > > > > > - Copy of the generated content into the configured folder. > > > > > - Single commit + push. > > > > > > > > > > Because multiple independent workflows (from different repositories) > > > > > can > > > > > target the same branch tip concurrently, non-fast-forward rejections > > > > > are > > > > > a normal occurrence. The current work (PR #110) makes the action > > > > > robust > > > > > to this by: > > > > > > > > > > - Detecting a genuine non-fast-forward rejection. > > > > > - Fetching the new tip into the existing shallow checkout. > > > > > - Confirming the fetched tip is a descendant of the originally > > > > > observed > > > > tip. > > > > > - Rebasing the unpublished local deployment commit. > > > > > - Retrying a normal (non-force) push (maximum five attempts). > > > > > - Failing closed on true content conflicts or non-descendant history. > > > > > > > > > > No force-push is ever performed. > > > > > > > > > > Why Cross-Repository Coordination Is Required > > > > > --------------------------------------------- > > > > > > > > > > GitHub Actions concurrency groups are scoped to a single repository. A > > > > > group defined in grails-core (grails-docs-publish) cannot coordinate > > > > > with jobs running in grails-static-website (or future publishers such > > > > > as > > > > > Forge). Serialising everything inside one repository simply creates > > > > > artificial queues and measurable release delays (observed 11–18 minute > > > > > waits, plus manual re-runs and runner contention). > > > > > > > > > > The fundamental problem is concurrent independent commits to a shared > > > > > branch. The correct coordination mechanism is ordinary Git > > > > > fast-forward > > > > > semantics (fetch + rebase + push), which is exactly what the improved > > > > > action implements. > > > > > > > > > > Resulting Architecture > > > > > ---------------------- > > > > > > > > > > grails-static-website ──publishMainSite──┐ > > > > > │ > > > > > grails-core (docs jobs) ──deploy-github-pages──┼──► > > > > > asf-site-production > > > > > ──ASF Infra──► grails.apache.org > > > > > │ > > > > > (future: Grails Forge, etc.) ──────────────┘ > > > > > > > > > > The live site is the pure aggregation of whatever the various > > > > > generators > > > > > last successfully pushed. There is no central “build the whole site” > > > > > job; each producer owns its own content generation and is responsible > > > > > for landing a clean, rebaseable commit on the shared branch. > > > > > > > > > > This is the architecture that the current pair of PRs > > > > > (grails-github-actions#110 + grails-core#16110) is designed to make > > > > > reliable under concurrent load without introducing additional artifact > > > > > storage, coordinating workflows, or force-pushes. > > > > > > > > > > On 8/6/2026 4:12 PM, James Fredley wrote: > > > > > > We have an ongoing issue with concurrent publishing to the grails- > > > > > > website repository. This has affected multiple releases over the > > > > > > past > > > > > > several months in addition to grails-website publishing. Any time > > > > > > two > > > > > > publishing actions are running at the same time. > > > > > > > > > > > > I previously opened https://github.com/apache/grails-github-actions/ > > > > > > pull/98 to make concurrent documentation pushes safe. It was closed > > > > > > in > > > > > > favor of a job-level concurrency group (`grails-docs-publish`) > > > > > > added in > > > > > > grails-core#15988 and then closed again, after re-opening, when it > > > > > > became clear that did not work. That approach serializes independent > > > > > > documentation jobs within Grails Core. Because GitHub Actions > > > > > > concurrency groups are scoped to a single repository, it cannot > > > > > > coordinate publishers from other repositories > > > > > > (grails-static-website, > > > > > > and eventually Grails Forge) that also target the same destination > > > > branch. > > > > > > > > > > > > In practice making this serial has introduced measurable delays > > > > > > during > > > > > > release sequences (for example, an 18-minute wait observed on the > > > > > > each > > > > > > release documentation job, plus manual synchronization, plus manual > > > > > > re- > > > > > > run on grail-website action which failed, plus waiting on github > > > > > > runner > > > > > > availability). This burned 3 hours this morning. > > > > > > > > > > > > The current pair of pull requests restores the ability to publish in > > > > > > parallel while remaining conservative: > > > > > > > > > > > > > > > > > > - https://github.com/apache/grails-github-actions/pull/110 – Safely > > > > > > retry concurrent documentation pushes > > > > > > When another publisher advances the destination branch first, the > > > > > > action recognizes a genuine non-fast-forward rejection, fetches the > > > > > > new > > > > > > tip into the existing shallow checkout, confirms the fetched tip > > > > > > descends from the originally observed tip, rebases the unpublished > > > > local > > > > > > deployment commit, and retries a normal push (maximum five > > > > > > attempts). > > > > > > There is no force-push. Conflicts, non-descendant history, and > > > > unrelated > > > > > > failures fail closed without modifying the remote. > > > > > > > > > > > > - https://github.com/apache/grails-core/pull/16110 – Allow > > > > > > concurrent > > > > > > documentation publishing > > > > > > Removes the repository-local `grails-docs-publish` queue so > > > > > > independent documentation jobs can proceed once the action-level > > > > > > retry > > > > > > is available. Git’s normal fast-forward rules then coordinate > > > > > > updates > > > > to > > > > > > the shared branch. > > > > > > > > > > > > This is a deliberately narrow solution. It covers the > > > > > > cross-repository > > > > > > case, avoids serializing unrelated release work, and fails safely. > > > > > > I am > > > > > > open to any alternative that fully addresses the same constraints > > > > > > (multiple independent publishers, no force-push, safe handling of > > > > > > true > > > > > > content conflicts, and no artificial serialization of independent > > > > jobs). > > > > > > Until a better complete solution is available and verified, these > > > > > > changes allow the project to move forward without further release > > > > delays. > > > > > > > > > > > > An alternate solution has also been proposed of uploading artifacts > > > > > > and > > > > > > then having a separate workflow process them: > > > > > > That approach would introduce additional moving parts (artifact > > > > storage, > > > > > > a new coordinating workflow, hand-off between producers and a > > > > > > consumer, > > > > > > extra latency, and new failure modes) for what is fundamentally a > > > > > > concurrent commit problem on a shared branch. Multiple independent > > > > > > publishers are simply generating documentation and pushing it to the > > > > > > same destination. When another push lands first, the required > > > > > > response > > > > > > is the standard Git sequence: fetch the new tip, rebase the > > > > > > unpublished > > > > > > local commit, and push again. That is exactly what the retry logic > > > > > > in > > > > > > apache/grails-github-actions#110 does, using ordinary fast-forward > > > > rules > > > > > > and failing closed on true conflicts. > > > > > > > > > > > > Let's debate this and get it fixed before the next release cycle, > > > > > > so we > > > > > > can get back to faster releases. > > > > > > > > > > > > James > > > > > > > > > > > > > > >
