I just want to thank you for the great work 👍 Gianluca Sartori
On Thu, 17 Apr 2025 at 14:48, James Fredley <jamesfred...@apache.org> wrote: > James, > > Thank you for the extremely detailed outline of the benefits and reasons > for finishing the move to a mono repo for the Core Grails Projects, by > consolidating the remaining two. > > It is my belief that 1/3 or more of the time invested over the last 8-9 > months was spent on task that are eliminated by a mono repo. Maybe 50%. > A mono repo makes releases simple vs taking days which was down from weeks > per release. > > I am in favor of merging Grails-geb & Grails-data-mapping. > > The details for build time and mongo are well detailed here and I see a > clear plan for how they will be addressed with changes before and after > these two projects are merged into grails-core. > > I strongly believe that a mono repo is key to quickly iterating on Grails > 7.0.x, 7.1.x and 8.0.x, this year. > > James Fredley > > On 2025/04/17 04:05:31 James Daugherty wrote: > > Hi Everyone, > > > > We've previously discussed [1] merging the grails-data-mapping repo into > > grails-core. Other than grails-geb, this is the last remaining > repository > > to merge into core to have a mono repo. This email attempts to summarize > > some of the previously expressed concerns and advocates for merging > > grails-data-mapping & grails-geb sooner rather than later. Let's gather > > people's thoughts so we can determine if a vote thread is feasible. > > > > Some of the recent concerns that have been raised on merging data mapping > > are: > > * slower build times (both locally & in GitHub actions) > > * the requirement for mongodb > > > > To answer those concerns: > > ------------------------------------------------ > > On slower build times: > > ------------------------------------------------ > > The current grails-core should not be viewed as slow. Across all of the > > recent mergers (cache, views, gradle plugins, docs), I've spent a lot of > > time optimizing the build. These optimizations include: > > A. Converting a substantial amount of our build files to lazy > > initialization instead of eager initialization & updating the grails > gradle > > plugins to make use of lazy where possible. This means we only spend a > > total of 8.8 seconds in configuration in a project being built from > > scratch. > > B. Updating parts of the build & parts of the grails gradle plugins to be > > cacheable by defining inputs/outputs. This means there's a higher chance > > that if a project dependency doesn't change, it won't rebuild now. > > C. Decoupling the gradle plugin dependencies so that there are not > circular > > references & that its dependencies can be managed separately from > > application dependencies (we produce a grails-bom for applications & > > grails-gradle-bom for gradle usage now). > > D. Eliminating unnecessary steps or processes in the build (i.e. stream > > lining the docs workflow) > > E. Parallelizing the build where possible (there are known issues that > > prevent us from being fully parallel, but we're very close to almost all > > projects being able to be run in parallel). > > F. Fixing our dependency graphs so that we generate proper platform POMs > > and proper gradle modules so dependencies can be calculated correctly > (and > > quickly). > > G. I have added properties to both configure tests that should run on an > > opt-in basis and an opt-out basis. This allows selectively running tests > > by setting a system property on your build. This allows further focused > > development when needed. > > > > The build for the grails-core library is now approximately 3 minutes if > > building from scratch on the most recent Mac hardware (assuming the > > libraries are already present locally). The build also peeks at 1.5 gig > of > > memory usage. It is also highly cacheable - only 30% of the tasks have > > remaining cache issues (namely ones related to gsp, gson, and asset > > compilation). I believe long term we can get the typical build time down > > even lower by improving these processes to be cacheable by gradle, by > > further decoupling our build, and by further parallelization. After > > merging grails-data-mapping, it should be possible to keep these build > > times down. > > > > Concerning the build times in github, the main slowness is caused by how > we > > matrix test now with windows, mac, linux across different versions of the > > JVM. We also get throttled more when we have to do this across every > > repository. Having one repository will mean there's less of a chance of > > being throttled. Moreover, we can pursue self hosted build agents to > solve > > this in the long term. In the short term, if it really becomes an issue, > > we can enable gradle caching which will result in very little code having > > to run to the aforementioned improvements. > > > > ------------------------------------------------ > > On the requirement for mongodb > > ------------------------------------------------ > > The requirement for a running mongodb causes several issues: > > A. It forces the build to be synchronous for testing mongo related > projects > > (plugin, mongo core, mongo ext, mongo bson, mongo templates, tck for > mongo, > > etc) > > B. It requires the user to have a running mongo instance. > > > > We know that B. is fixable by running a mongodb container. More over, if > > it could be run for each project, then that also solves the synchronous > > execution. We know that running a docker container for B is trivial and > > grails development already requires a container runtime to run it's > > functional tests with geb. The command for this is: `docker run -d > --name > > mongo-on-docker -p 27017:27017 mongo` > > > > A. will then be fixed if we can spin up a container over the lifecycle > of a > > given project's tests. For example, we could use test containers prior > to > > the GrailsApp.run() call in Application.groovy to ensure one exists per > > application. There will need to be some configuration rework, but it > > shouldn't be too hard to accomplish longer term. > > > > > > > > ------------------------------------------------ > > As for why we should merge these libraries: > > ------------------------------------------------ > > 1. While working on the merges of the previous builds, I have discovered > > numerous validations that gradle performs (circular dependencies, etc) > that > > were not being performed when these builds are separate. Combined, we > get > > the benefit of gradle warning us about circular dependencies and > benefiting > > from this feedback. > > > > 2. Somewhat related, we can institute code standards, code styles, and > code > > quality scans in a centralized manner. > > > > 3. Seperate from Gradle's validations, we can implement our own gradle > > plugins local to the grails-core repo that will enforce architecture > > separation - this includes ensuring that gsp & gorm can be used > separately > > from Grails in the long run. > > > > 4. If a gradle project that is a dependency of grails-core is partially > > published, it will break functional tests in all repositories. This > means > > you have to comment the tests out across repositories until all artifacts > > are published again. Inside of the same project, this issue does not > exist. > > > > 3. The iteration time on development is vastly improved in a single > > repository. The optimizations I made to the gradle plugin took seconds > to > > test and I would not have been able to make them in separate projects in > > less than a day. The feedback loop is a significant time saver. What > > would take 20-30minutes due to build publishing before I was doing in > > seconds. The gradle plugin changes would likely have taken over a week > (or > > longer) if these plugins were still in a separate repository. > > > > 4. The known issues with grails-data-mapping are not major blockers. > While > > they may initially slow the build times, we can address the majority of > the > > time by solving the mongo problem. We have an initial approach that > works, > > we'll just need to adjust the configuration in the mongo projects to > > connect to different containers. > > > > 5. Having a mono repo ensures that any change to Grails will be tested > > fully. Several of us have spent a significant amount of time chasing > down > > bugs, that we later have discovered are due to someone only running the > > tests in the project. If someone changes code related to the core of > > grails, they must run all of the associated tests. In a mono project, > this > > happens locally during development. Outside of it, it happens by users > > discovering the bug in a milestone. > > > > 6. Spending time on the build process - the github action release > workflow, > > etc - is a significant problem. We don't want to be working on a build > > process. We want to be developing code and fixing bugs for grails. > > > > 7. Spring Boot & Hibernate will have major upgrades on a more regular > basis > > going forward. To make the changes necessary, we don't want to be > working > > on separate processes. We need to adopt a more rapid release schedule > and > > react to library upgrades faster so that we don't end up in the situation > > we have been in for Grails 7. The reason Grails 7 development has taken > so > > incredibly long, is we're updating some libraries that are over 4 years > > old. The technical debt can be prevented by updating more often and > > staying up to date with upstream libraries - which also ensures the > > security of the framework. > > > > 8. Apache's release process requires a security review. This security > > review ensures that builds are not being tampered with and our current > > release process across many repos requires build tampering. We > eventually > > stop modifying a build, but to be able to release a milestone sooner with > > the apache coordinates, we need to be in one repository. > > > > 9. We no longer have admin rights to the GitHub organization we are > under. > > One of the discoveries we made after moving, was that we can't trigger > > workflow actions from one repo to another. Being in one repository, > means > > we don't have to do that and infrastructure does not need to find work > > arounds for our existing processes. > > > > 10. Contributing to grails will be made easier for newer contributors. > > They wont' have to learn to build projects in certain orders or how to > work > > around issues when something fails. > > > > > > I'm sure I've forgotten several of the reasons, but I'd like to propose > we > > go to a mono repo for the core libraries that make up a grails release. > > Data mapping & geb are the only ones that remain to have this. I'd like > to > > fast track this and deal with the mongo / slowness after merge. I > believe > > we can resolve these issues and by merging sooner we can get to releasing > > the first milestone under Apache. > > > > -James > > > > [1] https://lists.apache.org/thread/sfzzzbb1zo6k4w8hz0ro13wx4n4jyhr6 > > >
