With the performance improvements I have seen after the previous merges, and specifically the Gradle optimizations that were part of these, I believe this is the way to go.
Thanks James, for leading the way! Den tors 17 apr. 2025 kl 18:23 skrev David Estes <davydot...@gmail.com>: > I too am in huge favor of this. I believe I have voiced my opinions over > the past year that this should be the direction we should move. You will > get no argument from me. Huge thanks for James work on making this > performant and clean because it’s not enough just to make a mono repo. He > has gone above and beyond here to make this a pleasant experience. > > Thank you James! > > David > > > On Apr 17, 2025, at 8:48 AM, James Fredley <jamesfred...@apache.org> > wrote: > > > > James, > > > > Thank you for the extremely detailed outline of the benefits and reasons > for finishing the move to a mono repo for the Core Grails Projects, by > consolidating the remaining two. > > > > It is my belief that 1/3 or more of the time invested over the last 8-9 > months was spent on task that are eliminated by a mono repo. Maybe 50%. > A mono repo makes releases simple vs taking days which was down from weeks > per release. > > > > I am in favor of merging Grails-geb & Grails-data-mapping. > > > > The details for build time and mongo are well detailed here and I see a > clear plan for how they will be addressed with changes before and after > these two projects are merged into grails-core. > > > > I strongly believe that a mono repo is key to quickly iterating on > Grails 7.0.x, 7.1.x and 8.0.x, this year. > > > > James Fredley > > > > On 2025/04/17 04:05:31 James Daugherty wrote: > >> Hi Everyone, > >> > >> We've previously discussed [1] merging the grails-data-mapping repo into > >> grails-core. Other than grails-geb, this is the last remaining > repository > >> to merge into core to have a mono repo. This email attempts to summarize > >> some of the previously expressed concerns and advocates for merging > >> grails-data-mapping & grails-geb sooner rather than later. Let's gather > >> people's thoughts so we can determine if a vote thread is feasible. > >> > >> Some of the recent concerns that have been raised on merging data > mapping > >> are: > >> * slower build times (both locally & in GitHub actions) > >> * the requirement for mongodb > >> > >> To answer those concerns: > >> ------------------------------------------------ > >> On slower build times: > >> ------------------------------------------------ > >> The current grails-core should not be viewed as slow. Across all of the > >> recent mergers (cache, views, gradle plugins, docs), I've spent a lot of > >> time optimizing the build. These optimizations include: > >> A. Converting a substantial amount of our build files to lazy > >> initialization instead of eager initialization & updating the grails > gradle > >> plugins to make use of lazy where possible. This means we only spend a > >> total of 8.8 seconds in configuration in a project being built from > >> scratch. > >> B. Updating parts of the build & parts of the grails gradle plugins to > be > >> cacheable by defining inputs/outputs. This means there's a higher > chance > >> that if a project dependency doesn't change, it won't rebuild now. > >> C. Decoupling the gradle plugin dependencies so that there are not > circular > >> references & that its dependencies can be managed separately from > >> application dependencies (we produce a grails-bom for applications & > >> grails-gradle-bom for gradle usage now). > >> D. Eliminating unnecessary steps or processes in the build (i.e. stream > >> lining the docs workflow) > >> E. Parallelizing the build where possible (there are known issues that > >> prevent us from being fully parallel, but we're very close to almost all > >> projects being able to be run in parallel). > >> F. Fixing our dependency graphs so that we generate proper platform POMs > >> and proper gradle modules so dependencies can be calculated correctly > (and > >> quickly). > >> G. I have added properties to both configure tests that should run on an > >> opt-in basis and an opt-out basis. This allows selectively running > tests > >> by setting a system property on your build. This allows further focused > >> development when needed. > >> > >> The build for the grails-core library is now approximately 3 minutes if > >> building from scratch on the most recent Mac hardware (assuming the > >> libraries are already present locally). The build also peeks at 1.5 > gig of > >> memory usage. It is also highly cacheable - only 30% of the tasks have > >> remaining cache issues (namely ones related to gsp, gson, and asset > >> compilation). I believe long term we can get the typical build time > down > >> even lower by improving these processes to be cacheable by gradle, by > >> further decoupling our build, and by further parallelization. After > >> merging grails-data-mapping, it should be possible to keep these build > >> times down. > >> > >> Concerning the build times in github, the main slowness is caused by > how we > >> matrix test now with windows, mac, linux across different versions of > the > >> JVM. We also get throttled more when we have to do this across every > >> repository. Having one repository will mean there's less of a chance of > >> being throttled. Moreover, we can pursue self hosted build agents to > solve > >> this in the long term. In the short term, if it really becomes an > issue, > >> we can enable gradle caching which will result in very little code > having > >> to run to the aforementioned improvements. > >> > >> ------------------------------------------------ > >> On the requirement for mongodb > >> ------------------------------------------------ > >> The requirement for a running mongodb causes several issues: > >> A. It forces the build to be synchronous for testing mongo related > projects > >> (plugin, mongo core, mongo ext, mongo bson, mongo templates, tck for > mongo, > >> etc) > >> B. It requires the user to have a running mongo instance. > >> > >> We know that B. is fixable by running a mongodb container. More over, > if > >> it could be run for each project, then that also solves the synchronous > >> execution. We know that running a docker container for B is trivial and > >> grails development already requires a container runtime to run it's > >> functional tests with geb. The command for this is: `docker run -d > --name > >> mongo-on-docker -p 27017:27017 mongo` > >> > >> A. will then be fixed if we can spin up a container over the lifecycle > of a > >> given project's tests. For example, we could use test containers prior > to > >> the GrailsApp.run() call in Application.groovy to ensure one exists per > >> application. There will need to be some configuration rework, but it > >> shouldn't be too hard to accomplish longer term. > >> > >> > >> > >> ------------------------------------------------ > >> As for why we should merge these libraries: > >> ------------------------------------------------ > >> 1. While working on the merges of the previous builds, I have discovered > >> numerous validations that gradle performs (circular dependencies, etc) > that > >> were not being performed when these builds are separate. Combined, we > get > >> the benefit of gradle warning us about circular dependencies and > benefiting > >> from this feedback. > >> > >> 2. Somewhat related, we can institute code standards, code styles, and > code > >> quality scans in a centralized manner. > >> > >> 3. Seperate from Gradle's validations, we can implement our own gradle > >> plugins local to the grails-core repo that will enforce architecture > >> separation - this includes ensuring that gsp & gorm can be used > separately > >> from Grails in the long run. > >> > >> 4. If a gradle project that is a dependency of grails-core is partially > >> published, it will break functional tests in all repositories. This > means > >> you have to comment the tests out across repositories until all > artifacts > >> are published again. Inside of the same project, this issue does not > exist. > >> > >> 3. The iteration time on development is vastly improved in a single > >> repository. The optimizations I made to the gradle plugin took seconds > to > >> test and I would not have been able to make them in separate projects in > >> less than a day. The feedback loop is a significant time saver. What > >> would take 20-30minutes due to build publishing before I was doing in > >> seconds. The gradle plugin changes would likely have taken over a week > (or > >> longer) if these plugins were still in a separate repository. > >> > >> 4. The known issues with grails-data-mapping are not major blockers. > While > >> they may initially slow the build times, we can address the majority of > the > >> time by solving the mongo problem. We have an initial approach that > works, > >> we'll just need to adjust the configuration in the mongo projects to > >> connect to different containers. > >> > >> 5. Having a mono repo ensures that any change to Grails will be tested > >> fully. Several of us have spent a significant amount of time chasing > down > >> bugs, that we later have discovered are due to someone only running the > >> tests in the project. If someone changes code related to the core of > >> grails, they must run all of the associated tests. In a mono project, > this > >> happens locally during development. Outside of it, it happens by users > >> discovering the bug in a milestone. > >> > >> 6. Spending time on the build process - the github action release > workflow, > >> etc - is a significant problem. We don't want to be working on a build > >> process. We want to be developing code and fixing bugs for grails. > >> > >> 7. Spring Boot & Hibernate will have major upgrades on a more regular > basis > >> going forward. To make the changes necessary, we don't want to be > working > >> on separate processes. We need to adopt a more rapid release schedule > and > >> react to library upgrades faster so that we don't end up in the > situation > >> we have been in for Grails 7. The reason Grails 7 development has > taken so > >> incredibly long, is we're updating some libraries that are over 4 years > >> old. The technical debt can be prevented by updating more often and > >> staying up to date with upstream libraries - which also ensures the > >> security of the framework. > >> > >> 8. Apache's release process requires a security review. This security > >> review ensures that builds are not being tampered with and our current > >> release process across many repos requires build tampering. We > eventually > >> stop modifying a build, but to be able to release a milestone sooner > with > >> the apache coordinates, we need to be in one repository. > >> > >> 9. We no longer have admin rights to the GitHub organization we are > under. > >> One of the discoveries we made after moving, was that we can't trigger > >> workflow actions from one repo to another. Being in one repository, > means > >> we don't have to do that and infrastructure does not need to find work > >> arounds for our existing processes. > >> > >> 10. Contributing to grails will be made easier for newer contributors. > >> They wont' have to learn to build projects in certain orders or how to > work > >> around issues when something fails. > >> > >> > >> I'm sure I've forgotten several of the reasons, but I'd like to propose > we > >> go to a mono repo for the core libraries that make up a grails release. > >> Data mapping & geb are the only ones that remain to have this. I'd > like to > >> fast track this and deal with the mongo / slowness after merge. I > believe > >> we can resolve these issues and by merging sooner we can get to > releasing > >> the first milestone under Apache. > >> > >> -James > >> > >> [1] https://lists.apache.org/thread/sfzzzbb1zo6k4w8hz0ro13wx4n4jyhr6 > >> > >
