I have unfortunately not had the opportunity to dive deep into the core yet, so my opinion on this is week.
The arguments that James D bringes forward makes me want to support his proposal. You have my +1 Med venlig hilsen / Best regards Søren Berg Glasius Sent from my phone, thus brief On Thu, 17 Apr 2025, 20.20 Mattias Reichel, <mattias.reic...@gmail.com> wrote: > With the performance improvements I have seen after the previous merges, > and specifically the Gradle optimizations that were part of these, I > believe this is the way to go. > > Thanks James, for leading the way! > > Den tors 17 apr. 2025 kl 18:23 skrev David Estes <davydot...@gmail.com>: > > > I too am in huge favor of this. I believe I have voiced my opinions over > > the past year that this should be the direction we should move. You will > > get no argument from me. Huge thanks for James work on making this > > performant and clean because it’s not enough just to make a mono repo. He > > has gone above and beyond here to make this a pleasant experience. > > > > Thank you James! > > > > David > > > > > On Apr 17, 2025, at 8:48 AM, James Fredley <jamesfred...@apache.org> > > wrote: > > > > > > James, > > > > > > Thank you for the extremely detailed outline of the benefits and > reasons > > for finishing the move to a mono repo for the Core Grails Projects, by > > consolidating the remaining two. > > > > > > It is my belief that 1/3 or more of the time invested over the last 8-9 > > months was spent on task that are eliminated by a mono repo. Maybe 50%. > > A mono repo makes releases simple vs taking days which was down from > weeks > > per release. > > > > > > I am in favor of merging Grails-geb & Grails-data-mapping. > > > > > > The details for build time and mongo are well detailed here and I see a > > clear plan for how they will be addressed with changes before and after > > these two projects are merged into grails-core. > > > > > > I strongly believe that a mono repo is key to quickly iterating on > > Grails 7.0.x, 7.1.x and 8.0.x, this year. > > > > > > James Fredley > > > > > > On 2025/04/17 04:05:31 James Daugherty wrote: > > >> Hi Everyone, > > >> > > >> We've previously discussed [1] merging the grails-data-mapping repo > into > > >> grails-core. Other than grails-geb, this is the last remaining > > repository > > >> to merge into core to have a mono repo. This email attempts to > summarize > > >> some of the previously expressed concerns and advocates for merging > > >> grails-data-mapping & grails-geb sooner rather than later. Let's > gather > > >> people's thoughts so we can determine if a vote thread is feasible. > > >> > > >> Some of the recent concerns that have been raised on merging data > > mapping > > >> are: > > >> * slower build times (both locally & in GitHub actions) > > >> * the requirement for mongodb > > >> > > >> To answer those concerns: > > >> ------------------------------------------------ > > >> On slower build times: > > >> ------------------------------------------------ > > >> The current grails-core should not be viewed as slow. Across all of > the > > >> recent mergers (cache, views, gradle plugins, docs), I've spent a lot > of > > >> time optimizing the build. These optimizations include: > > >> A. Converting a substantial amount of our build files to lazy > > >> initialization instead of eager initialization & updating the grails > > gradle > > >> plugins to make use of lazy where possible. This means we only spend > a > > >> total of 8.8 seconds in configuration in a project being built from > > >> scratch. > > >> B. Updating parts of the build & parts of the grails gradle plugins to > > be > > >> cacheable by defining inputs/outputs. This means there's a higher > > chance > > >> that if a project dependency doesn't change, it won't rebuild now. > > >> C. Decoupling the gradle plugin dependencies so that there are not > > circular > > >> references & that its dependencies can be managed separately from > > >> application dependencies (we produce a grails-bom for applications & > > >> grails-gradle-bom for gradle usage now). > > >> D. Eliminating unnecessary steps or processes in the build (i.e. > stream > > >> lining the docs workflow) > > >> E. Parallelizing the build where possible (there are known issues that > > >> prevent us from being fully parallel, but we're very close to almost > all > > >> projects being able to be run in parallel). > > >> F. Fixing our dependency graphs so that we generate proper platform > POMs > > >> and proper gradle modules so dependencies can be calculated correctly > > (and > > >> quickly). > > >> G. I have added properties to both configure tests that should run on > an > > >> opt-in basis and an opt-out basis. This allows selectively running > > tests > > >> by setting a system property on your build. This allows further > focused > > >> development when needed. > > >> > > >> The build for the grails-core library is now approximately 3 minutes > if > > >> building from scratch on the most recent Mac hardware (assuming the > > >> libraries are already present locally). The build also peeks at 1.5 > > gig of > > >> memory usage. It is also highly cacheable - only 30% of the tasks > have > > >> remaining cache issues (namely ones related to gsp, gson, and asset > > >> compilation). I believe long term we can get the typical build time > > down > > >> even lower by improving these processes to be cacheable by gradle, by > > >> further decoupling our build, and by further parallelization. After > > >> merging grails-data-mapping, it should be possible to keep these build > > >> times down. > > >> > > >> Concerning the build times in github, the main slowness is caused by > > how we > > >> matrix test now with windows, mac, linux across different versions of > > the > > >> JVM. We also get throttled more when we have to do this across every > > >> repository. Having one repository will mean there's less of a chance > of > > >> being throttled. Moreover, we can pursue self hosted build agents to > > solve > > >> this in the long term. In the short term, if it really becomes an > > issue, > > >> we can enable gradle caching which will result in very little code > > having > > >> to run to the aforementioned improvements. > > >> > > >> ------------------------------------------------ > > >> On the requirement for mongodb > > >> ------------------------------------------------ > > >> The requirement for a running mongodb causes several issues: > > >> A. It forces the build to be synchronous for testing mongo related > > projects > > >> (plugin, mongo core, mongo ext, mongo bson, mongo templates, tck for > > mongo, > > >> etc) > > >> B. It requires the user to have a running mongo instance. > > >> > > >> We know that B. is fixable by running a mongodb container. More over, > > if > > >> it could be run for each project, then that also solves the > synchronous > > >> execution. We know that running a docker container for B is trivial > and > > >> grails development already requires a container runtime to run it's > > >> functional tests with geb. The command for this is: `docker run -d > > --name > > >> mongo-on-docker -p 27017:27017 mongo` > > >> > > >> A. will then be fixed if we can spin up a container over the lifecycle > > of a > > >> given project's tests. For example, we could use test containers > prior > > to > > >> the GrailsApp.run() call in Application.groovy to ensure one exists > per > > >> application. There will need to be some configuration rework, but it > > >> shouldn't be too hard to accomplish longer term. > > >> > > >> > > >> > > >> ------------------------------------------------ > > >> As for why we should merge these libraries: > > >> ------------------------------------------------ > > >> 1. While working on the merges of the previous builds, I have > discovered > > >> numerous validations that gradle performs (circular dependencies, etc) > > that > > >> were not being performed when these builds are separate. Combined, we > > get > > >> the benefit of gradle warning us about circular dependencies and > > benefiting > > >> from this feedback. > > >> > > >> 2. Somewhat related, we can institute code standards, code styles, and > > code > > >> quality scans in a centralized manner. > > >> > > >> 3. Seperate from Gradle's validations, we can implement our own gradle > > >> plugins local to the grails-core repo that will enforce architecture > > >> separation - this includes ensuring that gsp & gorm can be used > > separately > > >> from Grails in the long run. > > >> > > >> 4. If a gradle project that is a dependency of grails-core is > partially > > >> published, it will break functional tests in all repositories. This > > means > > >> you have to comment the tests out across repositories until all > > artifacts > > >> are published again. Inside of the same project, this issue does not > > exist. > > >> > > >> 3. The iteration time on development is vastly improved in a single > > >> repository. The optimizations I made to the gradle plugin took > seconds > > to > > >> test and I would not have been able to make them in separate projects > in > > >> less than a day. The feedback loop is a significant time saver. What > > >> would take 20-30minutes due to build publishing before I was doing in > > >> seconds. The gradle plugin changes would likely have taken over a > week > > (or > > >> longer) if these plugins were still in a separate repository. > > >> > > >> 4. The known issues with grails-data-mapping are not major blockers. > > While > > >> they may initially slow the build times, we can address the majority > of > > the > > >> time by solving the mongo problem. We have an initial approach that > > works, > > >> we'll just need to adjust the configuration in the mongo projects to > > >> connect to different containers. > > >> > > >> 5. Having a mono repo ensures that any change to Grails will be tested > > >> fully. Several of us have spent a significant amount of time chasing > > down > > >> bugs, that we later have discovered are due to someone only running > the > > >> tests in the project. If someone changes code related to the core of > > >> grails, they must run all of the associated tests. In a mono project, > > this > > >> happens locally during development. Outside of it, it happens by users > > >> discovering the bug in a milestone. > > >> > > >> 6. Spending time on the build process - the github action release > > workflow, > > >> etc - is a significant problem. We don't want to be working on a > build > > >> process. We want to be developing code and fixing bugs for grails. > > >> > > >> 7. Spring Boot & Hibernate will have major upgrades on a more regular > > basis > > >> going forward. To make the changes necessary, we don't want to be > > working > > >> on separate processes. We need to adopt a more rapid release schedule > > and > > >> react to library upgrades faster so that we don't end up in the > > situation > > >> we have been in for Grails 7. The reason Grails 7 development has > > taken so > > >> incredibly long, is we're updating some libraries that are over 4 > years > > >> old. The technical debt can be prevented by updating more often and > > >> staying up to date with upstream libraries - which also ensures the > > >> security of the framework. > > >> > > >> 8. Apache's release process requires a security review. This security > > >> review ensures that builds are not being tampered with and our current > > >> release process across many repos requires build tampering. We > > eventually > > >> stop modifying a build, but to be able to release a milestone sooner > > with > > >> the apache coordinates, we need to be in one repository. > > >> > > >> 9. We no longer have admin rights to the GitHub organization we are > > under. > > >> One of the discoveries we made after moving, was that we can't trigger > > >> workflow actions from one repo to another. Being in one repository, > > means > > >> we don't have to do that and infrastructure does not need to find work > > >> arounds for our existing processes. > > >> > > >> 10. Contributing to grails will be made easier for newer contributors. > > >> They wont' have to learn to build projects in certain orders or how to > > work > > >> around issues when something fails. > > >> > > >> > > >> I'm sure I've forgotten several of the reasons, but I'd like to > propose > > we > > >> go to a mono repo for the core libraries that make up a grails > release. > > >> Data mapping & geb are the only ones that remain to have this. I'd > > like to > > >> fast track this and deal with the mongo / slowness after merge. I > > believe > > >> we can resolve these issues and by merging sooner we can get to > > releasing > > >> the first milestone under Apache. > > >> > > >> -James > > >> > > >> [1] https://lists.apache.org/thread/sfzzzbb1zo6k4w8hz0ro13wx4n4jyhr6 > > >> > > > > >
