Thank you for responding, If there's an option that Benedict has suggested (to allow folks who push mostly-final patches, and the folks who push rather-early patches and then update them continuously, to coexist and be able to quickly switch between configs) I'd be more in favour of this rather than just enabling a build/compile step for everyone.
On Wed, Oct 20, 2021 at 8:25 PM Andrés de la Peña <adelap...@apache.org> wrote: > Hi all, > > As Ekaterina has mentioned I’ll be OOO until Monday, and I won’t be able to > make any changes until then. > > Alex, I missed to answer your suggestion about building on every commit, > apologies for that. The discussion was open on multiple fronts and I > somehow missed that one. I can easily change it back to automatic builds on > Monday if we prefer to do so. > > The pre-commit workflows have a single button to start the build and all > the tests that were previously run automatically. If we had automatic > builds we would still have a button to start the tests. Thus, I think that > automatic builds in the pre-commit workflow would’t make a difference in > terms of usability, and we would be wasting resources in intermediate > commits. > > As for the workflows with separate approval steps, the automatic build > would save us a click at the cost of wasting resources in some cases. Since > the cost of building is not that high it might make more sense to have > automatic builds in these workflows. Alternatively, a detail that could > improve things a bit in the separate-tests workflows is making the approval > steps for running the tests depend on the approval for the build. That > wouldn’t save us any clicks but it would make impossible to miss the build > approval, since we would need to click it in order to enable the buttons > starting the tests. > > It would be really great if the build started when one approves a certain > test job, but as it has been mentioned that doesn’t seem possible with the > current CircleCI features. > > Having a config that entirely satisfies the needs of everyone doesn’t seem > possible, and I also think that we’ll eventually need better tooling to > generate the config file, although we still need to agree on a default > config. CASSANDRA-16989 has recently added flags to the config generation > script allowing to swap resources and specify the environment vars. We > should probably continue adding similar options to be able to manipulate > the approval steps, parallelism, etc. > > Regards, > > El El mié, 20 oct 2021 a las 18:03, bened...@apache.org < > bened...@apache.org> > escribió: > > > Perhaps instead of a one-size-fits-all policy we should look to improve > > scripting here anyway. We already have to overwrite the config file, it > > would seem easy enough to instead invoke a script that enables the > relevant > > workflows? > > > > If we choose to just stick with a single workflow, I don’t have a super > > strong preference for one approach or the other. > > > > From: Stefan Miklosovic <stefan.mikloso...@instaclustr.com> > > Date: Wednesday, 20 October 2021 at 16:58 > > To: dev@cassandra.apache.org <dev@cassandra.apache.org> > > Subject: Re: Save CircleCI resources with optional test jobs > > Hi Ekaterina, all, > > > > If you eventually decide to switch it to automatic build on push (I > > like how it is now though), I would appreciate it if there was some > > option in generation scripts (generate.sh) so I could just have some > > shortcut / alias which would disable this very easily. > > > > It would be very nice if there was also an option to specify > > parallelism for highres nodes. My workflow so far was to 1) copy > > highres to config.yml, open editor and replace "100" with "70" for > > parallelism value. I am not sure what plan you guys are on but we can > > at most spin sometime like 80 nodes or so at once. > > > > Thanks > > > > Stefan > > > > On Wed, 20 Oct 2021 at 16:14, Ekaterina Dimitrova <e.dimitr...@gmail.com > > > > wrote: > > > > > > Hi everyone, > > > > > > First of all, I know Andres is off these days so he might not be > > following > > > the thread today. I will try to clear the situation as the committer > who > > > approved the change. > > > > > > Alex, I would like to apologize to you! You are right, I personally > > didn’t > > > have your email (sometimes gmail plays games with the mailing list > > emails, > > > not sure why) and I assumed the lazy consensus. Andres sent email that > > this > > > was implemented a month after the discussion was open and no one > > responded > > > here during that time. In his latest email he pointed to the Slack > > > discussion that happened and where agreement was reached. > > > I see your email was sent two days after he said it is implemented. I > > guess > > > we had to be more explicit that this email announces lazy consensus. > > > > > > Anyway, I am really sorry your email was missed and please, believe me > > when > > > I say I would have considered it as the ticket was not committed yet at > > > that point! I am almost sure something similar happened to Andres > knowing > > > how extremely punctual and fair he is. > > > > > > The main goal was not to push on every commit as many times it would be > > > unnecessary waste of resources and spending credits. At the same time > new > > > workflows with one precommit button were added to ensure people can run > > all > > > tests we as a community require with one click before a commit. Links > > with > > > the different versions/suggestions of the workflow are published on the > > > ticket. > > > > > > Yes, now we need to click on the build. It was agreed that many times > > > people push even just to preserve intermediate work and continue later > > > without caring of build or anything. If for some reason it is important > > to > > > you or anyone else from the community to build on every commit, we can > > open > > > a ticket to change that. It will save a click or two in the case of the > > > in-jvm upgrade tests. I would like to point out that Andres was also > > > experimenting to ensure that the graph will be still as much readable > as > > > possible. > > > > > > Benedict, on your comment of feature request - we can do that. I am > also > > > sceptic as you if that will happen but this doesn’t mean we can’t give > > it a > > > try. Who knows, maybe others are also raising the topic and they might > > > consider it. > > > > > > Honestly, I personally support the current workflow and approval > required > > > but if it is not acceptable to others and skipping the build approval > > click > > > is what others would prefer, I will open a ticket to restore that part. > > > Please let me know and one more time, apologize for missing your email, > > > Alex. > > > > > > Best regards, > > > Ekaterina > > > > > > > > > > > > On Wed, 20 Oct 2021 at 9:01, bened...@apache.org <bened...@apache.org> > > > wrote: > > > > > > > I think this would be fine if there were a way for approval of later > > steps > > > > to trigger the build automatically. As it is we have to traverse the > > > > dependency graph manually, which is a bit weird. > > > > > > > > I can’t figure out a way to do this with the CircleCI API however. It > > > > might be a feature request, and perhaps we can at least trigger the > > build > > > > until we get that (which may never happen). > > > > > > > > From: Oleksandr Petrov <oleksandr.pet...@gmail.com> > > > > Date: Wednesday, 20 October 2021 at 13:35 > > > > To: dev <dev@cassandra.apache.org> > > > > Subject: Re: Save CircleCI resources with optional test jobs > > > > I thought this discussion was still ongoing, but it looks like > > > > CASSANDRA-16882 is now committed. > > > > > > > > Could you give some context on why at least compilation is not done > on > > > > every commit now? > > > > > > > > > > > > On Fri, Oct 8, 2021 at 4:12 PM Oleksandr Petrov < > > > > oleksandr.pet...@gmail.com> > > > > wrote: > > > > > > > > > I personally rarely push tickets/post a patch in an raw state, but > > since > > > > I > > > > > almost always have to approve jobs when it gets close to commit, I > > don't > > > > > mind also confirming utest runs. I'd say it'd be good to run at > very > > > > least > > > > > a compilation step on every commit. I think it's fine if > > > > dtests/utests/jvm > > > > > tests require approval. > > > > > > > > > > On Wed, Oct 6, 2021 at 4:22 PM Andrés de la Peña < > > adelap...@apache.org> > > > > > wrote: > > > > > > > > > >> Hello all, > > > > >> > > > > >> The changes in CircleCI proposed in the previous messages are > > > > implemented > > > > >> in CASSANDRA-16882. They try to include the feedback from both the > > > > >> reviewers and the Slack discussion at > > > > >> https://the-asf.slack.com/archives/CK23JSY2K/p1631627458109000. > > > > >> > > > > >> The patch modifies the CircleCI config to have two pairs of j8/j11 > > > > >> workflows: > > > > >> > > > > >> * The javaX_pre-commit_tests workflows are meant to be used when a > > patch > > > > >> is > > > > >> definitively ready for final review and/or commit. They have a > > single > > > > >> approval step that starts all the basic tests, including unit > tests, > > > > >> dtests, etc. Additionally, it has approval steps for those tests > > that > > > > are > > > > >> not generally required in every ticket, such as upgrade tests and > > the > > > > >> multiplexer. This pair of workflows is quite similar to what we > > > > currently > > > > >> have, and the main difference is that there is an approval step to > > start > > > > >> the build. > > > > >> > > > > >> * The javaX_separate_tests workflows are meant to be used in > > > > intermediate > > > > >> steps and special cases such as fixing flaky tests. All the jobs > in > > > > these > > > > >> workflows have an individual approval step, so one can run any > test > > job > > > > >> without wasting resources in the others. For example, it makes > > possible > > > > to > > > > >> repeatedly run a unit test without firing the entire JVM dtests. > > > > >> > > > > >> Here is an example of how the workflows would look like: > > > > >> > > > > >> > > > > > > > https://app.circleci.com/pipelines/github/adelapena/cassandra?branch=16882-option-7-trunk-v04 > > > > >> > > > > >> Hopefully this approach would help us to save resources in > > intermediate > > > > >> development steps and special cases, while keeping the current > > concept > > > > of > > > > >> mandatory tests. > > > > >> > > > > >> If no one disagrees with this approach I'll commit the changes in > a > > few > > > > >> days. > > > > >> > > > > >> On Fri, 27 Aug 2021 at 11:54, Andrés de la Peña < > > adelap...@apache.org> > > > > >> wrote: > > > > >> > > > > >> > Hi, > > > > >> > > > > > >> > CASSANDRA-16882 has patches for any of the mentioned > > configurations > > > > >> aimed > > > > >> > to save CircleCI resources. > > > > >> > > > > > >> > There are CircleCI runs showing how each approach would look > like, > > > > plus > > > > >> an > > > > >> > additional fourth option: > > > > >> > > > > > >> > 1. Make the entire workflow optional: > > > > >> > > > > > >> > > > > > > > https://app.circleci.com/pipelines/github/adelapena/cassandra/800/workflows/9cb8ca7b-ab57-431e-a22b-643d61c92c29 > > > > >> > 2. Make all the test jobs optional: > > > > >> > > > > > >> > > > > > > > https://app.circleci.com/pipelines/github/adelapena/cassandra/798/workflows/a859cfbc-fdf8-4468-beb9-b2ee17dc1ae3 > > > > >> > 3. Make all the mandatory test jobs depend on a single optional > > job: > > > > >> > > > > > >> > > > > > > > https://app.circleci.com/pipelines/github/adelapena/cassandra/802/workflows/0372f5d6-d1f0-4f0e-91a3-aa75a2712bae > > > > >> > 4. Combine 2 and 3 to have approval jobs for both individual > > tests and > > > > >> the > > > > >> > grouped mandatory tests: > > > > >> > > > > > >> > > > > > > > https://app.circleci.com/pipelines/github/adelapena/cassandra/803/workflows/08ae07d5-6a1e-4e5b-bc0c-32bdc9b9f190 > > > > >> > > > > > >> > I think that the fourth option gives us more flexibility, > because > > it > > > > >> > allows us to start any combination of tests we want while it > > keeps the > > > > >> > concept of required tests. > > > > >> > > > > > >> > On Thu, 12 Aug 2021 at 17:47, Andrés de la Peña < > > > > >> a.penya.gar...@gmail.com> > > > > >> > wrote: > > > > >> > > > > > >> >> Hello all, > > > > >> >> > > > > >> >> The current CircleCI configuration automatically runs the unit > > tests, > > > > >> JVM > > > > >> >> dtests and cqhshlib tests. This is done by default for every > > commit > > > > or, > > > > >> >> with some configuration, for every push. > > > > >> >> > > > > >> >> Along the lifecycle of a ticket it is quite frequent to have > > multiple > > > > >> >> commits and pushes, all running these test jobs. I'd say that > > > > >> frequently it > > > > >> >> is not necessary to run the tests for some of those > intermediate > > > > >> commits > > > > >> >> and pushes. For example, one can show proofs of concept, or > have > > > > >> multiple > > > > >> >> rounds of review before actually running the tests. Running the > > tests > > > > >> for > > > > >> >> every change can produce an unnecessary expense of CircleCI > > > > resources. > > > > >> >> > > > > >> >> I think we could make running those tests optional, as well as > > > > clearly > > > > >> >> specifying in the documentation what are the tests runs that > are > > > > >> mandatory > > > > >> >> before actually committing. We could do this in different ways: > > > > >> >> > > > > >> >> 1. Make the entire CircleCI workflow optional, so the build job > > > > >> requires > > > > >> >> manual approval. Once the build is approved the mandatory test > > jobs > > > > >> would > > > > >> >> be run without any further approval, exactly as it's currently > > done. > > > > >> >> > > > > >> >> 2. Make all the test jobs optional, so every test job requires > > manual > > > > >> >> approval, and the documentation specifies which tests are > > mandatory > > > > in > > > > >> the > > > > >> >> final steps of a ticket. > > > > >> >> > > > > >> >> 3. Make all the mandatory test jobs depend on a single optional > > job, > > > > so > > > > >> >> we have a single button to optionally run all the mandatory > > tests. > > > > >> >> > > > > >> >> I think any of these changes, or a combination of them, would > > > > >> >> significantly reduce the usage of resources without making > things > > > > less > > > > >> >> tested. The only downside I can think of is that we would need > > some > > > > >> >> additional clicks on the CircleCI GUI. What do you think? > > > > >> >> > > > > >> > > > > > >> > > > > > > > > > > > > > > > -- > > > > > alex p > > > > > > > > > > > > > > > > > -- > > > > alex p > > > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > > For additional commands, e-mail: dev-h...@cassandra.apache.org > > > -- alex p
