I'm writing up a more complete proposal, but here are some examples.
Parameters can be set from either the UI (not my intent) or via the
circleci CLI. Effectively, the config-2_1.yml can have parameters specified
like:
parameters:
run_extra_test:
type: boolean
default: false
jobs:
extra_test:
steps:
when:
condition: << pipeline.parameters.run_extra_test >>
steps:
- run : # something
- run : # something
And then jobs can conditionally execute or not. The "when" clause can also
be applied at the workflows level. When you generate the configuration, you
pass a 2nd yaml file (or text) to the circleci CLI and it will override any
parameters with what you provide. For example, if I wanted to disable the
test:
circleci config process --pipeline-parameters "run_extra_test: false"
config-2_1.yml
The tradeoff is that the config will be more verbose with conditionals. My
first proposal is actually to just use parameters to get rid of the patch
files, since the current patch files simply change values. In this case,
parameters allow us to use the config yaml like a template, which is less
error prone than patch files. However, things like conditional execution
and matrix jobs might allow us to streamline the config, especially as we
support more JDK versions.
Cheers,
Derek
On Tue, Oct 25, 2022 at 10:40 AM David Capwell <dcapw...@apple.com> wrote:
> This could also be a pipeline parameter instead of hacking it in
> generate.sh
>
>
> Curious how this works… I run a script that deletes all the approvals and
> removes the testing workflows… I really don’t want to use the UI at all….
> I assumed pipeline params are a UI thing, but I think the goal here for
> many of us are to ignore the UI other than looking at the results… and even
> that can be scripted...
>
>
> On Oct 24, 2022, at 4:44 PM, Derek Chen-Becker <de...@chen-becker.org>
> wrote:
>
> This could also be a pipeline parameter instead of hacking it in
> generate.sh. I promise I'll have a proposal before the end of the week.
>
> Derek
>
> On Mon, Oct 24, 2022 at 2:13 PM Josh McKenzie <jmcken...@apache.org>
> wrote:
>
>> @Ekaterina: I recall us going back and forth on whether default should be
>> require approval or not and there not being a consensus. I'm fine not
>> changing the status quo and just parameterizing that in generate.sh so
>> folks can locally script how they want to setup when they alias up
>> generate.sh.
>>
>> I'll add C-17113 to the epic as well and any other tickets anyone has in
>> flight we can link up.
>>
>> Maybe we should remove them from the workflow when the free option is used
>>
>> That'd put us in the position of having a "smoke testing suite" for free
>> tier users and the expectation of a committer running the full suite
>> pre-merge. Which, now that I type it out, is a lot more representative of
>> our current reality so we should probably do that.
>>
>> Noted re: the -f flag; I could have checked that but just hacked that out
>> in the email spur of the moment. We could just default to low / free /
>> smoke test and have -p for paid tier.
>>
>>
>> On Mon, Oct 24, 2022, at 3:23 PM, Andrés de la Peña wrote:
>>
>> - Ticket for: remove -h, have -f and -p (free and paid)
>>
>>
>> +1 to this, probably there isn't anyone using -h. There are some jobs
>> that can't pass with the free option. Maybe we should remove them from the
>> workflow when the free option is used. Perhaps that could save new
>> contributors some confusion. Or should we leave them because a subset of
>> the tests inside those jobs can still pass even with the free tier?
>>
>> By the way, the generate.sh script already accepts a -f flag. It's used
>> to stop checking that the specified environment variables are known. It was
>> meant to be a kind of general "--force" flag.
>>
>> On Mon, 24 Oct 2022 at 20:07, Ekaterina Dimitrova <e.dimitr...@gmail.com>
>> wrote:
>>
>> Seems like my email crashed with Andres’ one.
>> My understanding is we will use the ticket CASSANDRA-17113 as
>> placeholder, the work there will be rebased/reworked etc depending on what
>> we agree with.
>> I also agree with the other points he made. Sounds reasonable to me
>>
>> On Mon, 24 Oct 2022 at 15:03, Ekaterina Dimitrova <e.dimitr...@gmail.com>
>> wrote:
>>
>> Thank you Josh
>>
>> So about push with/without a single click, I guess you mean to
>> parameterize whether the step build needs approval or not? Pre-commit the
>> new flag will use the “no-approval” version, but during development we
>> still will be able to push the tests without immediately starting all
>> tests, right?
>> - parallelism + -h being removed - just to confirm, that means we will
>> not use xlarge containers. As David confirmed, this is not needed for all
>> jibs and it is important as otherwise whoever uses paid account will burn
>> their credits time faster for very similar duration runs.
>>
>> CASSANDRA-17930 - I will use the opportunity also to mention that many of
>> the identified missing jobs in CircleCI will be soon there - Andres is
>> working on all variations unit tests, I am doing final testing on fixing
>> the Python upgrade tests (we weren’t using the right parameters and running
>> way more jobs then we should) and Derek is looking into the rest of the
>> Python test. I still need to check whether we need something regarding
>> in-jvm etc, the simulator ones are running only for jdk8 for now,
>> confirmed. All this should unblock us to be able to do next releases based
>> on CircleCI as we agreed. Then we move to do some
>> changes/additions/improvements to Jenkins. And of course, the future
>> improvements we agreed on.
>>
>> On Mon, 24 Oct 2022 at 14:10, Josh McKenzie <jmcken...@apache.org> wrote:
>>
>>
>> Auto-run on push? Can you elaborate?
>>
>> Yep - instead of having to go to circle and click, when you push your
>> branch the circle hook picks it up and kicks off the top level job
>> automatically. I tend to be paranoid and push a lot of incremental work
>> that's not ready for CI remotely so it's not great for me, but I think
>> having it be optional is the Right Thing.
>>
>> So here's the outstanding work I've distilled from this thread:
>> - Create an epic for circleci improvement work (we have a lot of little
>> augments to do here; keep it organized and try and avoid redundancy)
>> - Include CASSANDRA-17600 in epic umbrella
>> - Include CASSANDRA-17930 in epic umbrella
>> - Ticket to tune parallelism per job
>> -
>> > def java_parallelism(src_dir, kind, num_file_in_worker, include =
>> lambda a, b: True):
>> > d = os.path.join(src_dir, 'test', kind)
>> > num_files = 0
>> > for root, dirs, files in os.walk(d):
>> > for f in files:
>> > if f.endswith('Test.java') and
>> include(os.path.join(root, f), f):
>> > num_files += 1
>> > return math.floor(num_files / num_file_in_worker)
>> >
>> > def fix_parallelism(args, contents):
>> > jobs = contents['jobs']
>> >
>> > unit_parallelism = java_parallelism(args.src,
>> 'unit', 20)
>> > jvm_dtest_parallelism = java_parallelism(args.src,
>> 'distributed', 4, lambda full, name: 'upgrade' not in full)
>> > jvm_dtest_upgrade_parallelism = java_parallelism(args.src,
>> 'distributed', 2, lambda full, name: 'upgrade' in full)
>> - `TL;DR - I find all test files we are going to run, and based off a
>> pre-defined variable that says “idea” number of files per worker, I then
>> calculate how many workers we need. So unit tests are num_files / 20 ~= 35
>> workers. Can I be “smarter” by knowing which files have higher cost?
>> Sure… but the “perfect” and the “average” are too similar that it wasn’t
>> worth it...`
>> - Ticket to combine pre-commit jobs into 1 pipeline for all JDK's
>> - Path to activate all supported JDK's for pre-commit at root
>> (one-click pre-merge full validation)
>> - Path to activate per JDK below that (interim work partial
>> validation)
>> - Ticket to rename jobs in circleci
>> - Reference comment:
>> https://issues.apache.org/jira/browse/CASSANDRA-17939?focusedCommentId=17617016&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17617016
>> - (buildjdk)_(runjdk)_(testsuite) format:
>> - j8_j8_jvm_dtests
>> - j8_j11_jvm_dtests
>> - j11_j11_jvm_dtest_vnode
>> etc
>> - Ticket for flag in generate.sh to support auto run on push (see
>> response above)
>> - Ticket for: remove -h, have -f and -p (free and paid) (probably
>> intersects with https://issues.apache.org/jira/browse/CASSANDRA-17600)
>>
>> Anything wrong w/the above or anything missed? If not, I'll go do some
>> JIRA'ing.
>>
>>
>> ~Josh
>>
>>
>> On Fri, Oct 21, 2022, at 3:50 PM, Josh McKenzie wrote:
>>
>> I am cool with removing circle if apache CI is stable and works, we do
>> need to solve the non-committer issue but would argue that partially exists
>> in circle today (you can be a non-commuter with a paid account, but you
>> can’t be a non-committer with a free account)
>>
>> There's a few threads here:
>> 1. non-committers should be able to run ci
>> 2. People that have resources and want to run ci faster should be able to
>> do so (assuming the ci of record could serve to be faster)
>> 3. ci should be stable
>>
>> Thus far we haven't landed on 1 system that satisfies all 3. There's some
>> background discussions brainstorming how to get there; when / if things
>> come from that they'll as always be brought to the list for discussion.
>>
>> On Fri, Oct 21, 2022, at 1:44 PM, Ekaterina Dimitrova wrote:
>>
>> I agree with David with one caveat - last time I checked only some Python
>> tests lack enough resources with the free tier. The rest run slower than
>> with a paid account, but they do fine. In fact I use the free tier if I
>> want to test only unit or in-jvm tests sometimes. I guess that is what he
>> meant by partially but even being able to run the non-Python tests is a win
>> IMHO. If we find a solution for all tests though… even better.
>> @Derek your idea sounds interesting, I will be happy to see a proposal.
>> Thank you
>>
>> On Fri, 21 Oct 2022 at 13:39, David Capwell <dcapw...@apple.com> wrote:
>>
>> I am cool with removing circle if apache CI is stable and works, we do
>> need to solve the non-committer issue but would argue that partially exists
>> in circle today (you can be a non-commuter with a paid account, but you
>> can’t be a non-committer with a free account)
>>
>>
>>
>> On Oct 20, 2022, at 2:20 PM, Josh McKenzie <jmcken...@apache.org> wrote:
>>
>> I believe it's original intention to be just about CircleCI.
>>
>> It was but fwiw I'm good w/us exploring adjacent things regarding CI
>> here. I'm planning on deep diving on the thread tomorrow and distilling a
>> snapshot of the work we have a consensus on for circle and summarizing here
>> so we don't lose that. Seems like it's fairly non-controversial.
>>
>> On Thu, Oct 20, 2022, at 5:14 PM, Mick Semb Wever wrote:
>>
>>
>>
>> On Thu, 20 Oct 2022 at 22:07, Derek Chen-Becker <de...@chen-becker.org>
>> wrote:
>>
>> Would the preclusion of non-committers also prevent us from configuring
>> Jenkins to auto-test on PR independent of who opens it?
>>
>> One of my current concerns is that we're maintaining 2x the CI for 1x the
>> benefit, and I don't currently see an easy way to unify them (perhaps a
>> lack of imagination?). I know there's a long history behind the choice of
>> CircleCI, so I'm not trying to be hand-wavy about all of the thought that
>> went into that decision, but that decision has costs beyond just a paid
>> CircleCI account. My long term, probably naive, goals for CI would be to:
>>
>> 1. Have a CI system that is *fully* available to *any* contributor,
>> modulo safeguards to prevent abuse
>>
>>
>>
>> This thread is going off-topic, as I believe it's original intention to
>> be just about CircleCI.
>>
>> But on your point… our community CI won't be allowed (by ASF), nor have
>> capacity (limited donated resources), to run pre-commit testing by anyone
>> and everyone.
>>
>> Today, trusted contributors can be handed tokens to ci-cassandra.a.o
>> (make sure to label them so they can be revoked easily), but we still face
>> the issue that too many pre-commit runs impacts the throughput and quality
>> of the post-commit runs (though this has improved recently).
>>
>> It's on my wishlist to be able to: with a single command line; spin up
>> the ci-cassandra.a.o stack on any k8s cluster, run any git sha through it
>> and collect results, and tear it down. Variations on this would solve
>> non-committers being able to repeat, use, and work on their own (or a
>> separately donated) CI system, and folk/companies with money to be able to
>> run their own ci-cassandra.a.o stacks for faster pre-commit turnaround
>> time. Having this reproducibility of the CI system would make testing
>> changes to it easier as well, so I'd expect a positive feedback loop here.
>>
>> I have some rough ideas on how to get started on this, if anyone would
>> like to buddy up on it.
>>
>>
>>
>>
>>
>
> --
> +---------------------------------------------------------------+
> | Derek Chen-Becker |
> | GPG Key available at https://keybase.io/dchenbecker and |
> | https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
> | Fngrprnt: EB8A 6480 F0A3 C8EB C1E7 7F42 AFC5 AFEE 96E4 6ACC |
> +---------------------------------------------------------------+
>
>
>
--
+---------------------------------------------------------------+
| Derek Chen-Becker |
| GPG Key available at https://keybase.io/dchenbecker and |
| https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
| Fngrprnt: EB8A 6480 F0A3 C8EB C1E7 7F42 AFC5 AFEE 96E4 6ACC |
+---------------------------------------------------------------+
