@Ekaterina: I recall us going back and forth on whether default should be
require approval or not and there not being a consensus. I'm fine not changing
the status quo and just parameterizing that in generate.sh so folks can locally
script how they want to setup when they alias up generate.sh.
I'll add C-17113 to the epic as well and any other tickets anyone has in flight
we can link up.
> Maybe we should remove them from the workflow when the free option is used
That'd put us in the position of having a "smoke testing suite" for free tier
users and the expectation of a committer running the full suite pre-merge.
Which, now that I type it out, is a lot more representative of our current
reality so we should probably do that.
Noted re: the -f flag; I could have checked that but just hacked that out in
the email spur of the moment. We could just default to low / free / smoke test
and have -p for paid tier.
On Mon, Oct 24, 2022, at 3:23 PM, Andrés de la Peña wrote:
>> - Ticket for: remove -h, have -f and -p (free and paid)
>
> +1 to this, probably there isn't anyone using -h. There are some jobs that
> can't pass with the free option. Maybe we should remove them from the
> workflow when the free option is used. Perhaps that could save new
> contributors some confusion. Or should we leave them because a subset of the
> tests inside those jobs can still pass even with the free tier?
>
> By the way, the generate.sh script already accepts a -f flag. It's used to
> stop checking that the specified environment variables are known. It was
> meant to be a kind of general "--force" flag.
>
> On Mon, 24 Oct 2022 at 20:07, Ekaterina Dimitrova <e.dimitr...@gmail.com>
> wrote:
>> Seems like my email crashed with Andres’ one.
>> My understanding is we will use the ticket CASSANDRA-17113 as placeholder,
>> the work there will be rebased/reworked etc depending on what we agree with.
>> I also agree with the other points he made. Sounds reasonable to me
>>
>> On Mon, 24 Oct 2022 at 15:03, Ekaterina Dimitrova <e.dimitr...@gmail.com>
>> wrote:
>>> Thank you Josh
>>>
>>> So about push with/without a single click, I guess you mean to parameterize
>>> whether the step build needs approval or not? Pre-commit the new flag will
>>> use the “no-approval” version, but during development we still will be able
>>> to push the tests without immediately starting all tests, right?
>>> - parallelism + -h being removed - just to confirm, that means we will not
>>> use xlarge containers. As David confirmed, this is not needed for all jibs
>>> and it is important as otherwise whoever uses paid account will burn their
>>> credits time faster for very similar duration runs.
>>>
>>> CASSANDRA-17930 - I will use the opportunity also to mention that many of
>>> the identified missing jobs in CircleCI will be soon there - Andres is
>>> working on all variations unit tests, I am doing final testing on fixing
>>> the Python upgrade tests (we weren’t using the right parameters and running
>>> way more jobs then we should) and Derek is looking into the rest of the
>>> Python test. I still need to check whether we need something regarding
>>> in-jvm etc, the simulator ones are running only for jdk8 for now,
>>> confirmed. All this should unblock us to be able to do next releases based
>>> on CircleCI as we agreed. Then we move to do some
>>> changes/additions/improvements to Jenkins. And of course, the future
>>> improvements we agreed on.
>>>
>>> On Mon, 24 Oct 2022 at 14:10, Josh McKenzie <jmcken...@apache.org> wrote:
>>>> __
>>>>> Auto-run on push? Can you elaborate?
>>>> Yep - instead of having to go to circle and click, when you push your
>>>> branch the circle hook picks it up and kicks off the top level job
>>>> automatically. I tend to be paranoid and push a lot of incremental work
>>>> that's not ready for CI remotely so it's not great for me, but I think
>>>> having it be optional is the Right Thing.
>>>>
>>>> So here's the outstanding work I've distilled from this thread:
>>>> - Create an epic for circleci improvement work (we have a lot of little
>>>> augments to do here; keep it organized and try and avoid redundancy)
>>>> - Include CASSANDRA-17600 in epic umbrella
>>>> - Include CASSANDRA-17930 in epic umbrella
>>>> - Ticket to tune parallelism per job
>>>> -
>>>> > def java_parallelism(src_dir, kind, num_file_in_worker, include =
>>>> lambda a, b: True):
>>>> > d = os.path.join(src_dir, 'test', kind)
>>>> > num_files = 0
>>>> > for root, dirs, files in os.walk(d):
>>>> > for f in files:
>>>> > if f.endswith('Test.java') and
>>>> include(os.path.join(root, f), f):
>>>> > num_files += 1
>>>> > return math.floor(num_files / num_file_in_worker)
>>>> >
>>>> > def fix_parallelism(args, contents):
>>>> > jobs = contents['jobs']
>>>> >
>>>> > unit_parallelism = java_parallelism(args.src,
>>>> 'unit', 20)
>>>> > jvm_dtest_parallelism = java_parallelism(args.src,
>>>> 'distributed', 4, lambda full, name: 'upgrade' not in full)
>>>> > jvm_dtest_upgrade_parallelism = java_parallelism(args.src,
>>>> 'distributed', 2, lambda full, name: 'upgrade' in full)
>>>> - `TL;DR - I find all test files we are going to run, and based off a
>>>> pre-defined variable that says “idea” number of files per worker, I then
>>>> calculate how many workers we need. So unit tests are num_files / 20 ~=
>>>> 35 workers. Can I be “smarter” by knowing which files have higher cost?
>>>> Sure… but the “perfect” and the “average” are too similar that it wasn’t
>>>> worth it...`
>>>> - Ticket to combine pre-commit jobs into 1 pipeline for all JDK's
>>>> - Path to activate all supported JDK's for pre-commit at root
>>>> (one-click pre-merge full validation)
>>>> - Path to activate per JDK below that (interim work partial validation)
>>>> - Ticket to rename jobs in circleci
>>>> - Reference comment:
>>>> https://issues.apache.org/jira/browse/CASSANDRA-17939?focusedCommentId=17617016&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17617016
>>>> - (buildjdk)_(runjdk)_(testsuite) format:
>>>> - j8_j8_jvm_dtests
>>>> - j8_j11_jvm_dtests
>>>> - j11_j11_jvm_dtest_vnode
>>>> etc
>>>> - Ticket for flag in generate.sh to support auto run on push (see response
>>>> above)
>>>> - Ticket for: remove -h, have -f and -p (free and paid) (probably
>>>> intersects with https://issues.apache.org/jira/browse/CASSANDRA-17600)
>>>>
>>>> Anything wrong w/the above or anything missed? If not, I'll go do some
>>>> JIRA'ing.
>>>>
>>>>
>>>> ~Josh
>>>>
>>>>
>>>> On Fri, Oct 21, 2022, at 3:50 PM, Josh McKenzie wrote:
>>>>>> I am cool with removing circle if apache CI is stable and works, we do
>>>>>> need to solve the non-committer issue but would argue that partially
>>>>>> exists in circle today (you can be a non-commuter with a paid account,
>>>>>> but you can’t be a non-committer with a free account)
>>>>> There's a few threads here:
>>>>> 1. non-committers should be able to run ci
>>>>> 2. People that have resources and want to run ci faster should be able to
>>>>> do so (assuming the ci of record could serve to be faster)
>>>>> 3. ci should be stable
>>>>>
>>>>> Thus far we haven't landed on 1 system that satisfies all 3. There's some
>>>>> background discussions brainstorming how to get there; when / if things
>>>>> come from that they'll as always be brought to the list for discussion.
>>>>>
>>>>> On Fri, Oct 21, 2022, at 1:44 PM, Ekaterina Dimitrova wrote:
>>>>>> I agree with David with one caveat - last time I checked only some
>>>>>> Python tests lack enough resources with the free tier. The rest run
>>>>>> slower than with a paid account, but they do fine. In fact I use the
>>>>>> free tier if I want to test only unit or in-jvm tests sometimes. I guess
>>>>>> that is what he meant by partially but even being able to run the
>>>>>> non-Python tests is a win IMHO. If we find a solution for all tests
>>>>>> though… even better.
>>>>>> @Derek your idea sounds interesting, I will be happy to see a proposal.
>>>>>> Thank you
>>>>>>
>>>>>> On Fri, 21 Oct 2022 at 13:39, David Capwell <dcapw...@apple.com> wrote:
>>>>>>> I am cool with removing circle if apache CI is stable and works, we do
>>>>>>> need to solve the non-committer issue but would argue that partially
>>>>>>> exists in circle today (you can be a non-commuter with a paid account,
>>>>>>> but you can’t be a non-committer with a free account)
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>> On Oct 20, 2022, at 2:20 PM, Josh McKenzie <jmcken...@apache.org>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> I believe it's original intention to be just about CircleCI.
>>>>>>>> It was but fwiw I'm good w/us exploring adjacent things regarding CI
>>>>>>>> here. I'm planning on deep diving on the thread tomorrow and
>>>>>>>> distilling a snapshot of the work we have a consensus on for circle
>>>>>>>> and summarizing here so we don't lose that. Seems like it's fairly
>>>>>>>> non-controversial.
>>>>>>>>
>>>>>>>> On Thu, Oct 20, 2022, at 5:14 PM, Mick Semb Wever wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, 20 Oct 2022 at 22:07, Derek Chen-Becker
>>>>>>>>> <de...@chen-becker.org> wrote:
>>>>>>>>>> Would the preclusion of non-committers also prevent us from
>>>>>>>>>> configuring Jenkins to auto-test on PR independent of who opens it?
>>>>>>>>>>
>>>>>>>>>> One of my current concerns is that we're maintaining 2x the CI for
>>>>>>>>>> 1x the benefit, and I don't currently see an easy way to unify them
>>>>>>>>>> (perhaps a lack of imagination?). I know there's a long history
>>>>>>>>>> behind the choice of CircleCI, so I'm not trying to be hand-wavy
>>>>>>>>>> about all of the thought that went into that decision, but that
>>>>>>>>>> decision has costs beyond just a paid CircleCI account. My long
>>>>>>>>>> term, probably naive, goals for CI would be to:
>>>>>>>>>>
>>>>>>>>>> 1. Have a CI system that is *fully* available to *any* contributor,
>>>>>>>>>> modulo safeguards to prevent abuse
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> This thread is going off-topic, as I believe it's original intention
>>>>>>>>> to be just about CircleCI.
>>>>>>>>>
>>>>>>>>> But on your point… our community CI won't be allowed (by ASF), nor
>>>>>>>>> have capacity (limited donated resources), to run pre-commit testing
>>>>>>>>> by anyone and everyone.
>>>>>>>>>
>>>>>>>>> Today, trusted contributors can be handed tokens to ci-cassandra.a.o
>>>>>>>>> (make sure to label them so they can be revoked easily), but we still
>>>>>>>>> face the issue that too many pre-commit runs impacts the throughput
>>>>>>>>> and quality of the post-commit runs (though this has improved
>>>>>>>>> recently).
>>>>>>>>>
>>>>>>>>> It's on my wishlist to be able to: with a single command line; spin
>>>>>>>>> up the ci-cassandra.a.o stack on any k8s cluster, run any git sha
>>>>>>>>> through it and collect results, and tear it down. Variations on this
>>>>>>>>> would solve non-committers being able to repeat, use, and work on
>>>>>>>>> their own (or a separately donated) CI system, and folk/companies
>>>>>>>>> with money to be able to run their own ci-cassandra.a.o stacks for
>>>>>>>>> faster pre-commit turnaround time. Having this reproducibility of the
>>>>>>>>> CI system would make testing changes to it easier as well, so I'd
>>>>>>>>> expect a positive feedback loop here.
>>>>>>>>>
>>>>>>>>> I have some rough ideas on how to get started on this, if anyone
>>>>>>>>> would like to buddy up on it.
>>>>>
>>>>
