Re: [ACTION REQUIRED] Removal of v3 artifact actions on December 5th

2024-11-26 Thread Jarek Potiuk
Yep. V4 is the right one to use. On Wed, Nov 27, 2024 at 2:30 AM Maxime Beauchemin < maximebeauche...@gmail.com> wrote: > v4 ok? > > https://github.com/apache/superset/blob/master/.github/workflows/superset-e2e.yml#L139 > > On Mon, 25 Nov 2024 at 16:15, Jarek Potiuk wrote:

Re: [ACTION REQUIRED] Removal of v3 artifact actions on December 5th

2024-11-25 Thread Jarek Potiuk
the warnings. Fixes are coming. J. On Tue, Nov 26, 2024 at 1:01 AM Jacob Wujciak wrote: > Sure: > > https://github.com/apache/airflow-client-python/blob/main/.github/workflows/ci.yml#L31 > > Am Di., 26. Nov. 2024 um 00:57 Uhr schrieb Jarek Potiuk >: > > > > I ca

Re: [ACTION REQUIRED] Removal of v3 artifact actions on December 5th

2024-11-25 Thread Jarek Potiuk
I can't find any artifacts@v3 references in our workflows. Could you please share those scan results? > According to a quick code search this project is using one of the actions with a v3 tag in at least one of its repos. J. On Mon, Nov 25, 2024 at 7:36 PM Jacob Wujciak wrote: > Hello Everyo

Re: [DISCUSS] Funnding available for security for individual maintainers (follow up from last infra roundtable)

2024-11-19 Thread Jarek Potiuk
couraged people to apply). And I strongly encourage others to the same - making people aware and distributing information via word of mouth might be indeed more effective than organisational/institutional effort. J. sob., 16 lis 2024, 19:21 użytkownik Jarek Potiuk napisał: > Hell

[DISCUSS] Funnding available for security for individual maintainers (follow up from last infra roundtable)

2024-11-16 Thread Jarek Potiuk
Hello Drew (and others), Following up from last week's infrastructure roundtable. I promised to write a short wrap-up on how I see the situation with funding in the OSS. So here it is. *TL:DR; I think there is a need to somehow (not sure how and who should lead it from the Foundation "officer" po

Re: [ANNOUNCEMENT] -- Infrastructure Roundtable -- Wednesday, November 13th, 2024 @ 1000 UTC

2024-10-31 Thread Jarek Potiuk
Kind request: can we also add it to the "ASF Infrastructure" or ASF calendar? We are around the DST changes all over the world (either right after or right before) and I find it really easy to make mistakes about the hours (despite all the right timezone mentioning)? It's a deeply frustrating exper

Re: Sharing Apache Pulsar's CI solution for Docker image sharing with GitHub Actions Artifacts within a single workflow

2024-10-22 Thread Jarek Potiuk
Love it. Already created an issue in Airflow CI/CD that will help us to simplify our workflows based on your experiences: https://github.com/apache/airflow/issues/43268 - will report back when we turn it into code. Thanks for sharing! J. On Tue, Oct 22, 2024 at 4:02 PM Nathan Hartman wrote: >

Re: [ANNOUNCEMENT] Infrastructure Roundtable -- Wednesday, July 3rd, 2024 @ 1700 UTC

2024-06-26 Thread Jarek Potiuk
Very interesting topic - and looking forward to taking part in the discussion :) On Wed, Jun 26, 2024 at 6:41 PM Andrew Wetmore wrote: > That's the sort of thing we are going to talk about at the Roundtable. How > do we make this work with the least friction and the highest confidence > that the

Re: Using Github Actions Trusted Publisher for PyPI releases ?

2024-06-25 Thread Jarek Potiuk
> > > A quick read includes that these reviewers can include teams which I > interpret as it can easily be the whole of the PMC who have linked to a > GitHub account. > I will explore it in detail when we will set it up. > > I wonder if a future enhancement would be to use an API to connect to th

Re: Using Github Actions Trusted Publisher for PyPI releases ?

2024-06-25 Thread Jarek Potiuk
3qckcj45sxo [2] Github Actions Environments: https://docs.github.com/en/actions/deployment/targeting-different-environments/using-environments-for-deployment J. On Thu, Jun 20, 2024 at 10:05 PM Jarek Potiuk wrote: > I have not planned to write an action, I thought more of bash/python to > pull the a

Re: Using Github Actions Trusted Publisher for PyPI releases ?

2024-06-20 Thread Jarek Potiuk
others. > > Cheers, > Greg > InfraAdmin, ASF > > > On Thu, Jun 20, 2024 at 7:10 AM Jarek Potiuk wrote: > > > Unless I hear otherwise, I **assume** there are no big reasons against > > this. My plan is that I will add a Github Action (manually triggered, > > lim

Re: Using Github Actions Trusted Publisher for PyPI releases ?

2024-06-20 Thread Jarek Potiuk
We did something similar at Log4j to publish releases (I did not do > that set up but I do help on the project) so that a release manager > can use GitHub to publish releases instead of doing it on hardware > they control. > > Gary > > On Thu, Jun 20, 2024 at 8:10 AM Jarek Potiuk w

Re: Using Github Actions Trusted Publisher for PyPI releases ?

2024-06-20 Thread Jarek Potiuk
ps for all other projects that upload packages to PyPI and use GitHub. Does that make sense? J. On Fri, Jun 14, 2024 at 12:14 PM Jarek Potiuk wrote: > >> My only question is what do the users see in terms of the verified >> identity that performed the release. Does it still appear

Using Github Actions Trusted Publisher for PyPI releases ?

2024-06-14 Thread Jarek Potiuk
Hello everyone, TL;DR; I would like to hear what you think about using GitHub Actions as a Trusted Publisher to publish Python packages to PyPI? A bit more context: I've been involved in a few discussions recently about Python / PyPI trusted publisher [1] integration for Apache Airflow and other

Reproducible builds [Airflow] -> done

2024-01-14 Thread Jarek Potiuk
Hey everyone, I just wanted to share a little accomplishment I (mostly) implemented in Airflow - I just merged the last PR to get fully reproducible builds for all thePython artifacts we produce and publish in downloads.apache.org (python whl, sdist packages, source tarballs). All our 90 or so ar

Re: Github hard limit on job execution time

2023-04-18 Thread Jarek Potiuk
thanks for all the feedback. At this time there are no sponsors for > Geode > > > so cannot have self-hosted runners. I have already split the job > running > > > tests into multiple jobs by gradle module, and even then one particular > > > gradle module takes more

Re: Github hard limit on job execution time

2023-04-13 Thread Jarek Potiuk
In many cases it can be done with choosing a bigger machine with more CPUS and parallelising as others mentioned. This is cool if your tests are pure unit tests and you can add just `--xdist` flag or similar (this is a pytest extension to run your tests in parallel with as many CPUs as you can). Ho

Re: Multi-arch container images on DockerHub

2022-12-06 Thread Jarek Potiuk
ver 2 hours, compare to 10 > >> minutes for amd64), but this approach eliminates the need to build for > more > >> than one platform at a time, so if the arm64 job were to run on an arm64 > >> runner in the future, that wouldn't need to further complicate

Re: Multi-arch container images on DockerHub

2022-12-06 Thread Jarek Potiuk
In Airflow we have a bit more complex setting (we are building 2x5x2 different images and they are different sets of them for different branches), Building images for Airflow takes quite some time (installing many dependencies) so qemu was out of the question (several hours to build single image).

Bigger Public Runners on GiHub

2022-12-05 Thread Jarek Potiuk
Hello here, We've discussed recently on the possibility of enabling bigger public runners for GitHub Actions. I recall that it was supposed to be raised at a meeting with GitHub. We would very much like to start using bigger runners - we started to experience much more frequent intermittent failur

Re: Meeting tomorrow

2022-11-10 Thread Jarek Potiuk
Hmm. Is it just me, or the link from builds page is incorrect? Got "this link is invalid". On Wed, Nov 9, 2022 at 9:55 AM Jarek Potiuk wrote: > > +1 > > On Wed, Nov 9, 2022 at 8:20 AM Gavin McDonald wrote: >> >> Just a reminder, as advertised on the 27th Octobe

Re: Meeting tomorrow

2022-11-09 Thread Jarek Potiuk
+1 On Wed, Nov 9, 2022 at 8:20 AM Gavin McDonald wrote: > Just a reminder, as advertised on the 27th October so hopefully > plenty of notice this time! - We have a builds meeting tomorrow. > > Add your name to the cwiki page if you plan on attending. > > > https://cwiki.apache.org/confluence/dis

Re: Meeting Summary 20th October 2022

2022-10-26 Thread Jarek Potiuk
Thanks. Really helpful :) On Wed, Oct 26, 2022 at 8:18 AM Gavin McDonald wrote: > > Hi All, > > Meeting notes from last week are now up at: > > https://cwiki.apache.org/confluence/display/BUILDS/Builds+Agenda+--+2022-10-20 > > Future meeting are scheduled for 1530 UTC on the 2nd Thursday of each

Re: Meeting this Thursday

2022-10-19 Thread Jarek Potiuk
Unfortunately, I am at Cape Canaveral, Kennedy Space Centre, and watching the Starlink Launch about that time (or at Atlantis Shuttle tour depending on the launch progress) so I will have to pass this time :(. But I'm looking forward to the result of the Github discussion afterwards. I reviewed th

Re: Building with Travis - anyone?

2022-09-01 Thread Jarek Potiuk
We moved out in Airflow from Travis to GA 2 years ago - because for us it started to degrade back then. We never looked back. There are a few docs and on-going discussions that you might want to check out as well Phil: * The "status" and "some overview" - look here https://cwiki.apache.org/confl

Re: GitHub Actions runners

2022-08-16 Thread Jarek Potiuk
As an original author - not much has changed since. I am regularly following what Github Action released to see if the problems we raised to them were addressed or not. All the updates regarding the security tightening are updated in the wiki page (not much changed - there are a number of watchout

Re: GitHub Action Status

2022-05-11 Thread Jarek Potiuk
Pretty current. I refreshed it recently. On Wed, May 11, 2022 at 12:04 AM Giovanni Bechis wrote: > Hi, > I was planning to work on SpamAssassin regression tests using GitHub > actions but I noticed this wiki: > > https://cwiki.apache.org/confluence/display/BUILDS/GitHub+Actions+status > > is the

Re: GitHub Actions - Change in Behaviour - Automatic Pull Request Creation via GH

2022-05-05 Thread Jarek Potiuk
GitHub does NOT make things easy :) On Thu, May 5, 2022 at 10:38 AM Gavin McDonald wrote: > On Thu, May 5, 2022 at 10:15 AM Jarek Potiuk wrote: > > > See > > > > > https://github.blog/changelog/2022-05-03-github-actions-prevent-github-actions-from-creating-

Re: GitHub Actions - Change in Behaviour - Automatic Pull Request Creation via GH

2022-05-05 Thread Jarek Potiuk
See https://github.blog/changelog/2022-05-03-github-actions-prevent-github-actions-from-creating-and-approving-pull-requests/- that prevention was introduced by GitHub 2 days ago. On Thu, May 5, 2022 at 9:57 AM Richard Zowalla wrote: > Hi all, > > we automatically re-generate some files during a

Re: Meeting next week

2022-04-21 Thread Jarek Potiuk
+1 On Thu, Apr 21, 2022 at 10:35 PM Gavin McDonald wrote: > > Hi All, > > I have been terrible at remembering to organise our next builds@ meeting, > > I'd like to go for Next Thursday 28th April at 4PM UTC. > > If that is agreeable with everyone I'll lock it in and open up a wiki page > for > an

Re: ETA on GHA self-hosted runner security audit?

2022-04-13 Thread Jarek Potiuk
On Sat, Apr 9, 2022 at 8:10 AM Chesnay Schepler wrote: > > i think we'd be alright with those caveats. > > 1) It depends on how far you wanna go. If the goal is to only protect > the workflow files themselves (and not any scripts that the workflow calls), > then you only need to check out a single

Re: ETA on GHA self-hosted runner security audit?

2022-04-07 Thread Jarek Potiuk
f course be nicer, but our > machines don't support that. > > For 3rd party actions we, as of right now, only intend to use those > provided by Github (things like checkout or artifact management). Our > pipeline isn't really complex or fancy, just expensive. > Since thes

Re: ETA on GHA self-hosted runner security audit?

2022-04-06 Thread Jarek Potiuk
; > I have very much read that. > > On 06/04/2022 19:22, Jarek Potiuk wrote: > > Since you referred Ash's link you probably have not read this: > > > > However this is not something to tackle lightly, as Infra *will not manage > > or secure your VM* - that i

Re: ETA on GHA self-hosted runner security audit?

2022-04-06 Thread Jarek Potiuk
Since you referred Ash's link you probably have not read this: However this is not something to tackle lightly, as Infra *will not manage or secure your VM* - that is up to you. On Wed, Apr 6, 2022 at 7:21 PM Chesnay Schepler wrote: > This article also lists self-hosted runners as an option:

Re: ETA on GHA self-hosted runner security audit?

2022-04-06 Thread Jarek Potiuk
Hello Chesnay, I think you have a bit too high of expectations and I am not sure why. Not sure who you talked to at Airflow, but we always underline and stress and warn that our solution is really "experimental" and "works for us" because we invested awfully (and I mean awfully) lot of time in ma

Re: builds meeting

2022-02-18 Thread Jarek Potiuk
Fine for me if it's 5pm UTC. Proposal/Suggestion - I have been organising a few meetings with big group of people and "doodle" worked beautifully for that: https://doodle.com/dashboard - you can send a link with a number of proposals matching times where you are available and people can mark thei

Re: Disable GH Actions from approving PRs

2022-02-08 Thread Jarek Potiuk
It would only work If you want "All" your PRs to be "approved". => It would only work If you want "All" your PRs to be "approved" by the GH Actions. On Tue, Feb 8, 2022 at 4:00 PM Jarek Potiuk wrote: > Depends on the scheme you choose. > It

Re: Disable GH Actions from approving PRs

2022-02-08 Thread Jarek Potiuk
thub actions that verify stuff, then the project > >> configures this to N+1. > >> > >> Admittedly this would mean that every project has use this properly. > >> > >> On 08/02/2022 15:14, Jarek Potiuk wrote: > >>> In short - just to explain why:

Re: Disable GH Actions from approving PRs

2022-02-08 Thread Jarek Potiuk
Is it not possible to control the number of approvals that are required? > > So if a project has N github actions that verify stuff, then the project > configures this to N+1. > > Admittedly this would mean that every project has use this properly. > > On 08/02/2022 15:14, Ja

Re: Disable GH Actions from approving PRs

2022-02-08 Thread Jarek Potiuk
In short - just to explain why: The "protected branches" feature is the way how to make sure that the code is looked at and approved by at least 1 commiter in order to be merged. This is a strong protection - not only from UI but also prevents you from fast-forward the "prtected branch" to the tip

Re: dockerhub limitations - maximum user count reached

2022-01-21 Thread Jarek Potiuk
believe our jFrog artifactory instance can act as a docker > registry. > > -Chris > > > > > > On Jan 21, 2022, at 9:30 AM, Jarek Potiuk wrote: > > > > We are pushing DockerHub images using personal accounts in Airflow. I > > believe we have three "

Re: dockerhub limitations - maximum user count reached

2022-01-21 Thread Jarek Potiuk
We are pushing DockerHub images using personal accounts in Airflow. I believe we have three "active" users - Kaxil, Jarek, Ash, likely some more seasoned PMC members (those can be removed), Basically everyone who is a member of the "airflow" team in DockerHub. But we also would like to add one or t

Re: ephemeral builds via AWS ECS and/or EKS? GPU Nodes?

2021-12-31 Thread Jarek Potiuk
We do not use Jenkins but we do have self-hosted runners in Airflow - we use VMS and auto-scaling groups rather than ECS or EKS, no GPU nodes though, so I am not sure how it would be useful. But we are looking into using EKS instead at some point, so maybe that's a good opportunity to try it out b

Re: Pushing Docker Images

2021-11-16 Thread Jarek Potiuk
Cool! I will take a look at that in the coming weeks :) On Tue, Nov 16, 2021 at 11:35 AM Martin Grigorov wrote: > > Hi Allen, > > I've just documented how one could use Oracle Cloud free plan to build and > test on Linux ARM64 for free! > Please check > https://martin-grigorov.medium.com/github-a

Re: Next builds@ meeting

2021-11-01 Thread Jarek Potiuk
I updated the graphs. I think things have changed a lot since December last year. There are no absurd-high (and way above the limits) peaks of workflows in progress for some projects (e.g. Pulsar). However a number of apache repos using GA grows steadily. Number of workflows "queued" fluctuates fo

Re: Owner apache user license exceeded

2021-09-16 Thread Jarek Potiuk
Maybe the user licence exceed is somewhat related and someone is exploiting the leakage ? Remote possibility, but who knows. On Thu, Sep 16, 2021 at 11:22 PM Jarek Potiuk wrote: > Just learned something (not using Travis CI but I have not seen this > posted here). Several days ago Travis

Re: Owner apache user license exceeded

2021-09-16 Thread Jarek Potiuk
Just learned something (not using Travis CI but I have not seen this posted here). Several days ago Travis CI had a serious security breach. In short - for about 1 hr anyone could get access to any secrets stored in any public project. https://www.infoq.com/news/2021/09/travis-ci-secrets-leak/ "I

Re: docker login

2021-09-15 Thread Jarek Potiuk
Question out of curiosity (I am not a Jenkins user but I know why we had to run 'docker logins' on private runners). Are Jenkin's Public IP addresses excluded from the Pull Rate limit? I am asking, because if not then preventing login might cause a problem with hitting 100 pulls/day/IP limit (whi

Re: Dependabot-like solution for Apache projects

2021-09-03 Thread Jarek Potiuk
Agree with Christopher that "technically" this does not matter if branch is fork PR or branch PR. And I also see the usefulness of Dependabot. I used it in the past and it's been extremely easy and helpful - with all the changelogs/release notes right in the PR you could find in exactly the moment

Re: Re: Dependabot-like solution for Apache projects

2021-08-30 Thread Jarek Potiuk
I believe that changed when Github bought dependabot and it become "embedded" in GitHub soon after: https://dependabot.com/blog/hello-github/ J. On Mon, Aug 30, 2021 at 3:43 PM Lewis John McGibbney wrote: > Thanks Gary and Sebb. > How do I turn dependabot on? Last time I tried I was informed t

Re: Beta access to the new GitHub issues for Apache ?

2021-07-20 Thread Jarek Potiuk
Actually this is the conversation - https://the-asf.slack.com/archives/CBX4TSBQ8/p1624615203074400. The previous link was where I spoke about it in Slack. J. On Tue, Jul 20, 2021 at 4:17 PM Jarek Potiuk wrote: > Sure. Conversation here (mostly with Daniel): > https://the-asf.sla

Re: Beta access to the new GitHub issues for Apache ?

2021-07-20 Thread Jarek Potiuk
21 at 3:11 PM Jarek Potiuk wrote: > > > Thanks Gavin. As explained on slack, > > > Explained on Slack? - Can't recall this, can I get a link to the > conversation please. > > I have a meeting with Matthew Butler- > > who is the PM of the new GitHub Issues on T

Re: Beta access to the new GitHub issues for Apache ?

2021-07-18 Thread Jarek Potiuk
wrote: > Hi All, > > Jarek, signed up. > > On Fri, Jun 25, 2021 at 10:33 AM Jarek Potiuk wrote: > > > Hello everyone (especially the INFRA :) ). > > > > GitHub recently introduced a new beta feature (limited access) of a > > new way of managing Github

Beta access to the new GitHub issues for Apache ?

2021-06-25 Thread Jarek Potiuk
Hello everyone (especially the INFRA :) ). GitHub recently introduced a new beta feature (limited access) of a new way of managing Github Issues https://github.blog/2021-06-23-introducing-new-github-issues/ For now it can only be enabled for organisations, not individuals. It looks really cool,

Re: Failure with Github Actions from outside of the organization (out of a sudden!)

2021-06-20 Thread Jarek Potiuk
y get during your build. https://github.blog/changelog/2021-04-20-github-actions-control-permissions-for-github_token/ . This could help preventing sophisticated supply-chain attack for example like the recent codecov attack. > On Wed, Dec 30, 2020 at 5:25 PM Jarek Potiuk wrote: > > &

New Concurrency feature of GitHub Actions -> replaces "cancel-workflow" action of mine

2021-05-24 Thread Jarek Potiuk
Hello everyone, In case you have not noticed, Github Recently (19th of April) introduced a new feature "concurrency" which adds the possibility of cancelling duplicate workflows in a much "nicer" way than the "cancel-workflow-action" I wrote. https://github.blog/changelog/2021-04-19-github-action

Re: Ephemeral builds for manual testing

2021-05-04 Thread Jarek Potiuk
This is a great post and great approach from the Superset team. FYI. In Airflow we are working on enabling the very same approach that we did for AWS in GCP, with a slightly different approach when it comes to auto-scaling (a bit simpler I think). We will share it when we finish. J. On Mon, Ma

Re: Increase the number of parallel jobs in GitHub Actions at ASF organization level

2021-04-20 Thread Jarek Potiuk
> > Is it possible to share the raw data in some form? If you can publish > data in any form (csv? sqlite?) I can generate static html files with > python notebooks which can be shared with everybody... > > I am adding Tobiasz who can share it :). > (BTW, how do you get the data? Do you poll some

Re: Increase the number of parallel jobs in GitHub Actions at ASF organization level

2021-04-19 Thread Jarek Potiuk
Also some comments for the stats. This is good stuff Marton. > Apparently, what's named "jobhours" in your statistics is actually the > runtime for an entire workflow (the sum of all job runtimes for that > workflow). That's at least what I conclude if I look at this workflow, > which your table

Re: Increase the number of parallel jobs in GitHub Actions at ASF organization level

2021-04-19 Thread Jarek Potiuk
I really love what Hyukjin has done. I did not have the capacity to participate in this actively, but this is exactly the way to go, I think (with a caveat). Following the motorway metaphor - everyone (every contributor/committer, not every project) has their own lane and they do not interfere with

Re: Increase the number of parallel jobs in GitHub Actions at ASF organization level

2021-04-08 Thread Jarek Potiuk
> > > That's a good idea. We do need to thank Github to give free resources to > ASF projects, but it's better if we can make it a business: we allow > individual projects to sign deals with Github to get dedicated resources. > It's a bit wasteful to ask every project to set up its own dev ops, > u

Re: Increase the number of parallel jobs in GitHub Actions at ASF organization level

2021-04-07 Thread Jarek Potiuk
śr., 7 kwi 2021, 18:45 użytkownik ocket napisał: > If your project can afford it, you can add self-hosted GHA runners: > > https://docs.github.com/en/actions/hosting-your-own-runners/about-self-hosted-runners > The issue with that being that the machine running your actions will > necessaril

Re: Increase the number of parallel jobs in GitHub Actions at ASF organization level

2021-04-07 Thread Jarek Potiuk
Just a comment here - as I commented also in the ticket The document https://cwiki.apache.org/confluence/display/BUILDS/GitHub+Actions+status gives complete overview of where the Github Actions are for the ASF project. And we have some nice experiences in Apache Airflow that we will be able to

Re: Hitting GitHub API limits

2021-02-17 Thread Jarek Potiuk
Just one comment - you might want to take a look at https://cwiki.apache.org/confluence/display/BUILDS/GitHub+Actions+status. documents which describes some of the open issues we have with GitHub Actions - they relate to scalability/performance and security. Also - maybe the IP Addresses of ASF in

Re: GA workflow cancellation

2021-02-09 Thread Jarek Potiuk
ueue strain situation" chapter - where I explain how `allDuplicates` mode helps to fight it. J On Tue, Feb 9, 2021 at 7:53 PM Jarek Potiuk wrote: > > Er... so "source" workflow is the triggering workflow, and "target" > workflow is the cancelling workflow? Or

Re: GA workflow cancellation

2021-02-09 Thread Jarek Potiuk
Actually, the action is written in TypeScript not Javascript (Typescript is transpiled to javascript) and developing in Typescript if you have a good IDE is easy :). This might be easier than you think. J. On Tue, Feb 9, 2021 at 7:34 PM Antoine Pitrou wrote: > > Le 09/02/2021 à 19:28, Ja

Re: GA workflow cancellation

2021-02-09 Thread Jarek Potiuk
ed - this is why "alllDuplicates" cancel mode was introduced - I explained exactly how it works and how the "high strain" situation is handled in this comment: https://github.com/apache/pulsar/pull/9503#issuecomment-774644408 - I am moving part of the explanation to the documentation

Re: GA again unreasonably slow (again)

2021-02-09 Thread Jarek Potiuk
The reasoning for selective checks here: https://github.com/apache/airflow/blob/master/PULL_REQUEST_WORKFLOW.rst (correct link) On Tue, Feb 9, 2021 at 7:05 PM Jarek Potiuk wrote: > | The real hard problem is knowing when a change requires full regression > and integration testing

Re: GA again unreasonably slow (again)

2021-02-09 Thread Jarek Potiuk
with easy lightweight branches all being fully tested > > This is my 10,000 meter view. > > But then I’m old school and on my first job the mainframe printout > included how much the run I made was costing my boss in $. > > Best Regards, > Dave > > Sent from my iPhon

Re: GA again unreasonably slow (again)

2021-02-09 Thread Jarek Potiuk
I problem on every > platform. As your project scales up, CI becomes a Hard Problem. I don’t > think throwing hardware at it indefinitely works, though your research here > is finding most of the useful things. > > On Tue, Feb 9, 2021 at 02:21 Jarek Potiuk wrote: > > > The r

Re: GA again unreasonably slow (again)

2021-02-09 Thread Jarek Potiuk
y appreciate a little help from your side. So maybe you just submit 2-3 PRs yourself any time Monday - Friday 12pm CET -> 8pm CET - this is where regularly bottlenecks happen. Please let everyone know your findings J, On Tue, Feb 9, 2021 at 8:35 AM Allen Wittenauer wrote: > > > > O

Re: GA again unreasonably slow (again)

2021-02-08 Thread Jarek Potiuk
lpful for others who do. J On Tue, Feb 9, 2021 at 1:33 AM Allen Wittenauer wrote: > > > On Feb 7, 2021, at 4:44 PM, Jarek Potiuk wrote: > > > > If you are interested - my document is here. Open for comments - happy to > > add you as editors if you want (just send

Re: GA again unreasonably slow (again)

2021-02-08 Thread Jarek Potiuk
Lambertus wrote: > > > > On Feb 8, 2021, at 1:51 PM, Jarek Potiuk wrote: > > > > This uses https://github.com/actions/runner/pull/783 to not have > > un-trusted users run code (security is based on the actors of the commit > - > > commiter’s PRs and dire

Re: GA again unreasonably slow (again)

2021-02-08 Thread Jarek Potiuk
and an AWS Auto-Scaling Group pon., 8 lut 2021, 09:58 użytkownik Antoine Pitrou napisał: > > Hi Jarek, > > Thank you for the document. Could you tell us more about the "custom > security layer" that you implemented? > > Regards > > Antoine. > > &g

Re: GA again unreasonably slow (again)

2021-02-07 Thread Jarek Potiuk
we are going to validate them even more when we implement all the optimisations, but the conclusions should be pretty sound. https://docs.google.com/document/d/1ZZeZ4BYMNX7ycGRUKAXv0s6etz1g-90Onn5nRQQHOfE/edit# J. On Fri, Jan 8, 2021 at 10:02 PM Jarek Potiuk wrote: > > We should be able to

Re: Using GitHub Actions for Apache Hudi repo

2021-01-31 Thread Jarek Potiuk
their workflows - I am happy to help. Also if there are any other ideas how the current situation can be improved, you are most welcome. We are in this together :). J. On Sun, Jan 31, 2021 at 11:49 AM Jarek Potiuk wrote: > The space is created (thnks infra!), I turned my message into th

Re: Using GitHub Actions for Apache Hudi repo

2021-01-31 Thread Jarek Potiuk
. On Sun, Jan 31, 2021 at 9:06 AM Jarek Potiuk wrote: > Ticket created: > https://issues.apache.org/jira/projects/INFRA/issues/INFRA-21364 > > J. > > On Thu, Jan 28, 2021 at 5:59 AM Chris Lambertus wrote: > >> >> >> > On Jan 23, 2021, at 10:15 PM, Jarek

Re: Using GitHub Actions for Apache Hudi repo

2021-01-31 Thread Jarek Potiuk
Ticket created: https://issues.apache.org/jira/projects/INFRA/issues/INFRA-21364 J. On Thu, Jan 28, 2021 at 5:59 AM Chris Lambertus wrote: > > > > On Jan 23, 2021, at 10:15 PM, Jarek Potiuk wrote: > > > Let me explain then what I see as the current state of Github Actions

Re: Builds Meeting this Thursday

2021-01-13 Thread Jarek Potiuk
Sure. My bad and mental shortcut. > > Apache Airflow is a PMC. That Committee has *members* that are part of that > committee. Those members are NOT "PMCs". We have about 200 PMCs at the > Foundation, established by the Board. Please stop confusing *members* with > *committees*. > -- +48 660 7

Re: Builds Meeting this Thursday

2021-01-13 Thread Jarek Potiuk
One of PMCs @Airflow (in ASIA timezone) asked if the meeting can be recorded ? J. On Tue, Jan 12, 2021 at 10:42 PM Jarek Potiuk wrote: > Added my two topics. Thanks Gavin for this opportunity ! > > On Tue, Jan 12, 2021 at 10:20 PM Gavin McDonald > wrote: > >> Please list

Re: Builds Meeting this Thursday

2021-01-12 Thread Jarek Potiuk
Added my two topics. Thanks Gavin for this opportunity ! On Tue, Jan 12, 2021 at 10:20 PM Gavin McDonald wrote: > Please list on the cwiki meeting page any questions you have > for Brian so that I may send them to him ahead of time, if > possible. > > On Tue, Jan 12, 2021 at 10:00 PM Gavin McDon

Re: Builds Meeting this Thursday

2021-01-12 Thread Jarek Potiuk
Cool. Looking forward to it! On Tue, Jan 12, 2021 at 9:53 PM Gavin McDonald wrote: > Hi All, > > Sorry for the last minute notice, this Thursday the 14th January > at 1700 UTC time will be our next builds@ meeting. > > Just confirmed a few minutes ago, there will be a guest > representing Github

Re: ASF Jenkins usability [Was: Re: GA again unreasonably slow (again)]

2021-01-10 Thread Jarek Potiuk
> > I would suggest you have a direct chat with Greg Stein. > > > > Best Regards, > > Dave > > > > Sent from my iPhone > > > > > On Jan 10, 2021, at 9:08 AM, Jarek Potiuk wrote: > > > > > > On Sun, Jan 10, 2021 at 5:28 PM Matt Sic

Re: ASF Jenkins usability [Was: Re: GA again unreasonably slow (again)]

2021-01-10 Thread Jarek Potiuk
delegated to me and provided with the means of contacting GitHub and representing the ASF (but I doubt anyone would give me that power, it is a bit risky as with big power I would have no big responsibility. Tough call - I am not sure how else I can help INFRA/ASF to help me and others. J. > >

Re: ASF Jenkins usability [Was: Re: GA again unreasonably slow (again)]

2021-01-10 Thread Jarek Potiuk
end to fall short once a large organization like > Apache tries using something). > > Many of the features you’re asking from GA are likely non-trivial > architecture changes they’ll have to make to accommodate the non-trivial > use cases we have. Or maybe it isn’t and they’re just

Re: ASF Jenkins usability [Was: Re: GA again unreasonably slow (again)]

2021-01-09 Thread Jarek Potiuk
s": ["ashb"] } Or to only allow the given users, but not all contributors: "pullRequestSecurity": { "allowContributors": false, "allowedAuthors": ["ashb"] } Owners of the repo are always allowed to run jobs. On Sat, Ja

Re: ASF Jenkins usability [Was: Re: GA again unreasonably slow (again)]

2021-01-09 Thread Jarek Potiuk
> > > > > The multiple threads about how shitty those are in practice for your > > needs seem to indicate otherwise. Security and easy learning curves > > don't seem to get along too well, do they? > The usabilty, integration level (especially GitHub Actions), maintenance effort needed - thi is fa

Re: ASF Jenkins usability [Was: Re: GA again unreasonably slow (again)]

2021-01-09 Thread Jarek Potiuk
> > > > I presume you are referring to the fact that external non-committers > cannot force a build on a forked repo PR on the ASF Jenkins without > whitelisting [1]. This is by design because it is a huge security problem > to run unvetted 3rd party code on our build infrastructure. This is the >

Re: ASF Jenkins usability [Was: Re: GA again unreasonably slow (again)]

2021-01-09 Thread Jarek Potiuk
> > > > The ASF does not run Travis. We pay a large amount of money TO Travis for > additional workers on top of the free tier. The reason I ask for discussion > regarding Jenkins usability is because throwing infinite money at solutions > like Travis is not an option. Now you find yourself in the

Re: ASF Jenkins usability [Was: Re: GA again unreasonably slow (again)]

2021-01-08 Thread Jarek Potiuk
7;s where my > preference comes from. Jenkins is still useful for jobs that don't need to > run on forks, (e.g., periodically checking for Go version updates and > opening a PR if a minor version update is found) VAST majority of our traffic comes from PRs from forks. > > -Zac

Re: ASF Jenkins usability [Was: Re: GA again unreasonably slow (again)]

2021-01-08 Thread Jarek Potiuk
I used to love gradle and Jenkins and started to hate it. Once you move to python/javascript world, suddenly stuff that take you days in Java/Gradle take hours in Python/Javascript. And it is amplified in case of CI where you do not need to have an enterprise-grade system but a bunch of scripts to

Re: ASF Jenkins usability [Was: Re: GA again unreasonably slow (again)]

2021-01-08 Thread Jarek Potiuk
Let me just answer from my side. Personally - I hate and love Jenkins at the same time. I am a Jenkins user and developer for 15 years or so. I even developed mobile app plugins for jenkins and we run our own jenkins farm for mobile app CI. but some 5 years ago we switched to GitLab which was so

Re: GA again unreasonably slow (again)

2021-01-08 Thread Jarek Potiuk
> We should be able to make an efficient query via GraphQL API right? I found > the REST API for actions to be a little underwhelming. That was the first thing I checked when we started looking at the stats. Unfortunately last time that I checked (and I even opened an issue for that to Github sup

Re: GA again unreasonably slow (again)

2021-01-08 Thread Jarek Potiuk
worse by day > > The chart suggests that Pulsar, Spark and Airflow are the top contributors > to the queue. > I filed issues to Pulsar ( https://github.com/apache/pulsar/issues/9154 ) > and Spark ( https://issues.apache.org/jira/browse/SPARK-34053 ) > Hope they can do something to

GA again unreasonably slow (again)

2021-01-08 Thread Jarek Potiuk
least organize the meeting and urge them to fix the security problem for public self -hosted repositories? This is not a complaint, this is just crying for HELP ... We are terribly stuck. J, -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660

Re: Failure with Github Actions from outside of the organization (out of a sudden!)

2021-01-07 Thread Jarek Potiuk
t; SHA-pinned. > However, submodules and "git clone ...action" do not improve security, yet > they make it harder to automatically analyze workflow.yml. > > Jarek>Again is the matter of 'thinking' this way > > I have updated the PR so it uses SHA. Could you please

Re: Failure with Github Actions from outside of the organization (out of a sudden!)

2021-01-07 Thread Jarek Potiuk
an infra's decision. Also whether it is applicable to everyone, every repo, every project - I have no idea. I just wanted to share it with others so they can make informed decisions (and prepare for what's coming from infra). If anyone decides to bypass the requirements/policies which

Re: Failure with Github Actions from outside of the organization (out of a sudden!)

2021-01-07 Thread Jarek Potiuk
On Thu, Jan 7, 2021 at 6:56 PM Vladimir Sitnikov < sitnikov.vladi...@gmail.com> wrote: > What I mean is that there's a trivial workaround which does not require > significant changes to the repository layout. On top of that, it does not > change developer's workflows (they do not need to learn sub

Re: Failure with Github Actions from outside of the organization (out of a sudden!)

2021-01-07 Thread Jarek Potiuk
OpenJDK/install-jdk@v1 > with: > impl: openj9 > version: '8' > architecture: x64 > > After: > > - name: Prepare Actions > run: | > git clone -c advice.detachedHead=false --branch v1 --depth 1 > https://github.com/AdoptOpenJDK/install-j

Re: Failure with Github Actions from outside of the organization (out of a sudden!)

2021-01-06 Thread Jarek Potiuk
> > > > I am aware of how to use Docker in our build jobs... I am talking about > Docker based GitHub Actions. They can have published docker > images that they used so they do not get rebuilt all of the time. I called > out SuperLinter as just an example. > Same thing. You can build and push D

  1   2   >