I was away for the last few days too, and couldn't pay much attention to my email and Github.
I am looking forward to Hussein's ppt in the next dev call, so that I can help in some of these areas as well. I have a decent knowledge of our CI and tests. Thanks & Regards, Amogh Desai On Wed, Aug 14, 2024 at 3:43 AM Jarek Potiuk <ja...@potiuk.com> wrote: > Fantastic ! That might help us to speed up our builds a lot - and we need > it looking at the number of PRs we keep on receiving with Airflow 3. Also > Self-hosted runners from ASF are not helping - thay are pretty unstable and > ARM runners are on ubuntu 20.04 which does not have ARM Python, and the > issue is not solved for a long time already > https://issues.apache.org/jira/projects/INFRA/issues/INFRA-25990 > > On Wed, Aug 14, 2024 at 12:05 AM Hussein Awala <huss...@awala.fr> wrote: > > > I can prepare a short presentation for the next dev call (22 Aug) to > > explain the architecture we tried to implement, why we chose it, and what > > the blockers are (mainly related to the infrastructure team). > > > > Also, I am still interested in completing this work. > > > > On Mon, Aug 12, 2024 at 3:58 AM Kaxil Naik <kaxiln...@gmail.com> wrote: > > > > > Added as agenda item for the next dev call (22 Aug) > > > > > > On Mon, 12 Aug 2024 at 00:25, Jarek Potiuk <ja...@potiuk.com> wrote: > > > > > > > I will let Hussein (if he has time) to share some more details :). > > > > > > > > Generally speaking we are using Github Actions as CI - so what we > > > > **really** need is auto-scaling k8S cluster where K8S Controller is > > > deployd > > > > and connected (via ASF infrastructure's Github APP) > > > > https://github.com/actions/actions-runner-controller. The last state > > we > > > > had > > > > - as far as I remember - Hussein already had a (Terraform?) > deployment > > > for > > > > it and it generally was depending on the ASF/ Infra authorisation / > > > setup. > > > > Then some fine-tuning / labels (small/medium/big instances) to > > > > define/findalize and extend it to be able to also run ARM instances. > > > > > > > > J. > > > > > > > > On Mon, Aug 12, 2024 at 1:10 AM Neil <neil4r...@gmail.com> wrote: > > > > > > > > > I have solid AWS and EKS knowledge, I'd offer my help if my skills > > are > > > > > applicable. > > > > > Which Infrastructure as Code and CI/CD frameworks are being > utilized > > > for > > > > > the testing Terraform Cloudformation? > > > > > I've had good experiences with Pulumi python. > > > > > Have you considered using EFS to handle the disk space needs? > > > > > > > > > > On Sun, Aug 11, 2024 at 6:18 PM Jarek Potiuk <ja...@potiuk.com> > > wrote: > > > > > > > > > > > Hello here, > > > > > > > > > > > > It would be great to have someone (or better two people) to get > > > engaged > > > > > in > > > > > > our test infrastructure work - this will improve everyone's > > > experience. > > > > > I > > > > > > **REALLY** think we should have other people that have engaged so > > > far, > > > > so > > > > > > that we can decrease the bus factor we have for our > infrastructure. > > > > > > > > > > > > Just after I was away for 5 days and without too much > connectivity > > > our > > > > > main > > > > > > was broken (lack of disk space for constraints generation) and > some > > > > mypy > > > > > > checks were failing for the last few days. > > > > > > > > > > > > This is unsustainable and we need to find people who will know > and > > be > > > > > able > > > > > > to fix this infrastructure. > > > > > > > > > > > > *Early warning* - I am planning 3 weeks holidays after Airflow > > > Summit - > > > > > and > > > > > > I won't be looking at my email/github during those days, which > > means > > > > that > > > > > > whoever will be working on Airflow 3 might be severely impacted > by > > > some > > > > > of > > > > > > those failures. > > > > > > > > > > > > Just to remind - until we have the k8S controller set up on our > AWS > > > > > > account and connected to our repo - we won't be able to use the > > > > credits > > > > > > that we got recently. So this is a good start. > > > > > > > > > > > > I created a high-level issue for that > > > > > > https://github.com/apache/airflow/issues/41388 and it waits for > > some > > > > > > volunteers to pick it up. It's a very important thing to do - we > > can > > > > > speed > > > > > > up many parts of our builds (for example release preparation - > but > > > also > > > > > > likely most of our tests) up to 4 times, which means that a lot > of > > > time > > > > > can > > > > > > be saved for waiting. > > > > > > > > > > > > Kaxil - I propose we should add a point at the next devcall - and > > > keep > > > > it > > > > > > as an unresolved Airflow 3 issue until it is well, unresolved. > > > > > > > > > > > > J. > > > > > > > > > > > > > > > > > > > > >
