I have solid AWS and EKS knowledge, I'd offer my help if my skills are applicable. Which Infrastructure as Code and CI/CD frameworks are being utilized for the testing Terraform Cloudformation? I've had good experiences with Pulumi python. Have you considered using EFS to handle the disk space needs?
On Sun, Aug 11, 2024 at 6:18 PM Jarek Potiuk <ja...@potiuk.com> wrote: > Hello here, > > It would be great to have someone (or better two people) to get engaged in > our test infrastructure work - this will improve everyone's experience. I > **REALLY** think we should have other people that have engaged so far, so > that we can decrease the bus factor we have for our infrastructure. > > Just after I was away for 5 days and without too much connectivity our main > was broken (lack of disk space for constraints generation) and some mypy > checks were failing for the last few days. > > This is unsustainable and we need to find people who will know and be able > to fix this infrastructure. > > *Early warning* - I am planning 3 weeks holidays after Airflow Summit - and > I won't be looking at my email/github during those days, which means that > whoever will be working on Airflow 3 might be severely impacted by some of > those failures. > > Just to remind - until we have the k8S controller set up on our AWS > account and connected to our repo - we won't be able to use the credits > that we got recently. So this is a good start. > > I created a high-level issue for that > https://github.com/apache/airflow/issues/41388 and it waits for some > volunteers to pick it up. It's a very important thing to do - we can speed > up many parts of our builds (for example release preparation - but also > likely most of our tests) up to 4 times, which means that a lot of time can > be saved for waiting. > > Kaxil - I propose we should add a point at the next devcall - and keep it > as an unresolved Airflow 3 issue until it is well, unresolved. > > J. >
