Re: [DISCUSS] Create community "Apache YuniKorn" executor ?

2024-10-29 Thread Amogh Desai
> As I understand what It means - If I read it correctly, that it's mostly a deployment issue - we don't even have to have YuniKorn Executor - we can use K8S Executor, and it will work out of the box, with scheduling controlled by YuniKorn, but then we need to find a way to configure behaviour of t

Re: [VOTE] October 2024 PR of the Month

2024-10-29 Thread Amogh Desai
Wow, I wasn't aware of the PR: https://github.com/apache/airflow/pull/42943. Removing those many lines is extremely brave. My vote goes for that one too! Thanks & Regards, Amogh Desai On Tue, Oct 29, 2024 at 12:55 AM Jarek Potiuk wrote: > I was sure there were a few more candidates this mon

Re: [DISCUSS] Create community "Apache YuniKorn" executor ?

2024-10-29 Thread Jarek Potiuk
This all looks really good - sounds like something that we could only do in K8S executor and likely even make it compatible with Airflow 2 and release independently On Tue, Oct 29, 2024 at 1:30 PM Amogh Desai wrote: > > As I understand what It means - If I read it correctly, that it's mostly > a

Re: [VOTE] October 2024 PR of the Month

2024-10-29 Thread Jarek Potiuk
https://github.com/apache/airflow/pull/43445 On Tue, Oct 29, 2024 at 3:09 PM Jarek Potiuk wrote: > Just to add a bit - Daniel also attempted (for a few brave moments) > considered adding this PR with +60K lines of code. > > So not only brave, but also being a contender on both ends of the > spec

Re: [VOTE] October 2024 PR of the Month

2024-10-29 Thread Jarek Potiuk
Just to add a bit - Daniel also attempted (for a few brave moments) considered adding this PR with +60K lines of code. So not only brave, but also being a contender on both ends of the spectrum... The best ... and not the best PR of the month maybe :D J. On Tue, Oct 29, 2024 at 1:33 PM Amogh Des

Re: [DISCUSS] Create community "Apache YuniKorn" executor ?

2024-10-29 Thread Jarek Potiuk
This is cool. As I understand what It means - If I read it correctly, that it's mostly a deployment issue - we don't even have to have YuniKorn Executor - we can use K8S Executor, and it will work out of the box, with scheduling controlled by YuniKorn, but then we need to find a way to configure b

Re: [VOTE] Airflow Providers prepared on October 27, 2024

2024-10-29 Thread Wei Lee
-1 (non-binding) for amazon: 9.1.0rc1 and fab: 1.5.0rc1. They do not work with airflow 2.10.x on dataset permissions. corresponding PRs have been sent https://github.com/apache/airflow/pull/43470 https://github.com/apache/airflow/pull/43469 Best, Wei > On Oct 28, 2024, at 7:18 AM, Vishnu Chilu

Re: New committer: Gopal Dirisao (dirrao)

2024-10-29 Thread Vikram Koka
Congratulations Gopal, well deserved! Vikram On Tue, Oct 29, 2024 at 12:51 PM Oliveira, Niko wrote: > Congrats! Welcome aboard :) > > Cheers, > Niko > > > From: Bishundeo, Rajeshwar > Sent: Tuesday, October 29, 2024 9:42:34 AM > To: dev@airflow.apache.org > Su

Re: New committer: Pavan Kuman Gopidesu (gopidesupavan)

2024-10-29 Thread Vikram Koka
Congratulations Pavan, well deserved! Vikram On Tue, Oct 29, 2024 at 12:53 PM Oliveira, Niko wrote: > Congrats! Great to have you in the community :) > > Cheers, > Niko > > > > From: Bishundeo, Rajeshwar > Sent: Tuesday, October 29, 2024 9:42:49 AM > To: dev@a

Re: [VOTE] October 2024 PR of the Month

2024-10-29 Thread Jarek Potiuk
Damn.. I thought it was the previous month ;) . Been traveling for too long. Tough, Tough choice this time On Tue, Oct 29, 2024 at 10:58 PM Jens Scheffler wrote: > My vote is not on the list so far but I'd vote for #42505 - "Split > providers out of the main "airflow/" tree into a UV worksp

Re: New committer: Pavan Kuman Gopidesu (gopidesupavan)

2024-10-29 Thread Kaxil Naik
Congrats Pavan On Tue, 29 Oct 2024 at 21:55, Jens Scheffler wrote: > Welcome to the team! > > On 29.10.24 19:41, Vikram Koka wrote: > > Congratulations Pavan, well deserved! > > > > Vikram > > > > > > On Tue, Oct 29, 2024 at 12:53 PM Oliveira, Niko > > > wrote: > > > >> Congrats! Great to have

Re: [VOTE] October 2024 PR of the Month

2024-10-29 Thread Wei Lee
Wow, so many awesome PRs! My vote goes to https://github.com/apache/airflow/pull/42959. This is probably not the "best" PR. As a new contributor, he persists in finishing the PR despite numerous comments and conflicts regarding this not-so-huge request. I think this is something we could encourage

Re: New committer: Pavan Kuman Gopidesu (gopidesupavan)

2024-10-29 Thread Wei Lee
Congratulations Pavan! Best, Wei Kaxil Naik 於 2024年10月30日 週三 上午6:13寫道: > Congrats Pavan > > On Tue, 29 Oct 2024 at 21:55, Jens Scheffler > wrote: > > > Welcome to the team! > > > > On 29.10.24 19:41, Vikram Koka wrote: > > > Congratulations Pavan, well deserved! > > > > > > Vikram > > > > > >

Re: New committer: Gopal Dirisao (dirrao)

2024-10-29 Thread Wei Lee
Congratulations Gopal! Best, Wei Jens Scheffler 於 2024年10月30日 週三 上午5:55寫道: > Welcome to the team! Seems we grow rapidly! > > On 29.10.24 20:35, Ryan Hatter wrote: > > Congrats! So well deserved! > > > > On Tue, Oct 29, 2024 at 2:40 PM Vikram Koka > > > wrote: > > > >> Congratulations Gopal, we

Re: New committer: Karthikeyan Singaravelan (tirkarthi)

2024-10-29 Thread Wei Lee
Congratulations Karthikeyan! Best, Wei Jens Scheffler 於 2024年10月30日 週三 上午5:55寫道: > Welcome to the team! > > On 29.10.24 19:40, Vikram Koka wrote: > > Congratulations Karthikeyan, well deserved! > > > > Vikram > > > > > > On Tue, Oct 29, 2024 at 12:51 PM Oliveira, Niko > > > wrote: > > > >> Con

Re: [VOTE] October 2024 PR of the Month

2024-10-29 Thread Jens Scheffler
My vote is not on the list so far but I'd vote for #42505 - "Split providers out of the main "airflow/" tree into a UV workspace project" by ashb On 29.10.24 18:46, Jarek Potiuk wrote: Let's improve the script and the heuristics :) -> propose a PR (and may be that one will be candidate for the n

Re: New committer: Karthikeyan Singaravelan (tirkarthi)

2024-10-29 Thread Jens Scheffler
Welcome to the team! On 29.10.24 19:40, Vikram Koka wrote: Congratulations Karthikeyan, well deserved! Vikram On Tue, Oct 29, 2024 at 12:51 PM Oliveira, Niko wrote: Congrats! Welcome to the team :) Cheers, Niko From: Bishundeo, Rajeshwar Sent: Tuesday,

Re: New committer: Gopal Dirisao (dirrao)

2024-10-29 Thread Jens Scheffler
Welcome to the team! Seems we grow rapidly! On 29.10.24 20:35, Ryan Hatter wrote: Congrats! So well deserved! On Tue, Oct 29, 2024 at 2:40 PM Vikram Koka wrote: Congratulations Gopal, well deserved! Vikram On Tue, Oct 29, 2024 at 12:51 PM Oliveira, Niko wrote: Congrats! Welcome aboard

Re: New committer: Pavan Kuman Gopidesu (gopidesupavan)

2024-10-29 Thread Jens Scheffler
Welcome to the team! On 29.10.24 19:41, Vikram Koka wrote: Congratulations Pavan, well deserved! Vikram On Tue, Oct 29, 2024 at 12:53 PM Oliveira, Niko wrote: Congrats! Great to have you in the community :) Cheers, Niko From: Bishundeo, Rajeshwar Sent:

Re: New committer: Karthikeyan Singaravelan (tirkarthi)

2024-10-29 Thread Brent Bovenzi
Congratulations Karthikeyan, well deserved! On Tue, Oct 29, 2024 at 11:58 AM Jarek Potiuk wrote: > The Project Management Committee (PMC) for Apache Airflow > has invited Karthikeyan Singaravelan to become a committer and we are > pleased > to announce that they have accepted. > > Please join us

Re: [VOTE] October 2024 PR of the Month

2024-10-29 Thread Amogh Desai
So I played around a bit with the PROTM script and this is the result I got: Top 10 out of 461 PRs: * PR #42782: AIP-84 Migrate get connections to FastAPI API #42571. Score: 190.008 * PR #43040: Masking configuration values irrelevant to DAG author. Score: 162.89 * PR #42080: Fix consistent ret

Re: [VOTE] October 2024 PR of the Month

2024-10-29 Thread Amogh Desai
Ok I got that part! The reason the score is low is because there are fewer reviews, comments and commits. But there are too many lines changed, which we should value more! Thanks & Regards, Amogh Desai On Tue, Oct 29, 2024 at 10:32 PM Amogh Desai wrote: > So I played around a bit with the PR

Re: New committer: Gopal Dirisao (dirrao)

2024-10-29 Thread Bishundeo, Rajeshwar
Congratulations Gopal!! -- Rajesh On 2024-10-29, 12:31 PM, "Buğra Öztürk" mailto:ozturkbugr...@gmail.com>> wrote: CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.

Re: New committer: Karthikeyan Singaravelan (tirkarthi)

2024-10-29 Thread Bishundeo, Rajeshwar
Congratulations Karthikeyan! -- Rajesh On 2024-10-29, 12:31 PM, "Buğra Öztürk" mailto:ozturkbugr...@gmail.com>> wrote: CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.

Re: New committer: Pavan Kuman Gopidesu (gopidesupavan)

2024-10-29 Thread Bishundeo, Rajeshwar
Congratulations Pavan!! -- Rajesh On 2024-10-29, 12:32 PM, "Buğra Öztürk" mailto:ozturkbugr...@gmail.com>> wrote: CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.

Re: New committer: Pavan Kuman Gopidesu (gopidesupavan)

2024-10-29 Thread Oliveira, Niko
Congrats! Great to have you in the community :) Cheers, Niko From: Bishundeo, Rajeshwar Sent: Tuesday, October 29, 2024 9:42:49 AM To: dev@airflow.apache.org Subject: RE: [EXT] New committer: Pavan Kuman Gopidesu (gopidesupavan) CAUTION: This email originated f

Re: [VOTE] October 2024 PR of the Month

2024-10-29 Thread Jarek Potiuk
Let's improve the script and the heuristics :) -> propose a PR (and may be that one will be candidate for the next PR of the month :D ) On Tue, Oct 29, 2024 at 6:07 PM Amogh Desai wrote: > Ok I got that part! > > The reason the score is low is because there are fewer reviews, > comments and comm

New committer: Karthikeyan Singaravelan (tirkarthi)

2024-10-29 Thread Jarek Potiuk
The Project Management Committee (PMC) for Apache Airflow has invited Karthikeyan Singaravelan to become a committer and we are pleased to announce that they have accepted. Please join us in welcoming Karthikeyan to their new role and responsibility in our project community. Jarek On behalf of th

New committer: Pavan Kuman Gopidesu (gopidesupavan)

2024-10-29 Thread Jarek Potiuk
The Project Management Committee (PMC) for Apache Airflow has invited Pavan Kuman Gopidesu to become a committer and we are pleased to announce that they have accepted. Please join us in welcoming Pavan to their new role and responsibility in our project community. Jarek On behalf of the Airflow

New committer: Gopal Dirisao (dirrao)

2024-10-29 Thread Jarek Potiuk
The Project Management Committee (PMC) for Apache Airflow has invited Gopal Dirisao to become a committer and we are pleased to announce that they have accepted. Please join us in welcoming Gopal to their new role and responsibility in our project community. Jarek On behalf of the Airflow PMC

Re: New committer: Gopal Dirisao (dirrao)

2024-10-29 Thread Buğra Öztürk
Congratulations Gopal, well deserved! On Tue, 29 Oct 2024, 17:00 Jarek Potiuk, wrote: > The Project Management Committee (PMC) for Apache Airflow > has invited Gopal Dirisao to become a committer and we are pleased > to announce that they have accepted. > > Please join us in welcoming Gopal to t

Re: New committer: Karthikeyan Singaravelan (tirkarthi)

2024-10-29 Thread Buğra Öztürk
Congratulations Karthikeyan! On Tue, 29 Oct 2024, 17:20 Brent Bovenzi, wrote: > Congratulations Karthikeyan, well deserved! > > On Tue, Oct 29, 2024 at 11:58 AM Jarek Potiuk wrote: > > > The Project Management Committee (PMC) for Apache Airflow > > has invited Karthikeyan Singaravelan to become

Re: New committer: Pavan Kuman Gopidesu (gopidesupavan)

2024-10-29 Thread Buğra Öztürk
Congrats Pavan, well deserved! 🎉 On Tue, 29 Oct 2024, 17:00 Jarek Potiuk, wrote: > The Project Management Committee (PMC) for Apache Airflow > has invited Pavan Kuman Gopidesu to become a committer and we are pleased > to announce that they have accepted. > > Please join us in welcoming Pavan to

Re: [VOTE] Airflow Providers prepared on October 27, 2024

2024-10-29 Thread Amogh Desai
+1 non binding. Tested example DAGs from my test set focusing on CNCF, and mongo provider. Things seem to work as expected. Thanks & Regards, Amogh Desai On Tue, Oct 29, 2024 at 5:42 PM Wei Lee wrote: > -1 (non-binding) for amazon: 9.1.0rc1 and fab: 1.5.0rc1. They do not work > with airflow 2

Re: [VOTE] October 2024 PR of the Month

2024-10-29 Thread Amogh Desai
I am trying to take a stab at fixing the script for PROTM if there's any issue there. The script keeps running forever for me (or I am running out of patience too soon), trying to check it out. I will reach out to Briana if that's the case and try to figure it out. Thanks & Regards, Amogh Desai

Re: New committer: Karthikeyan Singaravelan (tirkarthi)

2024-10-29 Thread Oliveira, Niko
Congrats! Welcome to the team :) Cheers, Niko From: Bishundeo, Rajeshwar Sent: Tuesday, October 29, 2024 9:42:20 AM To: dev@airflow.apache.org Subject: RE: [EXT] New committer: Karthikeyan Singaravelan (tirkarthi) CAUTION: This email originated from outside of

Re: New committer: Gopal Dirisao (dirrao)

2024-10-29 Thread Oliveira, Niko
Congrats! Welcome aboard :) Cheers, Niko From: Bishundeo, Rajeshwar Sent: Tuesday, October 29, 2024 9:42:34 AM To: dev@airflow.apache.org Subject: RE: [EXT] New committer: Gopal Dirisao (dirrao) CAUTION: This email originated from outside of the organization. Do

Re: New committer: Gopal Dirisao (dirrao)

2024-10-29 Thread Ryan Hatter
Congrats! So well deserved! On Tue, Oct 29, 2024 at 2:40 PM Vikram Koka wrote: > Congratulations Gopal, well deserved! > > Vikram > > > On Tue, Oct 29, 2024 at 12:51 PM Oliveira, Niko > > wrote: > > > Congrats! Welcome aboard :) > > > > Cheers, > > Niko > > > >

Re: New committer: Karthikeyan Singaravelan (tirkarthi)

2024-10-29 Thread Vikram Koka
Congratulations Karthikeyan, well deserved! Vikram On Tue, Oct 29, 2024 at 12:51 PM Oliveira, Niko wrote: > Congrats! Welcome to the team :) > > > Cheers, > Niko > > > From: Bishundeo, Rajeshwar > Sent: Tuesday, October 29, 2024 9:42:20 AM > To: dev@airflow.ap

Re: [DISCUSS] Create community "Apache YuniKorn" executor ?

2024-10-29 Thread Amogh Desai
Building upon the POC done by Manikandan, I tried my hands at an experiment too. I wanted to mainly experiment with the Yunikorn admission controller, with an aim to make no changes to my older DAGs. Deployed a setup that looks like this: - Deployed Yunikorn in a kind cluster with the default