Re: [VOTE] Airflow Providers prepared on October 10, 2024

2024-10-10 Thread Wei Lee
+1 (non-binding), tested my changes and a few example dags Best, Wei > On Oct 11, 2024, at 5:23 AM, Jens Scheffler > wrote: > > +1 (binding) - Checked SVN, Check in Docker, Reproducible package build, > Licenses, Signatures > > On 10.10.24 13:15, Freddy Demiane wrote: >> +1 (non-binding) on g

Re: [DISCUSS] pools and deferrables

2024-10-10 Thread Daniel Standish
No worries. The other issue is good to know about. Just I'm trying to keep the discussion focused on the main question. People don't have a lot of excess time to attend to things like this and weigh in. We're lucky if we get 3 people to weigh in on a discussion item like this. So, I'm interest

Re: [DISCUSS] pools and deferrables

2024-10-10 Thread Pavankumar Gopidesu
Apologies if my earlier message wasn't clear! What I was trying to convey is that, since we're discussing pool and deferrable tasks. Yes, there's also been some discussion around managing deferred tasks at the DAG level, which might be relevant here. The PR I mentioned (https://github.com/apache/ai

Re: [VOTE] Airflow Providers prepared on October 10, 2024

2024-10-10 Thread Freddy Demiane
+1 (non-binding) on google providers package On Thu, Oct 10, 2024 at 3:40 AM Elad Kalif wrote: > Hey all, > > I have just cut the new wave Airflow Providers packages. This email is > calling a vote on the release, > which will last for 72 hours - which means that it will end on October 13, > 202

Re: [DISCUSS] pools and deferrables

2024-10-10 Thread Wei Lee
I’m wondering why do we want to remove this. The design seems to be reasonable, but yep, it might not be as helpful as mentioned. > is it useful to have it take up a slot at the first and last couple seconds of its lifecycle? methinks no. Some edge cases “might” be helpful. I guess? A deferrabl

RE: [EXT] Re: Dynamic Workflows Queries

2024-10-10 Thread Kirk Portas
Praveen, Great question! You cannot "add a new step" to a running workflow, but it is possible to dynamically decide which tasks to run based on the results of a previous (upstream) task. In practice this means that you would want to maintain a set of tasks in your DAG that you can dynamically

Re: [VOTE] Airflow Providers prepared on October 10, 2024

2024-10-10 Thread Jens Scheffler
+1 (binding) - Checked SVN, Check in Docker, Reproducible package build, Licenses, Signatures On 10.10.24 13:15, Freddy Demiane wrote: +1 (non-binding) on google providers package On Thu, Oct 10, 2024 at 3:40 AM Elad Kalif wrote: Hey all, I have just cut the new wave Airflow Providers packa