> > IMO, the approach on the patch isn't easily maintainable. Most of the > calculations are performed by SQL in a huge query. > It would be my preference to have many smaller queries and do part of the > calculations in python. This will be easier to understand, maintain and > debug in the future. Also, it will be easier to unit test.
You're talking about https://github.com/apache/airflow/pull/53492/ right? I agree. I share the skepticism that it must be one big ugly query. At a minimum it needs a lot more work and refinement. Not something that should be merged in the current state, even as experimental. Where is the PR from @Christos? On Wed, Aug 6, 2025 at 12:54 PM Jens Scheffler <j_scheff...@gmx.de.invalid> wrote: > Hi, > > I was (until now) not be able to re-read all the Slack discussion and > like to make this latest at the weekend. I also like Jarek fear that the > optimization makes the Scheduler rather hard to maintain. We also had > some points where we_thought_ we can contribute some optimizations > especially for Mapped Tasks and then considered the complexity of Mapped > Task Groups where the Depth-First Strategy would defeat all our drafted > optimizations. So also in our current apporach we are cutting down the > Dags in manageable pieces. > > So far (I believ, but anybody correct me if I am wrong) the scaling was > always documented only with options, no real upper boundary (other than > soft limtis) existing in the code. So the delivered product never > confirmed fixed upper limits. It might be good also to consider that we > document where we know there are natural or structural boundaries. So > hope that I can read more details the next days. > > Jens > > On 06.08.25 10:31, Jarek Potiuk wrote: > >> My main issue and the topic of this thread, has been that the scheduler > > does unnecessary work that leads to decreased throughput. My solution has > > been to limit the results of the query to the dag cap of active tasks > that > > the user has defined. > > > > Yes. I understand that. There are situations that cause this "unnecessary > > work" to be excessive and lead to lower performance and more memory > usage. > > This is quite "normal". No system in the world is optimized for all kinds > > of scenarios and sometimes you need to make trade-offs - for example > lower > > performance and maintainability (and support for MySQL and Postgres as > Ash > > pointed out in some other threads) which we have to make. There are > various > > optimisation goals we can chase: optimal performance and no wasted > > resources in certain situations and configurations is one of (many) goals > > we might have. Other goals might include: easier maintainability, better > > community collaboration, simplicity, less code to maintain, testability, > > also (what I mentioned before) sometimes deliberate not handling certain > > scenarios and introducing friction **might** be deliberate decision we > can > > take in order to push our users in the direction we want them to go. Yes. > > As community and maintainers we do not have to always "follow" our users > > behaviour - we can (and we often do) educate our users and show them > better > > ways of doing things. > > > > For example we had a LONG discussion whether to introduce caching of > > Variable values during Dag parsing - because we knew our users are often > > using Variables in top-level code of their Dags and this leads to a lot > of > > waste and high CPU and I/O usage by Dag processor. We finally implemented > > it as an experimental feature, but it was not at all certain we will - we > > had to carefully consider what we are trading in exchange for that > > performance - and whether it's worth it. > > > > Same here - I understand: there are some cases (arguably rather niche - > > with very large Dags) where scheduler does unnecessary processing and > > performance could be improved. Now - we need to understand what > trade-offs > > we need to make as maintainers and community (including our users) if we > > want to address it. We need to know what complexity is involved, whether > it > > will work with Postgres/MySQL and SQlite, whether we will be able to > > continue debugging and testing it. And whether we want to drive away our > > user from the modularisation strategy (smaller Dags) that we think makes > > more sense than bigger Dags. We have to think about what happens next. If > > we make "huge Dags" first-class-citizens, will it mean that we will have > to > > redesign our UI to support them? What should we do when someone opens up > an > > issue "I have this 1000000 task Dag and I cannot open Airflow UI - it > > crashes hard and makes my Airflow instance unusable - please fix it > ASAP". > > I certainly would like to avoid such a situation to stress our friend > > maintainers who work on UI - so also they should have a say on how > feasible > > it is to make it "easy" to have "huge Dags" for them. > > > > All those factors should be taken into account when you make a "product" > > decision. Performance gains for particular cases is just one of many > > factors to consider - and often not the most important ones. > > > > J. > > > > > > On Wed, Aug 6, 2025 at 7:34 AM Christos Bisias <christos...@gmail.com> > > wrote: > > > >> We also have a dag with dynamic task mapping that can grow immensely. > >> > >> I've been looking at https://github.com/apache/airflow/pull/53492. > >> > >> My main issue and the topic of this thread, has been that the scheduler > >> does unnecessary work that leads to decreased throughput. My solution > has > >> been to limit the results of the query to the dag cap of active tasks > that > >> the user has defined. > >> > >> The patch is more focused on the available pool slots. I get the idea > that > >> if we can only examine and queue as many tasks as available slots, then > we > >> will be efficiently utilizing the available slots to the max, the > >> throughput will increase and my issue will be solved as well. > >> > >> IMO, the approach on the patch isn't easily maintainable. Most of the > >> calculations are performed by SQL in a huge query. > >> > >> It would be my preference to have many smaller queries and do part of > the > >> calculations in python. This will be easier to understand, maintain and > >> debug in the future. Also, it will be easier to unit test. > >> > >> On Tue, Aug 5, 2025 at 10:20 PM Jarek Potiuk <ja...@potiuk.com> wrote: > >> > >>> Just a comment here - I am also not opposed as well if optimizations > will > >>> be implemented without impacting the more "regular"cases. And - > >> important - > >>> without adding huge complexity. > >>> > >>> The SQL queries I saw in recent PRs and discussions look both "smart" > and > >>> "scary" at the same time. Optimizations like that tend to lead to > >>> obfuscated, difficult to understand and reason code and "smart" > >> solutions - > >>> sometimes "too smart". And when it ends up with one or two people only > >>> being able to debug and fix problems connected with those, things > become > >> a > >>> little hairy. So whatever we do there, it **must** be not only "smart" > >> but > >>> also easy to read and well tested - so that anyone can run the tests > >> easily > >>> and reproduce potential failure cases. > >>> > >>> And yes I know I am writing this as someone who - for years was the > only > >>> one to understand our complex CI setup. But I think over the last two > >> years > >>> we are definitely going into, simpler, easier to understand setup and > we > >>> have more people on board who know how to deal with it and I think that > >> is > >>> a very good direction we are taking :). And I am sure that when I go > for > >> my > >>> planned 3 weeks holidays before the summit, everything will work as > >>> smoothly as when I am here - at least. > >>> > >>> Also I think there is quite a difference (when it comes to scheduling) > >> when > >>> you have mapped tasks versus "regular tasks". I think Airflow even > >>> currently behaves rather differently in those two different cases, and > >> also > >>> it has a well-thought and optimized UI experience to handle thousands > of > >>> them. Also the work of David Blain on Lazy Expandable Task Mapping will > >>> push the boundaries of what is possible there as well: > >>> https://github.com/apache/airflow/pull/51391. Even if we solve > >> scheduling > >>> optimization - the UI and ability to monitor such huge Dags is still > >> likely > >>> not something our UI was designed for. > >>> > >>> And I am fully on board with "splitting to even smaller pieces" and > >>> "modularizing" things - and "modularizing and splitting big Dags into > >>> smaller Dags" feels like precisely what should be done. And I think it > >>> would be a nice idea to try it and follow and see if you can't achieve > >> the > >>> same results without adding complexity. > >>> > >>> J. > >>> > >>> > >>> On Tue, Aug 5, 2025 at 8:47 PM Ash Berlin-Taylor <a...@apache.org> > wrote: > >>> > >>>> Yeah dynamic task mapping is a good case where you could easily end up > >>>> with thousands of tasksof in a dag. > >>>> > >>>> As I like to say, Airflow is a broad church and if we’re can > reasonably > >>>> support diverse workloads without impacting others (either the > >> workloads > >>>> out our available to support and maintain etc) then I’m all for it. > >>>> > >>>> In addition to your two items I’d like to add > >>>> > >>>> 3. That it doesn’t increase the db’s CPU disproportionally to the > >>>> increased task throughput > >>>> > >>>>> On 5 Aug 2025, at 19:14, asquator <asqua...@proton.me.invalid> > >> wrote: > >>>>> I'm glad this issue finally got enough attention and we can move it > >>>> forward. > >>>>> I took a look at @Christos's patch and it makes sense overall, it's > >>> fine > >>>> for the specific problem they experienced with max_active_tasks limit. > >>>>> For those unfamiliar with the core problem, the bug has a plenty of > >>>> variations where starvation happens due to different concurrency > >>>> limitations being nearly satiated, which creates the opportunity for > >> the > >>>> scheduler to pull many tasks and schedule none of them. > >>>>> To reproduce this bug, you need two conditions: > >>>>> 1. Many tasks (>> max_tis) belonging to one "pool", where "pool" is > >>> some > >>>> concurrency limitation of Airflow. Note that originally the bug was > >>>> discovered in context of task pools (see > >>>> https://github.com/apache/airflow/issues/45636). > >>>>> 2. The tasks are short enough (or the parallelism is large enough) > >> for > >>>> the tasks from the nearly starved pool to free some slots in every > >>>> scheduler's iteration. > >>>>> When we discovered a bug that starved our less prioritized pool, even > >>>> when the most prioritized pool was almost full (thanks to @nevcohen), > >> we > >>>> wanted to implement a similar patch @Christos suggested above, but for > >>>> pools. But then we realized this issue can arise due to limits > >> different > >>>> from task pools, including: > >>>>> max_active_tasks > >>>>> max_active_tis_per_dag > >>>>> max_active_tis_per_dagrun > >>>>> > >>>>> So we were able to predict the forecoming bug reports for different > >>>> kinds of starvation, and we started working on the most general > >> solution > >>>> which is the topic of this discussion. > >>>>> I want to also answer @potiuk regarding "why you need such large > >> DAGs", > >>>> but I will be brief. > >>>>> Airflow is an advanced tool for scheduling large data operations, and > >>>> over the years it has pushed to production many features that lead to > >>>> organizations writing DAGs that contain thousands of tasks. Most > >>> prominent > >>>> one is dynamic task mapping. This feature made us realize we can > >>> implement > >>>> a batching work queue pattern and create a task for every unit we have > >> to > >>>> process, say it's a file in a specific folder, a path in the > >> filesystem, > >>> a > >>>> pointer to some data stored in object storage, etc. We like to think > in > >>>> terms of splitting the work into many tasks. Is it good? I don't know, > >>> but > >>>> Airflow has already stepped onto this path, and we have to make it > >>>> technologically possible (if we can). > >>>>> Nevertheless, even if such DAGs are considered too big and splitting > >>>> them is a good idea (though you still have nothing to do with mapped > >>> tasks > >>>> - we create tens of thousands of them sometimes and expect them to be > >>>> processed in parallel), this issue does not only address the described > >>>> case, but many others, including prioritized pools, mapped tasks or > >>>> max_active_runs starvation on large backfills. > >>>>> The only part that's missing now is measuring query time (static > >>>> benchmarks) and measuring overall scheduling metrics in production > >>>> workloads (dynamic benchmarks). > >>>>> We're working hard on this crucial part now. > >>>>> > >>>>> We'd be happy to have any assistance from the community as regard to > >>> the > >>>> dynamic benchmarks, because every workload is different and it's > pretty > >>>> difficult to simulate the general case in such a hard-to-reproduce > >> issue. > >>>> We have to make sure that: > >>>>> 1. In a busy workload, the new logic boosts the scheduler's > >> throughput. > >>>>> 2. In a light workload, the nested windowing doesn't significantly > >> slow > >>>> down the computation. > >>>>> > >>>>>> On Monday, August 4th, 2025 at 9:00 PM, Christos Bisias < > >>>> christos...@gmail.com> wrote: > >>>>>> I created a draft PR for anyone interested to take a look at the > >> code > >>>>>> https://github.com/apache/airflow/pull/54103 > >>>>>> > >>>>>> I was able to demonstrate the issue in the unit test with much fewer > >>>> tasks. > >>>>>> All we need is the tasks brought back by the db query to belong to > >> the > >>>> same > >>>>>> dag_run or dag. This can happen when the first SCHEDULED tasks in > >> line > >>>> to > >>>>>> be examined are at least as many as the number of the tis per query. > >>>>>> > >>>>>> On Mon, Aug 4, 2025 at 8:37 PM Daniel Standish > >>>>>> daniel.stand...@astronomer.io.invalid wrote: > >>>>>> > >>>>>>>> The configurability was my recommendation for > >>>>>>>> https://github.com/apache/airflow/pull/53492 > >>>>>>>> Given the fact that this change is at the heart of Airflow I think > >>> the > >>>>>>>> changes should be experimental where users can switch between > >>>> different > >>>>>>>> strategies/modes of the scheduler. > >>>>>>>> If and when we have enough data to support that specific option is > >>>> always > >>>>>>>> better we can make decisions accordingly. > >>>>>>> Yeah I guess looking at #53492 > >>>>>>> https://github.com/apache/airflow/pull/53492 it does seem too > >> risky > >>> to > >>>>>>> just change the behavior in airflow without releasing it first as > >>>>>>> experimental. > >>>>>>> > >>>>>>> I doubt we can get sufficient real world testing without doing > >> that. > >>>>>>> So if this is introduced, I think it should just be introduced as > >>>>>>> experimental optimization. And the intention would be that > >> ultimately > >>>>>>> there will only be one scheduling mode, and this is just a way to > >>> test > >>>> this > >>>>>>> out more widely. Not that we are intending to have two scheduling > >>> code > >>>>>>> paths on a permanent basis. > >>>>>>> > >>>>>>> WDYT > >>>>>>> > >>>>>>> On Mon, Aug 4, 2025 at 12:50 AM Christos Bisias > >>> christos...@gmail.com > >>>>>>> wrote: > >>>>>>> > >>>>>>>>> So my question to you is: is it impossible, or just demanding or > >>>>>>>>> difficult > >>>>>>>>> to split your Dags into smaller dags connected with asset aware > >>>>>>>>> scheduling? > >>>>>>>> Jarek, I'm going to discuss this with the team and I will get you > >> an > >>>>>>>> answer > >>>>>>>> on that. > >>>>>>>> > >>>>>>>> I've shared this again on the thread > >>>>>>> > >> > https://github.com/xBis7/airflow/compare/69ab304ffa3d9b847b7dd0ee90ee6ef100223d66..scheduler-perf-patch > >>>>>>>> I haven't created a PR because this is just a POC and it's also > >>>> setting a > >>>>>>>> limit per dag. I would like to get feedback on whether it's better > >>> to > >>>>>>>> make > >>>>>>>> it per dag or per dag_run. > >>>>>>>> I can create a draft PR if that's helpful and makes it easier to > >> add > >>>>>>>> comments. > >>>>>>>> > >>>>>>>> Let me try to explain the issue better. From a high level > >> overview, > >>>> the > >>>>>>>> scheduler > >>>>>>>> > >>>>>>>> 1. moves tasks to SCHEDULED > >>>>>>>> 2. runs a query to fetch SCHEDULED tasks from the db > >>>>>>>> 3. examines the tasks > >>>>>>>> 4. moves tasks to QUEUED > >>>>>>>> > >>>>>>>> I'm focusing on step 2 and afterwards. The current code doesn't > >> take > >>>> into > >>>>>>>> account the max_active_tasks_per_dag. When it runs the query it > >>>> fetches > >>>>>>>> up to max_tis which is determined here > >>>>>>>> < > >>>>>>> > >> > https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/jobs/scheduler_job_runner.py#L697-L705 > >>>>>>>> . > >>>>>>>> > >>>>>>>> For example, > >>>>>>>> > >>>>>>>> - if the query number is 32 > >>>>>>>> - all 32 tasks in line belong to the same dag, dag1 > >>>>>>>> - we are not concerned how the scheduler picks them > >>>>>>>> - dag1 has max_active_tasks set to 5 > >>>>>>>> > >>>>>>>> The current code will > >>>>>>>> > >>>>>>>> - get 32 tasks from dag1 > >>>>>>>> - start examining them one by one > >>>>>>>> - once 5 are moved to QUEUED, it won't stop, it will keep > >> examining > >>>>>>>> the other 27 but won't be able to queue them because it has > >> reached > >>>>>>>> the > >>>>>>>> limit > >>>>>>>> > >>>>>>>> In the next loop, although we have reached the maximum number of > >>> tasks > >>>>>>>> for > >>>>>>>> dag1, the query will fetch again 32 tasks from dag1 to examine > >> them > >>>>>>>> and > >>>>>>>> try to queue them. > >>>>>>>> > >>>>>>>> The issue is that it gets more tasks than it can queue from the db > >>> and > >>>>>>>> then > >>>>>>>> examines them all. > >>>>>>>> > >>>>>>>> This all leads to unnecessary processing that builds up and the > >> more > >>>> load > >>>>>>>> there is on the system, the more the throughput drops for the > >>>> scheduler > >>>>>>>> and > >>>>>>>> the workers. > >>>>>>>> > >>>>>>>> What I'm proposing is to adjust the query in step 2, to check the > >>>>>>>> max_active_tasks_per_dag > >>>>>>>> > >>>>>>>>> run a query to fetch SCHEDULED tasks from the db > >>>>>>>> If a dag has already reached the maximum number of tasks in active > >>>>>>>> states, > >>>>>>>> it will be skipped by the query. > >>>>>>>> > >>>>>>>> Don't we already stop examining at that point? I guess there's two > >>>>>>>> things > >>>>>>>> > >>>>>>>>> you might be referring to. One is, which TIs come out of the db > >> and > >>>>>>>>> into > >>>>>>>>> python, and the other is, what we do in python. Just might be > >>> helpful > >>>>>>>>> to > >>>>>>>>> be clear about the specific enhancements & changes you are > >> making. > >>>>>>>> I think that if we adjust the query and fetch the right number of > >>>> tasks, > >>>>>>>> then we won't have to make changes to what is done in python. > >>>>>>>> > >>>>>>>> On Mon, Aug 4, 2025 at 8:01 AM Daniel Standish > >>>>>>>> daniel.stand...@astronomer.io.invalid wrote: > >>>>>>>> > >>>>>>>>> @Christos Bisias > >>>>>>>>> > >>>>>>>>> If you have a very large dag, and its tasks have been scheduled, > >>> then > >>>>>>>>> the > >>>>>>>>> > >>>>>>>>>> scheduler will keep examining the tasks for queueing, even if it > >>> has > >>>>>>>>>> reached the maximum number of active tasks for that particular > >>> dag. > >>>>>>>>>> Once > >>>>>>>>>> that fails, then it will move on to examine the scheduled tasks > >> of > >>>>>>>>>> the > >>>>>>>>>> next > >>>>>>>>>> dag or dag_run in line. > >>>>>>>>> Can you make this a little more precise? There's some protection > >>>>>>>>> against > >>>>>>>>> "starvation" i.e. dag runs recently considered should go to the > >>> back > >>>> of > >>>>>>>>> the > >>>>>>>>> line next time. > >>>>>>>>> > >>>>>>>>> Maybe you could clarify why / how that's not working / not > >> optimal > >>> / > >>>>>>>>> how > >>>>>>>>> to > >>>>>>>>> improve. > >>>>>>>>> > >>>>>>>>> If there are available slots in the pool and > >>>>>>>>> > >>>>>>>>>> the max parallelism hasn't been reached yet, then the scheduler > >>>>>>>>>> should > >>>>>>>>>> stop > >>>>>>>>>> processing a dag that has already reached its max capacity of > >>> active > >>>>>>>>>> tasks. > >>>>>>>>> If a dag run (or dag) is already at max capacity, it doesn't > >> really > >>>>>>>>> matter > >>>>>>>>> if there are slots available or parallelism isn't reached -- > >>>> shouldn't > >>>>>>>>> it > >>>>>>>>> stop anyway? > >>>>>>>>> > >>>>>>>>> In addition, the number of scheduled tasks picked for examining, > >>>> should > >>>>>>>>> be > >>>>>>>>> > >>>>>>>>>> capped at the number of max active tasks if that's lower than > >> the > >>>>>>>>>> query > >>>>>>>>>> limit. If the active limit is 10 and we already have 5 running, > >>> then > >>>>>>>>>> we > >>>>>>>>>> can > >>>>>>>>>> queue at most 5 tasks. In that case, we shouldn't examine more > >>> than > >>>>>>>>>> that. > >>>>>>>>> Don't we already stop examining at that point? I guess there's > >> two > >>>>>>>>> things > >>>>>>>>> you might be referring to. One is, which TIs come out of the db > >> and > >>>>>>>>> into > >>>>>>>>> python, and the other is, what we do in python. Just might be > >>> helpful > >>>>>>>>> to > >>>>>>>>> be clear about the specific enhancements & changes you are > >> making. > >>>>>>>>> There is already a patch with the changes mentioned above. IMO, > >>> these > >>>>>>>>>> changes should be enabled/disabled with a config flag and not by > >>>>>>>>>> default > >>>>>>>>>> because not everyone has the same needs as us. In our testing, > >>>>>>>>>> adding a > >>>>>>>>>> limit on the tasks retrieved from the db requires more > >> processing > >>> on > >>>>>>>>>> the > >>>>>>>>>> query which actually makes things worse when you have multiple > >>> small > >>>>>>>>>> dags. > >>>>>>>>> I would like to see a stronger case made for configurability. Why > >>>> make > >>>>>>>>> it > >>>>>>>>> configurable? If the performance is always better, it should not > >> be > >>>>>>>>> made > >>>>>>>>> configurable. Unless it's merely released as an opt-in > >> experimental > >>>>>>>>> feature. If it is worse in some profiles, let's be clear about > >>> that. > >>>>>>>>> I did not read anything after `Here is a simple test case that > >>> makes > >>>> the benefits of the improvements noticeable` because, it seemed rather > >>> long > >>>> winded detail about a test > >>>>>>>>> case. A higher level summary might be helpful to your audience. > >> Is > >>>>>>>>> there > >>>>>>>>> a PR with your optimization. You wrote "there is a patch" but did > >>>> not, > >>>>>>>>> unless I miss something, share it. I would take a look if you > >> share > >>>> it > >>>>>>>>> though. > >>>>>>>>> > >>>>>>>>> Thanks > >>>>>>>>> > >>>>>>>>> On Sun, Aug 3, 2025 at 5:08 PM Daniel Standish < > >>>>>>>>> daniel.stand...@astronomer.io> wrote: > >>>>>>>>> > >>>>>>>>>> Yes Ui is another part of this. > >>>>>>>>>> > >>>>>>>>>> At some point the grid and graph views completely stop making > >>> sense > >>>>>>>>>> for > >>>>>>>>>> that volume, and another type of view would be required both for > >>>>>>>>>> usability > >>>>>>>>>> and performance > >>>>>>>>>> > >>>>>>>>>> On Sun, Aug 3, 2025 at 11:04 AM Jens Scheffler > >>>>>>>>>> j_scheff...@gmx.de.invalid > >>>>>>>>>> wrote: > >>>>>>>>>> > >>>>>>>>>>> Hi, > >>>>>>>>>>> > >>>>>>>>>>> We also have a current demand to have a workflow to execute 10k > >>> to > >>>>>>>>>>> 100k > >>>>>>>>>>> tasks. Together with @AutomationDev85 we are working on a local > >>>>>>>>>>> solution > >>>>>>>>>>> because we also saw problems in the Scheduler that are not > >>> linearly > >>>>>>>>>>> scaling. And for sure not easy to be fixed. But from our > >>>>>>>>>>> investigation > >>>>>>>>>>> also there are other problems to be considered like UI will > >> also > >>>>>>>>>>> potentially have problems. > >>>>>>>>>>> > >>>>>>>>>>> I am a bit sceptic that PR 49160 completely fixes the problems > >>>>>>>>>>> mentioned > >>>>>>>>>>> here and made some comments. I do not want to stop enthusiasm > >> to > >>>> fix > >>>>>>>>>>> and > >>>>>>>>>>> improve things but the Scheduler is quite complex and changed > >>> need > >>>>>>>>>>> to > >>>>>>>>>>> be > >>>>>>>>>>> made with care. > >>>>>>>>>>> > >>>>>>>>>>> Actually I like the patch > >>>>>>> > >> > https://github.com/xBis7/airflow/compare/69ab304ffa3d9b847b7dd0ee90ee6ef100223d66..scheduler-perf-patch > >>>>>>>>>>> as it just adds some limit preventing scheduler to focus on > >> only > >>>> one > >>>>>>>>>>> run. But complexity is a bit big for a "patch" :-D > >>>>>>>>>>> > >>>>>>>>>>> I'd also propose atm the way that Jarek described and split-up > >>> the > >>>>>>>>>>> Dag > >>>>>>>>>>> into multiple parts (divide and conquer) for the moment. > >>>>>>>>>>> > >>>>>>>>>>> Otherwise if there is a concrete demand on such large Dags... > >> we > >>>>>>>>>>> maybe > >>>>>>>>>>> need rather a broader initiative if we want to ensure 10k, > >> 100k, > >>>> 1M? > >>>>>>>>>>> tasks are supported per Dag. Because depending on the magnitude > >>> we > >>>>>>>>>>> strive for different approaches are needed. > >>>>>>>>>>> > >>>>>>>>>>> Jens > >>>>>>>>>>> > >>>>>>>>>>> On 03.08.25 16:33, Daniel Standish wrote: > >>>>>>>>>>> > >>>>>>>>>>>> Definitely an area of the scheduler with some opportunity for > >>>>>>>>>>>> performance > >>>>>>>>>>>> improvement. > >>>>>>>>>>>> > >>>>>>>>>>>> I would just mention that, you should also attempt to include > >>> some > >>>>>>>>>>>> performance testing at load / scale because, window functions > >>> are > >>>>>>>>>>>> going > >>>>>>>>>>>> to > >>>>>>>>>>>> be more expensive. > >>>>>>>>>>>> > >>>>>>>>>>>> What happens when you have many dags, many historical dag > >> runs & > >>>>>>>>>>>> TIs, > >>>>>>>>>>>> lots > >>>>>>>>>>>> of stuff running concurrently. You need to be mindful of the > >>>>>>>>>>>> overall > >>>>>>>>>>>> impact of such a change, and not look only at the time spent > >> on > >>>>>>>>>>>> scheduling > >>>>>>>>>>>> this particular dag. > >>>>>>>>>>>> > >>>>>>>>>>>> I did not look at the PRs yet, maybe you've covered this, but, > >>>>>>>>>>>> it's > >>>>>>>>>>>> important. > >>>>>>>>>>>> > >>>>>>>>>>>> On Sun, Aug 3, 2025 at 5:57 AM Christos Bisias< > >>>>>>>>>>>> christos...@gmail.com> > >>>>>>>>>>>> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>>> I'm going to review the PR code and test it more thoroughly > >>>>>>>>>>>>> before > >>>>>>>>>>>>> leaving > >>>>>>>>>>>>> a comment. > >>>>>>>>>>>>> > >>>>>>>>>>>>> This is my code for reference > >>>>>>> > >> > https://github.com/xBis7/airflow/compare/69ab304ffa3d9b847b7dd0ee90ee6ef100223d66..scheduler-perf-patch > >>>>>>>>>>>>> The current version is setting a limit per dag, across all > >>>>>>>>>>>>> dag_runs. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Please correct me if I'm wrong, but the PR looks like it's > >>>>>>>>>>>>> changing > >>>>>>>>>>>>> the way > >>>>>>>>>>>>> that tasks are prioritized to avoid starvation. If that's the > >>>>>>>>>>>>> case, > >>>>>>>>>>>>> I'm not > >>>>>>>>>>>>> sure that this is the same issue. My proposal is that, if we > >>> have > >>>>>>>>>>>>> reached > >>>>>>>>>>>>> the max resources assigned to a dag, then stop processing its > >>>>>>>>>>>>> tasks > >>>>>>>>>>>>> and > >>>>>>>>>>>>> move on to the next one. I'm not changing how or which tasks > >>> are > >>>>>>>>>>>>> picked. > >>>>>>>>>>>>> > >>>>>>>>>>>>> On Sun, Aug 3, 2025 at 3:23 PM asquator<asqua...@proton.me > >>>>>>>>>>>>> .invalid> > >>>>>>>>>>>>> wrote: > >>>>>>>>>>>>> > >>>>>>>>>>>>>> Thank you for the feedback. > >>>>>>>>>>>>>> Please, describe the case with failing limit checks in the > >> PR > >>>>>>>>>>>>>> (DAG's > >>>>>>>>>>>>>> parameters and it's tasks' parameters and what fails to be > >>>>>>>>>>>>>> checked) > >>>>>>>>>>>>>> and > >>>>>>>>>>>>>> we'll try to fix it ASAP before you can test it again. Let's > >>>>>>>>>>>>>> continue > >>>>>>>>>>>>>> the > >>>>>>>>>>>>>> PR-related discussion in the PR itself. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> On Sunday, August 3rd, 2025 at 2:21 PM, Christos Bisias < > >>>>>>>>>>>>>> christos...@gmail.com> wrote: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Thank you for bringing this PR to my attention. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> I haven't studied the code but I ran a quick test on the > >>> branch > >>>>>>>>>>>>>>> and > >>>>>>>>>>>>>>> this > >>>>>>>>>>>>>>> completely ignores the limit on scheduled tasks per dag or > >>>>>>>>>>>>>>> dag_run. > >>>>>>>>>>>>>>> It > >>>>>>>>>>>>>>> grabbed 70 tasks from the first dag and then moved all 70 > >> to > >>>>>>>>>>>>>>> QUEUED > >>>>>>>>>>>>>>> without > >>>>>>>>>>>>>>> any further checks. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> This is how I tested it > >>>>>>> > >> > https://github.com/Asquator/airflow/compare/feature/pessimistic-task-fetching-with-window-function...xBis7:airflow:scheduler-window-function-testing?expand=1 > >>>>>>>>>>>>>>> On Sun, Aug 3, 2025 at 1:44 PM asquatorasqua...@proton.me > >>>>>>>>>>>>>>> .invalid > >>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Hello, > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> This is a known issue stemming from the optimistic > >>> scheduling > >>>>>>>>>>>>>>>> strategy > >>>>>>>>>>>>>>>> used in Airflow. We do address this in the above-mentioned > >>>>>>>>>>>>>>>> PR. I > >>>>>>>>>>>>>>>> want > >>>>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>> note that there are many cases where this problem may > >>>>>>>>>>>>>>>> appear—it > >>>>>>>>>>>>>>>> was > >>>>>>>>>>>>>>>> originally detected with pools, but we are striving to fix > >>> it > >>>>>>>>>>>>>>>> in > >>>>>>>>>>>>>>>> all > >>>>>>>>>>>>>>>> cases, > >>>>>>>>>>>>>>>> such as the one described here with > >> max_active_tis_per_dag, > >>> by > >>>>>>>>>>>>>>>> switching to > >>>>>>>>>>>>>>>> pessimistic scheduling with SQL window functions. While > >> the > >>>>>>>>>>>>>>>> current > >>>>>>>>>>>>>>>> strategy simply pulls the max_tis tasks and drops the ones > >>>>>>>>>>>>>>>> that > >>>>>>>>>>>>>>>> do > >>>>>>>>>>>>>>>> not > >>>>>>>>>>>>>>>> meet > >>>>>>>>>>>>>>>> the constraints, the new strategy will pull only the tasks > >>>>>>>>>>>>>>>> that > >>>>>>>>>>>>>>>> are > >>>>>>>>>>>>>>>> actually ready to be scheduled and that comply with all > >>>>>>>>>>>>>>>> concurrency > >>>>>>>>>>>>>>>> limits. > >>>>>>>>>>>>>>>> It would be very helpful for pushing this change to > >>> production > >>>>>>>>>>>>>>>> if > >>>>>>>>>>>>>>>> you > >>>>>>>>>>>>>>>> could assist us in alpha-testing it. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> See also: > >>>>>>>>>>>>>>>> https://github.com/apache/airflow/discussions/49160 > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Sent with Proton Mail secure email. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> On Sunday, August 3rd, 2025 at 12:59 PM, Elad Kalif > >>>>>>>>>>>>>>>> elad...@apache.org > >>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> i think most of your issues will be addressed by > >>>>>>>>>>>>>>>>> https://github.com/apache/airflow/pull/53492 > >>>>>>>>>>>>>>>>> The PR code can be tested with Breeze so you can set it > >> up > >>>>>>>>>>>>>>>>> and > >>>>>>>>>>>>>>>>> see > >>>>>>>>>>>>>>>>> if it > >>>>>>>>>>>>>>>>> solves the problem this will also help with confirming > >> it's > >>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>> right > >>>>>>>>>>>>>>>>> fix. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> On Sun, Aug 3, 2025 at 10:46 AM Christos Bisias > >>>>>>>>>>>>>>>>> christos...@gmail.com > >>>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Hello, > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> The scheduler is very efficient when running a large > >>> amount > >>>>>>>>>>>>>>>>>> of > >>>>>>>>>>>>>>>>>> dags > >>>>>>>>>>>>>>>>>> with up > >>>>>>>>>>>>>>>>>> to 1000 tasks each. But in our case, we have dags with > >> as > >>>>>>>>>>>>>>>>>> many > >>>>>>>>>>>>>>>>>> as > >>>>>>>>>>>>>>>>>> 10.000 > >>>>>>>>>>>>>>>>>> tasks. And in that scenario the scheduler and worker > >>>>>>>>>>>>>>>>>> throughput > >>>>>>>>>>>>>>>>>> drops > >>>>>>>>>>>>>>>>>> significantly. Even if you have 1 such large dag with > >>>>>>>>>>>>>>>>>> scheduled > >>>>>>>>>>>>>>>>>> tasks, > >>>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>> performance hit becomes noticeable. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> We did some digging and we found that the issue comes > >> from > >>>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>> scheduler's > >>>>>>>>>>>>>>>>>> _executable_task_instances_to_queued > >>>>>>>>>>>>>>>>>> < > >>>>>>> > >> > https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/jobs/scheduler_job_runner.py#L293C9-L647 > >>>>>>>>>>>>>>>>>> method. > >>>>>>>>>>>>>>>>>> In particular with the db query here > >>>>>>>>>>>>>>>>>> < > >>>>>>> > >> > https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/jobs/scheduler_job_runner.py#L364-L375 > >>>>>>>>>>>>>>>>>> and > >>>>>>>>>>>>>>>>>> examining the results here > >>>>>>>>>>>>>>>>>> < > >>>>>>> > >> > https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/jobs/scheduler_job_runner.py#L425 > >>>>>>>>>>>>>>>>>> . > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> If you have a very large dag, and its tasks have been > >>>>>>>>>>>>>>>>>> scheduled, > >>>>>>>>>>>>>>>>>> then > >>>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>> scheduler will keep examining the tasks for queueing, > >> even > >>>>>>>>>>>>>>>>>> if > >>>>>>>>>>>>>>>>>> it > >>>>>>>>>>>>>>>>>> has > >>>>>>>>>>>>>>>>>> reached the maximum number of active tasks for that > >>>>>>>>>>>>>>>>>> particular > >>>>>>>>>>>>>>>>>> dag. > >>>>>>>>>>>>>>>>>> Once > >>>>>>>>>>>>>>>>>> that fails, then it will move on to examine the > >> scheduled > >>>>>>>>>>>>>>>>>> tasks > >>>>>>>>>>>>>>>>>> of > >>>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>> next > >>>>>>>>>>>>>>>>>> dag or dag_run in line. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> This is inefficient and causes the throughput of the > >>>>>>>>>>>>>>>>>> scheduler > >>>>>>>>>>>>>>>>>> and > >>>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>> workers to drop significantly. If there are available > >>> slots > >>>>>>>>>>>>>>>>>> in > >>>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>> pool and > >>>>>>>>>>>>>>>>>> the max parallelism hasn't been reached yet, then the > >>>>>>>>>>>>>>>>>> scheduler > >>>>>>>>>>>>>>>>>> should > >>>>>>>>>>>>>>>>>> stop > >>>>>>>>>>>>>>>>>> processing a dag that has already reached its max > >> capacity > >>>>>>>>>>>>>>>>>> of > >>>>>>>>>>>>>>>>>> active > >>>>>>>>>>>>>>>>>> tasks. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> In addition, the number of scheduled tasks picked for > >>>>>>>>>>>>>>>>>> examining, > >>>>>>>>>>>>>>>>>> should be > >>>>>>>>>>>>>>>>>> capped at the number of max active tasks if that's lower > >>>>>>>>>>>>>>>>>> than > >>>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>> query > >>>>>>>>>>>>>>>>>> limit. If the active limit is 10 and we already have 5 > >>>>>>>>>>>>>>>>>> running, > >>>>>>>>>>>>>>>>>> then > >>>>>>>>>>>>>>>>>> we can > >>>>>>>>>>>>>>>>>> queue at most 5 tasks. In that case, we shouldn't > >> examine > >>>>>>>>>>>>>>>>>> more > >>>>>>>>>>>>>>>>>> than > >>>>>>>>>>>>>>>>>> that. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> There is already a patch with the changes mentioned > >> above. > >>>>>>>>>>>>>>>>>> IMO, > >>>>>>>>>>>>>>>>>> these > >>>>>>>>>>>>>>>>>> changes should be enabled/disabled with a config flag > >> and > >>>>>>>>>>>>>>>>>> not > >>>>>>>>>>>>>>>>>> by > >>>>>>>>>>>>>>>>>> default > >>>>>>>>>>>>>>>>>> because not everyone has the same needs as us. In our > >>>>>>>>>>>>>>>>>> testing, > >>>>>>>>>>>>>>>>>> adding a > >>>>>>>>>>>>>>>>>> limit on the tasks retrieved from the db requires more > >>>>>>>>>>>>>>>>>> processing > >>>>>>>>>>>>>>>>>> on > >>>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>> query which actually makes things worse when you have > >>>>>>>>>>>>>>>>>> multiple > >>>>>>>>>>>>>>>>>> small > >>>>>>>>>>>>>>>>>> dags. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Here is a simple test case that makes the benefits of > >> the > >>>>>>>>>>>>>>>>>> improvements > >>>>>>>>>>>>>>>>>> noticeable > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> - we have 3 dags with thousands of tasks each > >>>>>>>>>>>>>>>>>> - for simplicity let's have 1 dag_run per dag > >>>>>>>>>>>>>>>>>> - triggering them takes some time and due to that, the > >>> FIFO > >>>>>>>>>>>>>>>>>> order > >>>>>>>>>>>>>>>>>> of > >>>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>> tasks is very clear > >>>>>>>>>>>>>>>>>> - e.g. 1000 tasks from dag1 were scheduled first and > >> then > >>>>>>>>>>>>>>>>>> 200 > >>>>>>>>>>>>>>>>>> tasks > >>>>>>>>>>>>>>>>>> from dag2 etc. > >>>>>>>>>>>>>>>>>> - the executor has parallelism=100 and > >> slots_available=100 > >>>>>>>>>>>>>>>>>> which > >>>>>>>>>>>>>>>>>> means > >>>>>>>>>>>>>>>>>> that it can run up to 100 tasks concurrently > >>>>>>>>>>>>>>>>>> - max_active_tasks_per_dag is 4 which means that we can > >>> have > >>>>>>>>>>>>>>>>>> up > >>>>>>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>>>> 4 > >>>>>>>>>>>>>>>>>> tasks running per dag. > >>>>>>>>>>>>>>>>>> - For 3 dags, it means that we can run up to 12 tasks at > >>> the > >>>>>>>>>>>>>>>>>> same > >>>>>>>>>>>>>>>>>> time (4 tasks from each dag) > >>>>>>>>>>>>>>>>>> - max tis per query are set to 32, meaning that we can > >>>>>>>>>>>>>>>>>> examine > >>>>>>>>>>>>>>>>>> up > >>>>>>>>>>>>>>>>>> to 32 > >>>>>>>>>>>>>>>>>> scheduled tasks if there are available pool slots > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> If we were to run the scheduler loop repeatedly until it > >>>>>>>>>>>>>>>>>> queues > >>>>>>>>>>>>>>>>>> 12 > >>>>>>>>>>>>>>>>>> tasks > >>>>>>>>>>>>>>>>>> and test the part that examines the scheduled tasks and > >>>>>>>>>>>>>>>>>> queues > >>>>>>>>>>>>>>>>>> them, > >>>>>>>>>>>>>>>>>> then > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> - with the query limit > >>>>>>>>>>>>>>>>>> - 1 iteration, total time 0.05 > >>>>>>>>>>>>>>>>>> - During the iteration > >>>>>>>>>>>>>>>>>> - we have parallelism 100, available slots 100 and query > >>>>>>>>>>>>>>>>>> limit > >>>>>>>>>>>>>>>>>> 32 > >>>>>>>>>>>>>>>>>> which means that it will examine up to 32 scheduled > >> tasks > >>>>>>>>>>>>>>>>>> - it can queue up to 100 tasks > >>>>>>>>>>>>>>>>>> - examines 12 tasks (instead of 32) > >>>>>>>>>>>>>>>>>> - 4 tasks from dag1, reached max for the dag > >>>>>>>>>>>>>>>>>> - 4 tasks from dag2, reached max for the dag > >>>>>>>>>>>>>>>>>> - and 4 tasks from dag3, reached max for the dag > >>>>>>>>>>>>>>>>>> - queues 4 from dag1, reaches max for the dag and moves > >> on > >>>>>>>>>>>>>>>>>> - queues 4 from dag2, reaches max for the dag and moves > >> on > >>>>>>>>>>>>>>>>>> - queues 4 from dag3, reaches max for the dag and moves > >> on > >>>>>>>>>>>>>>>>>> - stops queueing because we have reached the maximum per > >>>>>>>>>>>>>>>>>> dag, > >>>>>>>>>>>>>>>>>> although there are slots for more tasks > >>>>>>>>>>>>>>>>>> - iteration finishes > >>>>>>>>>>>>>>>>>> - without > >>>>>>>>>>>>>>>>>> - 3 iterations, total time 0.29 > >>>>>>>>>>>>>>>>>> - During iteration 1 > >>>>>>>>>>>>>>>>>> - Examines 32 tasks, all from dag1 (due to FIFO) > >>>>>>>>>>>>>>>>>> - queues 4 from dag1 and tries to queue the other 28 but > >>>>>>>>>>>>>>>>>> fails > >>>>>>>>>>>>>>>>>> - During iteration 2 > >>>>>>>>>>>>>>>>>> - examines the next 32 tasks from dag1 > >>>>>>>>>>>>>>>>>> - it can't queue any of them because it has reached the > >>> max > >>>>>>>>>>>>>>>>>> for > >>>>>>>>>>>>>>>>>> dag1, since the previous 4 are still running > >>>>>>>>>>>>>>>>>> - examines 32 tasks from dag2 > >>>>>>>>>>>>>>>>>> - queues 4 from dag2 and tries to queue the other 28 but > >>>>>>>>>>>>>>>>>> fails > >>>>>>>>>>>>>>>>>> - During iteration 3 > >>>>>>>>>>>>>>>>>> - examines the next 32 tasks from dag1, same tasks that > >>> were > >>>>>>>>>>>>>>>>>> examined in iteration 2 > >>>>>>>>>>>>>>>>>> - it can't queue any of them because it has reached the > >>> max > >>>>>>>>>>>>>>>>>> for > >>>>>>>>>>>>>>>>>> dag1 and the first 4 are still running > >>>>>>>>>>>>>>>>>> - examines 32 tasks from dag2 , can't queue any of them > >>>>>>>>>>>>>>>>>> because > >>>>>>>>>>>>>>>>>> it has reached max for dag2 as well > >>>>>>>>>>>>>>>>>> - examines 32 tasks from dag3 > >>>>>>>>>>>>>>>>>> - queues 4 from dag3 and tries to queue the other 28 but > >>>>>>>>>>>>>>>>>> fails > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> I used very low values for all the configs so that I can > >>>>>>>>>>>>>>>>>> make > >>>>>>>>>>>>>>>>>> the > >>>>>>>>>>>>>>>>>> point > >>>>>>>>>>>>>>>>>> clear and easy to understand. If we increase them, then > >>> this > >>>>>>>>>>>>>>>>>> patch > >>>>>>>>>>>>>>>>>> also > >>>>>>>>>>>>>>>>>> makes the task selection more fair and the resource > >>>>>>>>>>>>>>>>>> distribution > >>>>>>>>>>>>>>>>>> more > >>>>>>>>>>>>>>>>>> even. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> I would appreciate it if anyone familiar with the > >>>>>>>>>>>>>>>>>> scheduler's > >>>>>>>>>>>>>>>>>> code > >>>>>>>>>>>>>>>>>> can > >>>>>>>>>>>>>>>>>> confirm this and also provide any feedback. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Additionally, I have one question regarding the query > >>> limit. > >>>>>>>>>>>>>>>>>> Should it > >>>>>>>>>>>>>>>>>> be > >>>>>>>>>>>>>>>>>> per dag_run or per dag? I've noticed that > >>>>>>>>>>>>>>>>>> max_active_tasks_per_dag > >>>>>>>>>>>>>>>>>> has > >>>>>>>>>>>>>>>>>> been changed to provide a value per dag_run but the docs > >>>>>>>>>>>>>>>>>> haven't > >>>>>>>>>>>>>>>>>> been > >>>>>>>>>>>>>>>>>> updated. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Thank you! > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Regards, > >>>>>>>>>>>>>>>>>> Christos Bisias > >>>>>>> > >> --------------------------------------------------------------------- > >>>>>>>>>>>>>>>> To unsubscribe, e-mail:dev-unsubscr...@airflow.apache.org > >>>>>>>>>>>>>>>> For additional commands, > >> e-mail:dev-h...@airflow.apache.org > >>>>>>>>> > >>> --------------------------------------------------------------------- > >>>>>>>>>>>>>> To unsubscribe, e-mail:dev-unsubscr...@airflow.apache.org > >>>>>>>>>>>>>> For additional commands, e-mail:dev-h...@airflow.apache.org > >>>>> --------------------------------------------------------------------- > >>>>> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > >>>>> For additional commands, e-mail: dev-h...@airflow.apache.org > >>>>> > >>>> > >>>> --------------------------------------------------------------------- > >>>> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > >>>> For additional commands, e-mail: dev-h...@airflow.apache.org > >>>> > >>>> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > For additional commands, e-mail: dev-h...@airflow.apache.org > >
