Great to hear from you, Mani. I am interested in collaborating with you on this one. Seems like a promising initial demo, yet to catch up on the specifics.
Thanks & Regards, Amogh Desai On Tue, Oct 22, 2024 at 8:56 PM Jarek Potiuk <ja...@potiuk.com> wrote: > > Would it be possible to develop this out-of-tree for the time being? > > Oh absolutely. I definitely do not want to add "more" on the Airflow 3 > band-wagon. > I am even quite sure I will not be the one implementing it, nor anyone > involved in Airflow 3, It's more of a conceptual discussion - and an > attempt to make it as an interesting idea someone could take a closer look > at. > > Manikandan, > > Great to hear from you, it's fantastic to hear from maintainers of other > projects. But for your information - we are now in the process of > completely rewriting part of Airflow 3, which means that we are deep down > and busy in various parts - and as Ash mentioned, we do not want to > "complicate" things but adding more **just now**. > > And I really love the idea of starting something outside in parallel. There > are a number of people here who are not that deeply involved in Airflow 3 > and they could take on the discussion - and maybe even work with Manikandan > directly on such an executor - somewhere on the side - just to explore more > of what Manikandan just started. > > I'd be curious to see the result of such a work where someone with deeper > Airflow understanding (but not necessarily involved in Airflow 3 work) > could make some self-guided experiment and look at what could be achieved > even by developing an executor POC that would work for Airflow 2 ? > > Maybe someone ? > > > J. > > On Tue, Oct 22, 2024 at 1:22 PM Ash Berlin-Taylor <a...@apache.org> wrote: > > > This looks like it has some really cool features. > > > > > *Tl;DR; I would love to start discussion about creating (for Airflow > 3.x > > - > > it does not have to be Airflow 3.0) a new community executor based on > > YuniKorn* > > > > I think this caveat to me is the main point, as long as it’s not in 3.0 > > (and ideally for me not even in repo for the next few months) for two > > reasons: > > > > 1. AIP 72 is going to change the Executor interface somewhat, and we > don’t > > know the exact details of how yet, so having to not worry about another > > executor to fix up and ensure works would be good to now slow down > > development of 3.0; and > > 2. I’m slightly nervous about the extra support load of a new executor at > > this time. It’s probably not all that much on Airflow side of things, but > > this is just an unknown risk to me right now. > > > > Would it be possible to develop this out-of-tree for the time being? > > > > Thanks, > > Ash > > > > > On 18 Oct 2024, at 08:41, Shubham Raj <shubhamraj....@gmail.com> > wrote: > > > > > > Hi Jarek, Amogh, and everyone, > > > > > > I wanted to share my thoughts on the proposal to integrate YuniKorn, > and > > > I'm definitely on board with it! As others mentioned, adding YuniKorn > as > > > another executor could really enhance our scheduling capabilities, > > > especially for the more complex scenarios that Celery and Kubernetes > > > executors struggle with. > > > > > > One of the standout features of YuniKorn is its hierarchical queueing > and > > > resource quota management, which is fantastic for handling multi-tenant > > > environments. This will help us keep resource-heavy Airflow tasks from > > > bogging down shared clusters and ensure that resources are allocated > > fairly > > > across different services. Now, regarding gang scheduling as per my > > > understanding, I think it’s interesting to note that Airflow operates > on > > a > > > sequential model because of its DAG structure, tasks must wait for > their > > > dependencies to finish before they can run. This might seem at odds > with > > > the idea of gang scheduling, but there are definitely scenarios where > it > > > could be useful. For instance, if we have several independent data > > > processing tasks that need to share resources, gang scheduling could > help > > > us optimize resource use and reduce latency by allowing those tasks to > be > > > scheduled at the same time. > > > > > > Overall, I believe that integrating these YuniKorn features could > really > > > boost Airflow’s capabilities, especially for complex workflows or > atleast > > > in resource-constrained environments. Looking forward to hearing > > everyone’s > > > thoughts! > > > > > > Thanks & Regards, > > > Shubham > > > > > > On Fri, Oct 18, 2024 at 10:19 AM Amogh Desai <amoghdesai....@gmail.com > > > > > wrote: > > > > > >> Hi Jarek, Everyone, > > >> > > >> Thanks for starting this discussion! > > >> I agree with everyone so far that this will be more of an additional > > >> executor rather than a replacement for > > >> anything we currently have. > > >> > > >> I had submitted a talk that was mainly trying to explain about how we > > can > > >> leverage some features of Yunikorn > > >> such as priority scheduling, multi tenancy (per deployment in terms of > > >> resources) and preemption. > > >> Not all of these features are fully implemented / integrated yet, but > I > > had > > >> planned to explore them and share my > > >> findings if my session got selected. I was trying to explore mainly > > around > > >> integration with hierarchical queues > > >> and resource quotas. > > >> > > >> To set a tone, we already have some examples running in our cluster > > >> deployments. We use Airflow in Kubernetes > > >> with theK8sExecutor, where we share space to run Airflow jobs and > other > > >> data engineering workloads. > > >> > > >> Via the integration with Yunikorn, we are able to achieve a few > things: > > >> 1. Priority Scheduling > > >> We’ve set priorities for different services running in our cluster. > For > > >> example, let's say, both Airflow jobs and Spark jobs > > >> run in a cluster. We prioritize Spark Drivers equally with Airflow > > workers, > > >> which ensures that Airflow workers get more > > >> priority over Spark Executors. This way, Airflow schedules won’t be > > missed, > > >> and it doesn’t negatively impact > > >> spark jobs because they can still run with fewer executors. > > >> > > >> 2. Resource Quotas: We also link Airflow namespaces (where the workers > > and > > >> the core services run) with resource quotas > > >> to prevent a malformed or a resource heavy Airflow task from taking > over > > >> the entire K8s cluster with a faulty DAG. This is > > >> important since we have both Airflow and other data engineering > > workloads > > >> running together. > > >> > > >> I had a chat with some folks from the Yunikron team and apart from > > this, I > > >> think a few other features of Yunikorn such as > > >> gang scheduling, preemption, etc. could be beneficial to Airflow: > > >> 1. Gang Scheduling > > >> Airflow DAGs generally have a pattern where tasks are dependent on > each > > >> other - so lets say task1 -> task2 -> task3 ... > > >> So even though there are so many tasks, there's just one DAG process. > So > > >> this could benefit from gang scheduling. > > >> If the whole task set can be considered as a single app and benefit > from > > >> gang scheduling. For those of you who > > >> aren't too familiar with gang scheduling, gang scheduling can be > > thought of > > >> as waiting for all your friends to join you > > >> for a game rather than waiting for them one by one (easiest example I > > could > > >> think of). > > >> > > >> 2. Preemption > > >> We can think of different angles to preemption based on the use cases. > > Like > > >> preempting the entire app instead of using a > > >> per request preemption OR not preempting a task if it has a dependent > > task > > >> because preemption is expensive. > > >> > > >> Overall, I believe the community would benefit from this integration, > > and I > > >> think the Yunikorn team will support it as well. > > >> > > >> Thanks & Regards, > > >> Amogh Desai > > >> > > >> > > >> On Thu, Oct 17, 2024 at 11:06 PM Jarek Potiuk <ja...@potiuk.com> > wrote: > > >> > > >>>> As Jens said "K8sExecutor++". > > >>>> Just to be precise, I don't believe that this can be a replacement > for > > >>> Celery Executor (at least at first glance). > > >>> > > >>> Yes. Fully agree. My bad framing from the initial message. > > >>> > > >>>> I also believe that for this to be effective, this will need some > > >>> dedicated work including additional information about the task. > > >>> > > >>> Oh absolutely. For me it's more of a (when we agree it's a good > > >> direction) > > >>> - let's keep it as something that **might** eventually happen and not > > in > > >>> 3.0. This is really "if we hear more cases that it might solve, let's > > see > > >>> if we need any changes in current Airflow 3 work to enable it or make > > it > > >>> easier." kinda thing. More like making a mental space for this to > > happen > > >>> when we are discussing other things. Last thing I want to do is to > add > > >> more > > >>> substantial work for our 3.0 efforts. > > >>> > > >>>> I am very curious for Amogh to chime in on this :) > > >>> > > >>> Knowing that there was a talk in-preparation, me too :D > > >>> > > >>>> The biggest decision is whether this is a community managed executor > > or > > >>> if we can find stakeholders to create this outside of Airflow (those > > >>> stakeholders could be some of us from the community). > > >>> > > >>> That's an excellent point Niko. Yes. It could be done outside. It > could > > >> be > > >>> done by Yunikorn people (unlikely - they likely have more work than > > they > > >>> can handle) or one of the stakeholders (at least initially) - and > > >> published > > >>> and released and battle-tested by them and eventually contributed to > > the > > >>> community. This is I think a very good pattern for Open Source, where > > >>> commercial users might reap the benefits of their investment as > "first > > >>> movers" while paying the price for "teething problems" - but then > > later > > >>> contributing back to the community. A company starting with C and > > ending > > >>> with a comes to my mind immediately as an obvious candidate if you > ask > > >> me. > > >>> > > >>> J. > > >>> > > >>> > > >>> On Thu, Oct 17, 2024 at 7:19 PM Oliveira, Niko > > >> <oniko...@amazon.com.invalid > > >>>> > > >>> wrote: > > >>> > > >>>> I love the idea. Generally it is quite easy now to add new executors > > >> and > > >>>> there is no harm in having more options. I don't think we need to > > >> justify > > >>>> it as a replacement of anything honestly. > > >>>> > > >>>> The biggest decision is whether this is a community managed executor > > or > > >>> if > > >>>> we can find stakeholders to create this outside of Airflow (those > > >>>> stakeholders could be some of us from the community). > > >>>> > > >>>> Cheers, > > >>>> Niko > > >>>> > > >>>> ________________________________ > > >>>> From: Vikram Koka <vik...@astronomer.io.INVALID> > > >>>> Sent: Wednesday, October 16, 2024 4:13:27 PM > > >>>> To: dev@airflow.apache.org > > >>>> Subject: RE: [EXT] [DISCUSS] Create community "Apache YuniKorn" > > >> executor > > >>> ? > > >>>> > > >>>> CAUTION: This email originated from outside of the organization. Do > > not > > >>>> click links or open attachments unless you can confirm the sender > and > > >>> know > > >>>> the content is safe. > > >>>> > > >>>> > > >>>> > > >>>> AVERTISSEMENT: Ce courrier électronique provient d’un expéditeur > > >> externe. > > >>>> Ne cliquez sur aucun lien et n’ouvrez aucune pièce jointe si vous ne > > >>> pouvez > > >>>> pas confirmer l’identité de l’expéditeur et si vous n’êtes pas > certain > > >>> que > > >>>> le contenu ne présente aucun risque. > > >>>> > > >>>> > > >>>> > > >>>> I am supportive of this in the long term (i.e. post-3.0) as an > > >> additional > > >>>> Executor similar to the Kubernetes Executor. > > >>>> As Jens said "K8sExecutor++". > > >>>> > > >>>> Just to be precise, I don't believe that this can be a replacement > for > > >>>> Celery Executor (at least at first glance). > > >>>> > > >>>> I also believe that for this to be effective, this will need some > > >>> dedicated > > >>>> work including additional information about the task. > > >>>> I am very curious for Amogh to chime in on this :) > > >>>> > > >>>> > > >>>> > > >>>> On Tue, Oct 15, 2024 at 1:58 PM Jarek Potiuk <ja...@potiuk.com> > > wrote: > > >>>> > > >>>>> Yeah - it was a bit of dramatisation when I recalled the Celery > > >>>>> "replacement" ;) . And yes it's not really "alternative" to Celery, > > >>>> Celery > > >>>>> is there to stay for short tasks. > > >>>>> > > >>>>> Almost by definition it is meant to run more heavy tasks (for > example > > >>>> batch > > >>>>> inference) where multiple tasks running in parallel share the same > > >> GPU > > >>>> for > > >>>>> example - because that's what we want to optimize. > > >>>>> > > >>>>> And yes - it provides features that K8S executor does not - gang > > >>>>> scheduling, and sophisticated preemption logic. > > >>>>> > > >>>>> J. > > >>>>> > > >>>>> On Tue, Oct 15, 2024 at 8:40 PM Jens Scheffler > > >>>> <j_scheff...@gmx.de.invalid > > >>>>>> > > >>>>> wrote: > > >>>>> > > >>>>>> Hi Jarek, > > >>>>>> > > >>>>>> scanning but not reading the full docs I understand that YuniKorn > > >> is > > >>> a > > >>>>>> specialized, more advanced K8sExecutor - all workload also runs in > > >>>> PODs? > > >>>>>> > > >>>>>> If this is the right understanding then it might be a > K8sExecutor++ > > >>> or > > >>>>>> could replace this... but Celery is playing very good usually if > > >> you > > >>>>>> have very small and high-frequency tasks. Don't know if I > > >>> mis-interpret > > >>>>>> the docs... but would it be scaling down to very small > > >>>>>> PythonOperator/@task decorated tasks with a few lines of code as > > >>> well? > > >>>>>> > > >>>>>> Jens > > >>>>>> > > >>>>>> On 15.10.24 12:55, Jarek Potiuk wrote: > > >>>>>>> Hello here, > > >>>>>>> > > >>>>>>> *Tl;DR; I would love to start discussion about creating (for > > >>> Airflow > > >>>>> 3.x > > >>>>>> - > > >>>>>>> it does not have to be Airflow 3.0) a new community executor > > >> based > > >>> on > > >>>>>>> YuniKorn* > > >>>>>>> > > >>>>>>> You might remember my point "replacing Celery Executor" when I > > >>> raised > > >>>>> the > > >>>>>>> Airflow 3 question. I never actually "meant" to replace (and > > >>> remove) > > >>>>>> Celery > > >>>>>>> Executor, but I was more in a quest to see if we have a viable > > >>>>>> alternative. > > >>>>>>> > > >>>>>>> And I think we have one with Apache Yunicorn. > > >>>>>> https://yunikorn.apache.org/ > > >>>>>>> > > >>>>>>> While it is not a direct replacement (so I'd say it should be an > > >>>>>> additional > > >>>>>>> executor), I think Yunikorn can provide us with a number of > > >>> features > > >>>>> that > > >>>>>>> we currently cannot give to our users and from the discussions I > > >>> had > > >>>>> and > > >>>>>>> talk I saw at the Community Over Code in Denver, I believe it > > >> might > > >>>> be > > >>>>>>> something that might make Airflow also more capable especially in > > >>> the > > >>>>>>> "optimization wars" context that I wrote about in > > >>>>>>> https://lists.apache.org/thread/1mp6jcfvx67zd3jjt9w2hlj0c5ysbh8r > > >>>>>>> > > >>>>>>> It seems like quite a good fit for the "Inference" use case that > > >> we > > >>>>> want > > >>>>>> to > > >>>>>>> support for Airflow 3. > > >>>>>>> > > >>>>>>> At the Community Over Code I attended a talk (and had quite nice > > >>>>>> follow-up > > >>>>>>> discussion) from Apple engineers - named: "Maximizing GPU > > >>>> Utilization: > > >>>>>>> Apache YuniKorn Preemption" and had a very long discussion with > > >>>>> Cloudera > > >>>>>>> people who are using YuniKorn for years to optimize their > > >>> workloads. > > >>>>>>> > > >>>>>>> The presentation is not recorded, but I will try to get slides > > >> and > > >>>> send > > >>>>>> it > > >>>>>>> your way. > > >>>>>>> > > >>>>>>> I think we should take a close look at it - because it seems to > > >>>> save a > > >>>>>> ton > > >>>>>>> of implementation effort for the Apple team running Batch > > >> inference > > >>>> for > > >>>>>>> their multi-tenant internal environment - which I think is > > >>> precisely > > >>>>> what > > >>>>>>> you want to do. > > >>>>>>> > > >>>>>>> YuniKorn (https://yunikorn.apache.org/) is an "app-aware" > > >>> scheduler > > >>>>> that > > >>>>>>> has a number of queue / capacity management models, policies that > > >>>> allow > > >>>>>>> controlling various applications - competing for GPUs from a > > >> common > > >>>>> pool. > > >>>>>>> > > >>>>>>> They mention things like: > > >>>>>>> > > >>>>>>> * Gang Scheduling / with gang scheduling preemption where there > > >> are > > >>>>>>> workloads requiring minimum number of workers > > >>>>>>> * Supports Latency sensitive workloads > > >>>>>>> * Resource quota management - things like priorities of execution > > >>>>>>> * YuniKorn preemption - with guaranteed capacity and preemption > > >>> when > > >>>>>> needed > > >>>>>>> - which improves the utilisation > > >>>>>>> * Preemption that minimizes preemption cost (Pod level preemption > > >>>>> rather > > >>>>>>> than application level preemption) - very customizable preemption > > >>>> with > > >>>>>>> opt-in/opt-out, queues, resource weights, fencing, supporting > > >>>> fifo/lifo > > >>>>>>> sorting etc. > > >>>>>>> * Runs in Cloud and on-premise > > >>>>>>> > > >>>>>>> The talk described quite a few scenarios of > > >> preemption/utilization/ > > >>>>>>> guaranteed resources etc. They also outlined on what YuniKorn > > >> works > > >>>> on > > >>>>>> new > > >>>>>>> features (intra-queue preemption etc.) and what future things can > > >>> be > > >>>>>> done. > > >>>>>>> > > >>>>>>> > > >>>>>>> Coincidentally - Amogh Desai with a friend submitted a talk for > > >>>> Airflow > > >>>>>>> Summit: > > >>>>>>> > > >>>>>>> "A Step Towards Multi-Tenant Airflow Using Apache YuniKorn" > > >>>>>>> > > >>>>>>> Which did not make it to the Summit (other talk of Amogh did) - > > >>> but I > > >>>>>> think > > >>>>>>> back then we have not realized about the potential of utilising > > >>>>> YuniKorn > > >>>>>> to > > >>>>>>> optimize workflows managed by Airflow. > > >>>>>>> > > >>>>>>> But we seem to have people in the community who know more about > > >>>>> YuniKorn > > >>>>>> <> > > >>>>>>> Airflow relation (Amogh :) ) and could probably comment and add > > >>> some > > >>>>>> "from > > >>>>>>> the trenches" experience to the discussion. > > >>>>>>> > > >>>>>>> Here is the description of the talk that Amoghs submitted: > > >>>>>>> > > >>>>>>> Multi-tenant Airflow is hard and there have been novel approaches > > >>> in > > >>>>> the > > >>>>>>> recent past to converge this gap. A key obstacle in multi-tenant > > >>>>> Airflow > > >>>>>> is > > >>>>>>> the management of cluster resources. This is crucial to avoid one > > >>>>>> malformed > > >>>>>>> workload from hijacking an entire cluster. It is also vital to > > >>>> restrict > > >>>>>>> users and groups from monopolizing resources in a shared cluster > > >>>> using > > >>>>>>> their workloads. > > >>>>>>> > > >>>>>>> To tackle these challenges, we turn to Apache YuniKorn, a K8s > > >>>> scheduler > > >>>>>>> catering all kinds of workloads. We leverage YuniKorn’s > > >>> hierarchical > > >>>>>> queues > > >>>>>>> in conjunction with resource quotas to establish multi-tenancy at > > >>>> both > > >>>>>> the > > >>>>>>> shared namespace level and within individual namespaces where > > >>> Airflow > > >>>>> is > > >>>>>>> deployed. > > >>>>>>> > > >>>>>>> YuniKorn also introduces Airflow to a new dimension of > > >> preemption. > > >>>> Now, > > >>>>>>> Airflow workers can preempt resources from lower-priority jobs, > > >>>>> ensuring > > >>>>>>> critical schedules in our data pipelines are met without > > >>> compromise. > > >>>>>>> > > >>>>>>> Join us for a discussion on integrating Airflow with YuniKorn, > > >>>>> unraveling > > >>>>>>> solutions to these multi-tenancy challenges. We will also share > > >> our > > >>>>> past > > >>>>>>> experiences while scaling Airflow and the steps we have taken to > > >>>> handle > > >>>>>>> real world production challenges in equitable multi-tenant K8s > > >>>>> clusters. > > >>>>>>> > > >>>>>>> I would love to hear what you think about it. I know we are deep > > >>> into > > >>>>>>> Airflow 3.0 implementation - but that one can be > > >>>> discussed/implemented > > >>>>>>> independently and maybe it's a good idea to start doing it > > >> earlier > > >>>> than > > >>>>>>> later if we see that it has good potential. > > >>>>>>> > > >>>>>>> J. > > >>>>>>> > > >>>>>> > > >>>>>> > > >> --------------------------------------------------------------------- > > >>>>>> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > >>>>>> For additional commands, e-mail: dev-h...@airflow.apache.org > > >>>>>> > > >>>>>> > > >>>>> > > >>>> > > >>> > > >> > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > For additional commands, e-mail: dev-h...@airflow.apache.org > > > > >
