Hello Pavan Kumar, I really like your proposition as this would have facilitated my implementation of the MSGraphSensor, which is actually implemented as a deferrable using a triggerer.
You can check here how I did it: https://github.com/apache/airflow/blob/main/airflow/providers/microsoft/azure/sensors/msgraph.py The idea was the same as in your proposition to not block workers when using a sensor by implementing it as a deferrable. Kind regards, David -----Original Message----- From: Pavankumar Gopidesu <gopidesupa...@gmail.com> Sent: Thursday, September 26, 2024 11:01 AM To: dev@airflow.apache.org Subject: Re: [DISCUSS] Sensor Improvements With Tirggers EXTERNAL MAIL: Indien je de afzender van deze e-mail niet kent en deze niet vertrouwt, klik niet op een link of open geen bijlages. Bij twijfel, stuur deze e-mail als bijlage naar ab...@infrabel.be<mailto:ab...@infrabel.be>. Hello Everyone, After discussion on the slack channel, I have decided to retain the current synchronous sensor to offer users the choice to execute sensors on workers. I have revised the proposal to introduce a new asynchronous version of the sensor, eliminating the need for workers to run sensors. Additional details have been included below. Your review and feedback are greatly appreciated. I am eager to hear from you. https://cwiki.apache.org/confluence/display/AIRFLOW/AIP+87%3A+Run+Sensors+in+Triggerer+with+AsyncBaseSensor Regards, Pavan Kumar On Wed, Aug 14, 2024 at 11:48 PM Pavankumar Gopidesu < gopidesupa...@gmail.com> wrote: > Hi Wei Lee, > > Thanks for the review, While I was working on the POC , I had a bit of > confusion about how to use the logic present inside the sensor execute > method. for both with and without triggerer flow. > so to make it work for both flows, I have moved out to execute logic > with two methods. > > I appreciate the idea of a simple helper function, I'm eager to hear > any suggestions you might have or anyone :). > > Regards, > Pavan Kumar > > On Wed, Aug 14, 2024 at 12:16 PM Wei Lee <weilee...@gmail.com> wrote: > >> Thank you for bringing this up! I have added some comments to the >> document. I'm unsure if we really want or need to implement more >> complex logic for this. What I have in mind is simply adding helper >> functions to InternalSensorTrigger and continuing to use the run method in >> BaseTrigger. >> The main purpose of those methods is to allow operator authors to >> change the sensors to leverage this start/end from trigger more easily. >> >> Best, >> Wei >> >> > On Aug 14, 2024, at 5:19 PM, Pavankumar Gopidesu < >> gopidesupa...@gmail.com> wrote: >> > >> > Hi Jarek, >> > >> > Thanks for the questions :) , >> > >> > I completely agree with you , from 2.10 we have the >> > start_from_trigger parameter, which , when set , allows a task to >> > be executed directly from the triggerrer without worker >> > involvement. I believe that for any sensor to be executed in the >> > triggerer, a corresponding trigger implementation must be in place. >> > Good thing is , most of the required trigger implementations are >> > already available in current airflow providers. >> > >> > Additionally, i agree that there is no real difference between a >> deferrable >> > sensor and a deferrable operator, >> > generally , users utilize the BaseSensorOperator to create custom >> sensors, >> > However if they want to run >> > these sensors in triggerer, they must implement the triggerer's run >> method >> > by extending the base trigger class. >> > >> > The key difference with this proposal is the introduction of a >> > common trigger implementation (referred to in this POC as >> > InternalSensorTrigger )[1]. This would allow users to use the new >> > feature start_from_trigger param with their custom sensor and >> > eliminates the need for individual trigger implementations for each >> > custom sensor when they create. >> > >> > Alternatively, we could leave the start_from_trigger parameter in >> > BaseSensorOperator with a default value as false and let users >> > decide whether they want to run the sensor >> in >> > triggerrer. If they choose to do so, they can simply set the >> > parameter to true and triggerrer uses the InternalSensorTrigger >> > class. >> > >> > Apologies if my answer is too confusing :) >> > >> > [1]: >> > >> https://git/ >> hub.com%2Fapache%2Fairflow%2Fpull%2F41355%2Ffiles%23diff-7486f32e385d >> 7ad0376cccda08d80e54939aa901a24616d11fb1f5cba6af7f83R144&data=05%7C02 >> %7Cdavid.blain%40infrabel.be%7Ce13a7cc2a5fd4807701e08dcde09ebc7%7Cb82 >> bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C638629381398366280%7CUnknown% >> 7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJ >> XVCI6Mn0%3D%7C0%7C%7C%7C&sdata=3wwm43IOv4KUwciQatVOM5CKRmwYn4HBsiBZEM >> p2jvI%3D&reserved=0 >> > >> > Regards, >> > Pavan Kumar >> > >> > On Wed, Aug 14, 2024 at 12:29 AM Jarek Potiuk <ja...@potiuk.com> wrote: >> > >> >> How does it differ from the upcoming 2.10 feature? >> >> >> >> * Deferrable operators can now execute directly from the triggerer >> without >> >> needing to go through the worker. This is especially efficient for >> certain >> >> operators, like sensors, and can help teams save both time and money. >> >> >> >> As of 2.10 - Sensors already can run mostly in Triggerrer and >> >> basically there is no big difference any more between deferrable >> >> sensor and deferrable operator. >> >> >> >> >> >> On Mon, Aug 12, 2024 at 3:59 AM Kaxil Naik <kaxiln...@gmail.com> >> wrote: >> >> >> >>> Thanks for putting this together, I will take a look this week. >> >>> >> >>> On Fri, 9 Aug 2024 at 13:12, Pavankumar Gopidesu < >> >> gopidesupa...@gmail.com> >> >>> wrote: >> >>> >> >>>> Hi All, >> >>>> >> >>>> I have created a draft document for Sensor Improvements using >> triggers. >> >>>> >> >>>> Details: >> >>>> >> >>>> >> >>> >> >> >> https://doc/ >> s.google.com%2Fdocument%2Fd%2F1Kb_wL-T1DHkOpmR_QNa3O5p_2hMTLzM-sb_HzQ >> 0PYCo%2Fedit%3Fusp%3Dsharing&data=05%7C02%7Cdavid.blain%40infrabel.be >> %7Ce13a7cc2a5fd4807701e08dcde09ebc7%7Cb82bc314ab8e4d6fb18946f02e1f27f >> 2%7C0%7C0%7C638629381398384359%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLj >> AwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sda >> ta=ve%2FbzO8Vqe8K5yKdMVVsnAmD6zPl0gh7WyBrO6IeKjw%3D&reserved=0 >> >>>> >> >>>> POC: >> >>>> >> >>>> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%25 >> >>>> 2Fgithub.com%2Fapache%2Fairflow%2Fpull%2F41355&data=05%7C02%7Cda >> >>>> vid.blain%40infrabel.be%7Ce13a7cc2a5fd4807701e08dcde09ebc7%7Cb82 >> >>>> bc314ab8e4d6fb18946f02e1f27f2%7C0%7C0%7C638629381398402033%7CUnk >> >>>> nown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6I >> >>>> k1haWwiLCJXVCI6Mn0%3D%7C0%7C%7C%7C&sdata=sYLKcAYQsysHC2%2FPsyV9m >> >>>> Ze%2BWu%2BL2HyJ0SlPulVIvEg%3D&reserved=0 >> >>>> >> >>>> This is my first draft post, apologies if any mistakes in the >> >> document. I >> >>>> would greatly appreciate your insights and suggestions on draft. >> >>>> >> >>>> Thank you very much for your time. >> >>>> >> >>>> Regards, >> >>>> Pavan >> >>>> >> >>> >> >> >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org >> For additional commands, e-mail: dev-h...@airflow.apache.org >> >>
