Hello Everyone, After discussion on the slack channel, I have decided to retain the current synchronous sensor to offer users the choice to execute sensors on workers. I have revised the proposal to introduce a new asynchronous version of the sensor, eliminating the need for workers to run sensors. Additional details have been included below. Your review and feedback are greatly appreciated. I am eager to hear from you.
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP+87%3A+Run+Sensors+in+Triggerer+with+AsyncBaseSensor Regards, Pavan Kumar On Wed, Aug 14, 2024 at 11:48 PM Pavankumar Gopidesu < gopidesupa...@gmail.com> wrote: > Hi Wei Lee, > > Thanks for the review, While I was working on the POC , I had a bit of > confusion about how to use the logic present inside the sensor execute > method. for both with and without triggerer flow. > so to make it work for both flows, I have moved out to execute logic with > two methods. > > I appreciate the idea of a simple helper function, I'm eager to hear any > suggestions you might have or anyone :). > > Regards, > Pavan Kumar > > On Wed, Aug 14, 2024 at 12:16 PM Wei Lee <weilee...@gmail.com> wrote: > >> Thank you for bringing this up! I have added some comments to the >> document. I'm unsure if we really want or need to implement more complex >> logic for this. What I have in mind is simply adding helper functions to >> InternalSensorTrigger and continuing to use the run method in BaseTrigger. >> The main purpose of those methods is to allow operator authors to change >> the sensors to leverage this start/end from trigger more easily. >> >> Best, >> Wei >> >> > On Aug 14, 2024, at 5:19 PM, Pavankumar Gopidesu < >> gopidesupa...@gmail.com> wrote: >> > >> > Hi Jarek, >> > >> > Thanks for the questions :) , >> > >> > I completely agree with you , from 2.10 we have the start_from_trigger >> > parameter, which , when set , >> > allows a task to be executed directly from the triggerrer without worker >> > involvement. I believe that for >> > any sensor to be executed in the triggerer, a corresponding trigger >> > implementation must be in place. >> > Good thing is , most of the required trigger implementations are already >> > available in current airflow providers. >> > >> > Additionally, i agree that there is no real difference between a >> deferrable >> > sensor and a deferrable operator, >> > generally , users utilize the BaseSensorOperator to create custom >> sensors, >> > However if they want to run >> > these sensors in triggerer, they must implement the triggerer's run >> method >> > by extending the base trigger class. >> > >> > The key difference with this proposal is the introduction of a common >> > trigger implementation (referred to in this >> > POC as InternalSensorTrigger )[1]. This would allow users to use the >> > new feature start_from_trigger param >> > with their custom sensor and eliminates the need for individual trigger >> > implementations for each custom sensor >> > when they create. >> > >> > Alternatively, we could leave the start_from_trigger parameter in >> > BaseSensorOperator with a default >> > value as false and let users decide whether they want to run the sensor >> in >> > triggerrer. If they choose to do so, they >> > can simply set the parameter to true and triggerrer uses the >> > InternalSensorTrigger class. >> > >> > Apologies if my answer is too confusing :) >> > >> > [1]: >> > >> https://github.com/apache/airflow/pull/41355/files#diff-7486f32e385d7ad0376cccda08d80e54939aa901a24616d11fb1f5cba6af7f83R144 >> > >> > Regards, >> > Pavan Kumar >> > >> > On Wed, Aug 14, 2024 at 12:29 AM Jarek Potiuk <ja...@potiuk.com> wrote: >> > >> >> How does it differ from the upcoming 2.10 feature? >> >> >> >> * Deferrable operators can now execute directly from the triggerer >> without >> >> needing to go through the worker. This is especially efficient for >> certain >> >> operators, like sensors, and can help teams save both time and money. >> >> >> >> As of 2.10 - Sensors already can run mostly in Triggerrer and basically >> >> there is no big difference any more between deferrable sensor and >> >> deferrable operator. >> >> >> >> >> >> On Mon, Aug 12, 2024 at 3:59 AM Kaxil Naik <kaxiln...@gmail.com> >> wrote: >> >> >> >>> Thanks for putting this together, I will take a look this week. >> >>> >> >>> On Fri, 9 Aug 2024 at 13:12, Pavankumar Gopidesu < >> >> gopidesupa...@gmail.com> >> >>> wrote: >> >>> >> >>>> Hi All, >> >>>> >> >>>> I have created a draft document for Sensor Improvements using >> triggers. >> >>>> >> >>>> Details: >> >>>> >> >>>> >> >>> >> >> >> https://docs.google.com/document/d/1Kb_wL-T1DHkOpmR_QNa3O5p_2hMTLzM-sb_HzQ0PYCo/edit?usp=sharing >> >>>> >> >>>> POC: >> >>>> https://github.com/apache/airflow/pull/41355 >> >>>> >> >>>> This is my first draft post, apologies if any mistakes in the >> >> document. I >> >>>> would greatly appreciate your insights and suggestions on draft. >> >>>> >> >>>> Thank you very much for your time. >> >>>> >> >>>> Regards, >> >>>> Pavan >> >>>> >> >>> >> >> >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org >> For additional commands, e-mail: dev-h...@airflow.apache.org >> >>
