Thank you for bringing this up! I have added some comments to the document. I'm unsure if we really want or need to implement more complex logic for this. What I have in mind is simply adding helper functions to InternalSensorTrigger and continuing to use the run method in BaseTrigger. The main purpose of those methods is to allow operator authors to change the sensors to leverage this start/end from trigger more easily.
Best, Wei > On Aug 14, 2024, at 5:19 PM, Pavankumar Gopidesu <gopidesupa...@gmail.com> > wrote: > > Hi Jarek, > > Thanks for the questions :) , > > I completely agree with you , from 2.10 we have the start_from_trigger > parameter, which , when set , > allows a task to be executed directly from the triggerrer without worker > involvement. I believe that for > any sensor to be executed in the triggerer, a corresponding trigger > implementation must be in place. > Good thing is , most of the required trigger implementations are already > available in current airflow providers. > > Additionally, i agree that there is no real difference between a deferrable > sensor and a deferrable operator, > generally , users utilize the BaseSensorOperator to create custom sensors, > However if they want to run > these sensors in triggerer, they must implement the triggerer's run method > by extending the base trigger class. > > The key difference with this proposal is the introduction of a common > trigger implementation (referred to in this > POC as InternalSensorTrigger )[1]. This would allow users to use the > new feature start_from_trigger param > with their custom sensor and eliminates the need for individual trigger > implementations for each custom sensor > when they create. > > Alternatively, we could leave the start_from_trigger parameter in > BaseSensorOperator with a default > value as false and let users decide whether they want to run the sensor in > triggerrer. If they choose to do so, they > can simply set the parameter to true and triggerrer uses the > InternalSensorTrigger class. > > Apologies if my answer is too confusing :) > > [1]: > https://github.com/apache/airflow/pull/41355/files#diff-7486f32e385d7ad0376cccda08d80e54939aa901a24616d11fb1f5cba6af7f83R144 > > Regards, > Pavan Kumar > > On Wed, Aug 14, 2024 at 12:29 AM Jarek Potiuk <ja...@potiuk.com> wrote: > >> How does it differ from the upcoming 2.10 feature? >> >> * Deferrable operators can now execute directly from the triggerer without >> needing to go through the worker. This is especially efficient for certain >> operators, like sensors, and can help teams save both time and money. >> >> As of 2.10 - Sensors already can run mostly in Triggerrer and basically >> there is no big difference any more between deferrable sensor and >> deferrable operator. >> >> >> On Mon, Aug 12, 2024 at 3:59 AM Kaxil Naik <kaxiln...@gmail.com> wrote: >> >>> Thanks for putting this together, I will take a look this week. >>> >>> On Fri, 9 Aug 2024 at 13:12, Pavankumar Gopidesu < >> gopidesupa...@gmail.com> >>> wrote: >>> >>>> Hi All, >>>> >>>> I have created a draft document for Sensor Improvements using triggers. >>>> >>>> Details: >>>> >>>> >>> >> https://docs.google.com/document/d/1Kb_wL-T1DHkOpmR_QNa3O5p_2hMTLzM-sb_HzQ0PYCo/edit?usp=sharing >>>> >>>> POC: >>>> https://github.com/apache/airflow/pull/41355 >>>> >>>> This is my first draft post, apologies if any mistakes in the >> document. I >>>> would greatly appreciate your insights and suggestions on draft. >>>> >>>> Thank you very much for your time. >>>> >>>> Regards, >>>> Pavan >>>> >>> >> --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org For additional commands, e-mail: dev-h...@airflow.apache.org
