Hi Jarek, Thanks for the questions :) ,
I completely agree with you , from 2.10 we have the start_from_trigger parameter, which , when set , allows a task to be executed directly from the triggerrer without worker involvement. I believe that for any sensor to be executed in the triggerer, a corresponding trigger implementation must be in place. Good thing is , most of the required trigger implementations are already available in current airflow providers. Additionally, i agree that there is no real difference between a deferrable sensor and a deferrable operator, generally , users utilize the BaseSensorOperator to create custom sensors, However if they want to run these sensors in triggerer, they must implement the triggerer's run method by extending the base trigger class. The key difference with this proposal is the introduction of a common trigger implementation (referred to in this POC as InternalSensorTrigger )[1]. This would allow users to use the new feature start_from_trigger param with their custom sensor and eliminates the need for individual trigger implementations for each custom sensor when they create. Alternatively, we could leave the start_from_trigger parameter in BaseSensorOperator with a default value as false and let users decide whether they want to run the sensor in triggerrer. If they choose to do so, they can simply set the parameter to true and triggerrer uses the InternalSensorTrigger class. Apologies if my answer is too confusing :) [1]: https://github.com/apache/airflow/pull/41355/files#diff-7486f32e385d7ad0376cccda08d80e54939aa901a24616d11fb1f5cba6af7f83R144 Regards, Pavan Kumar On Wed, Aug 14, 2024 at 12:29 AM Jarek Potiuk <ja...@potiuk.com> wrote: > How does it differ from the upcoming 2.10 feature? > > * Deferrable operators can now execute directly from the triggerer without > needing to go through the worker. This is especially efficient for certain > operators, like sensors, and can help teams save both time and money. > > As of 2.10 - Sensors already can run mostly in Triggerrer and basically > there is no big difference any more between deferrable sensor and > deferrable operator. > > > On Mon, Aug 12, 2024 at 3:59 AM Kaxil Naik <kaxiln...@gmail.com> wrote: > > > Thanks for putting this together, I will take a look this week. > > > > On Fri, 9 Aug 2024 at 13:12, Pavankumar Gopidesu < > gopidesupa...@gmail.com> > > wrote: > > > > > Hi All, > > > > > > I have created a draft document for Sensor Improvements using triggers. > > > > > > Details: > > > > > > > > > https://docs.google.com/document/d/1Kb_wL-T1DHkOpmR_QNa3O5p_2hMTLzM-sb_HzQ0PYCo/edit?usp=sharing > > > > > > POC: > > > https://github.com/apache/airflow/pull/41355 > > > > > > This is my first draft post, apologies if any mistakes in the > document. I > > > would greatly appreciate your insights and suggestions on draft. > > > > > > Thank you very much for your time. > > > > > > Regards, > > > Pavan > > > > > >
