Thanks Jarek and Ash for your feedback. Your concerns regarding performance are valid - I'll check if it can be addressed.
Shahar On Tue, May 5, 2026, 18:13 Ash Berlin-Taylor <[email protected]> wrote: > +1 to everything Jarek said. On point A, I’d go so far as to ”not might, > but almost certainly will cause performance regression”. > > Trigger rule evaluation is the hottest of hot paths in the scheduler, and > any change to that area of the code needs to be done with extreme care and > attention. > > -ash > > > On 2 May 2026, at 18:03, Jarek Potiuk <[email protected]> wrote: > > > > Hi Shahar, > > > > I am not against this proposal, but I have three concerns regarding > points > > not fully covered in the AIP: > > > > a) Performance: Trigger evaluation occurs in the critical scheduler loop. > > Moving from hard-coded Python criteria to a generalized expression model > > might introduce a performance hit during the iteration of task > candidates. > > > > b) Complexity and Reasoning: Allowing arbitrary combinations could lead > to > > contradicting or nonsensical conditions (e.g., requirements that exceed > the > > total number of upstream tasks). This complicates debugging for users > > asking "why is my task not running?" and makes it harder for maintainers > to > > discern the original author's intent compared to clear enums like > > ALL_FAILED. > > > > c) Flexibility vs. Intention: The proposed primitives feel more > > mathematical than business-logical. If a user doesn't care which specific > > upstream tasks succeed—only that a certain number do—those tasks are > likely > > "flavors" of the same work. Such logic might be better handled by > extending > > "mapped task" success criteria (e.g., a percentage threshold) rather than > > modifying trigger rules for distinct upstream nodes. > > > > I'd love to hear your thoughts and those of others on whether this > > complexity aligns with what DAG authors actually need. > > > > Best, > > Jarek Potiuk > > > > On Sat, May 2, 2026 at 11:40 AM Jens Scheffler <[email protected]> > wrote: > > > >> Hi Shahar, > >> > >> ich think this is a good extension for cases needing this withou adding > >> too much complexity. Whereas I am not sure whether in our environment we > >> have a use case. Would be good to hear from others if there are any. > >> Alternatives otherwise in my mind would only be adding more enum keys > >> for other cases (which is not really preferrable) or modelling the Dag > >> with a few EmptyOperators to collect pre-aggregate counts to leverage > >> existing enums... which would in turn create artificial Dag complexity > >> just because of limited enums. > >> > >> For me OK but would be cool to have other opinions as well. > >> > >> Jens > >> > >> On 02.05.26 11:16, Shahar Epstein wrote: > >>> Bumping, I'll appreciate your feedback. > >>> > >>> > >>> Shahar > >>> > >>> On Tue, Apr 21, 2026 at 2:00 PM Shahar Epstein <[email protected]> > >> wrote: > >>>> Hey everyone, > >>>> > >>>> I'd like to get your feedback on a new AIP proposal: > >>>> > >>>> AIP-106: Composable Trigger Rules > >>>> https://cwiki.apache.org/confluence/x/64wmGQ > >>>> > >>>> tl;dr > >>>> > >>>> This proposal introduces a composable expression model for trigger > >> rules, enabling conditions not expressible with the current enum-based > >> system, while fully preserving existing trigger rules for simple cases. > >>>> For example, "All done, at least two succeeded" can be expressed as > >> TR.expr(done="all", success=">=2"). > >>>> > >>>> Design Notes/Constraints: > >>>> 1. The aggregate form is AND-only by design, to preserve determinism > >> and keep evaluation predictable. > >>>> 2. The composable model complements (rather than replaces) the enum. > >> Some trigger rules (e.g. ALL_DONE_SETUP_SUCCESS, ONE_DONE, ALWAYS) > remain > >> intentionally out of scope, and enums remain the more readable option > for > >> simple cases. > >>>> > >>>> I'd appreciate your feedback on: > >>>> > >>>> Backward compatibility with existing trigger rules (see Appendix A) > >>>> > >>>> Semantics of eager vs complete evaluation > >>>> > >>>> Scope and limits of the expression model > >>>> > >>>> Potential edge cases and performance implications > >>>> > >>>> UI/observability considerations for debugging expressions > >>>> > >>>> Anything else > >>>> > >>>> > >>>> Thank you! > >>>> > >>>> > >>>> Shahar > >>> --------------------------------------------------------------------- > >>> To unsubscribe, e-mail: [email protected] > >>> For additional commands, e-mail: [email protected] > >>> > >> > >> --------------------------------------------------------------------- > >> To unsubscribe, e-mail: [email protected] > >> For additional commands, e-mail: [email protected] > >> > >> > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [email protected] > For additional commands, e-mail: [email protected] > >
