Ramit I think we also agreed that we need to improve the common notifiers
(at least Slack) with templates that will prevent the need to customize the
code from the user side. The problem is explained in
https://github.com/apache/airflow/issues/35381 and was solved for SMTP only
in https://github.co
Following today’s dev call discussion, the community aligned on proceeding with
a variation of option 1. The check for deadline misses will happen in the
scheduler main loop (not a separate or child process)
For callbacks, suggested practice will be to use async callbacks, and the
existing conce
Yeah, that’s how I understood it to mean, and then the callback would go to a
worker (or a trigger) to run the user code.
-a
> On 22 May 2025, at 16:06, Jarek Potiuk wrote:
>
>> Option 1 as originally proposed in this thread only does the "is a
> callback
> required" check in scheduler -- not
> Option 1 as originally proposed in this thread only does the "is a
callback
required" check in scheduler -- not running the callback in scheduler
Ah - then OK. I thought it's also callback execution. Just checking is fine
in scheduler.
On Thu, May 22, 2025 at 4:55 PM Daniel Standish
wrote:
>
Option 1 as originally proposed in this thread only does the "is a callback
required" check in scheduler -- not running the callback in scheduler.
On Thu, May 22, 2025 at 7:22 AM Jarek Potiuk wrote:
> > So I very strongly vote for Option 1, and if needed make the scheduler
> itself more resilien
> So I very strongly vote for Option 1, and if needed make the scheduler
itself more resilient. The Airflow Scheduler _IS_ airflow. Let’s do what we
need to in order to make it more stable, rather than working around a
problem of our own making, whilst also making it operationally more complex
to r
My comment on the name is for the suggested component that runs the
workload. It's not about the feature itself. I just suggest a more generic
name so if the need comes it would be easier to execute different kind of
workloads on it (like callbacks).
As for reuse the Triggerer I am not a fan of th
I’m -1 on making it a new separate process — we already have too many than I
feel comfortable with, and each new one we add seriously impacts the ability to
usefully run “small” airflow deployments.
I also think I disagree with the framing of your original message:
> * It would be more robus
How about Option 3) making it part of triggerer.
I think that goes in the direction we've been discussing in the past where
we have 'generic workload" that we can submit from any of the other
components that will be executed in triggerer.
* that would not add too much complexity - no extra proces
I prefer option 2 but I have questions.
1. Naming wise maybe we should prefer a more generic name as I am not sure
if it should be limited to deadlines? (maybe should be shared with
executing callbacks?)
2. How do you plan to manage the queue of alerts? What happens if the
process is unhealthy whil
+1 for option 2, primarily because of:
It would be more robust and resilient, and therefore be able to run the
> callbacks *even in presence of certain kinds of issues like the scheduler
> being bogged-down*
On Wed, May 21, 2025 at 5:09 PM Kataria, Ramit
wrote:
> Hi all,
>
> I’m working with
11 matches
Mail list logo
