This is actually the majority of our airflow work.
We use this pattern:
Sensor pings api (fairly quickly, in a dag that’s constrained to only run one
instance, every minute)
If the sensor gets a valid response, the next task is a custom operator that
extends Trigger, builds up the DagRun context, and triggers the actual
processing dag.
This has worked very well, and doesn’t require a custom scheduler.
Additionally, the “trigger dags” are fairly light on resources- 1 active sensor
each.
Sent from a device with less than stellar autocorrect
> On Jan 31, 2019, at 10:55 AM, abhishek sharma <abhioncbr.apa...@gmail.com>
> wrote:
>
> Hi Ben,
>
> Just copying my comment form ticket.
>
> I think current airflow scheduler schedule DAGs only on time-basis (based
> on cron schedule string). ***Is it correct understanding?*
>
> How to approach a scenario where I want to trigger a DAG based on some
> event which is not so predictable/regular on time basis?
>
> - One is to use sensor and DAG flow will start by first running a sensor
> task for checking an event and when an event happened actual processing get
> a start.
> - Second, is to have a DAG with None schedule and it gets triggered by
> some other application or utility which checks for an event occurrence .
>
> If your most of the flow(DAGs) are suppose to run on this logic, then
> sensor doesn't make sense and hence left with the second approach only,
> which is nothing but a custom scheduling of DAGs. ****Is it agreeable use
> case for the custom scheduler?*
>
> Thanks, {{Abhishek}}
>
>
>
>> On Thu, Jan 31, 2019 at 11:53 AM Ben Tallman <btall...@gmail.com> wrote:
>>
>> Can you explain a bit more what you are thinking for a custom scheduler?
>> It's been awhile, but we added support for cron schedules without backfill
>> awhile back, so I'm wondering what you are thinking of adding with this?
>>
>> Thanks,
>> Ben
>>
>> --
>> Ben Tallman - 503.680.5709
>>
>>
>> On Thu, Jan 31, 2019 at 8:29 AM abhishek sharma <
>> abhioncbr.apa...@gmail.com>
>> wrote:
>>
>>> Hi All,
>>>
>>> Created a ticket(https://issues.apache.org/jira/browse/AIRFLOW-3775) for
>>> supporting custom scheduler in Airflow.
>>>
>>> The idea is to have a scheduler base class which can be extended for
>>> writing a custom scheduler. The logic of custom scheduling is user
>>> specific, and at the DAGs task level we can mention the scheduler type,
>> and
>>> that scheduler will be used for starting a task. [Naive Idea]
>>>
>>> Can we please discuss whether we need such functionality in Airflow or
>> not
>>> and If yes then we will proceed with the design and implementation.
>>>
>>> Thanks
>>> Abhishek Sharma
>>>
>>
