Hi all,
yes, diverged a bit. I'll add an agenda item for the next Airflow 3 dev
call.
Main question as outcome would be: Shall we split the Providers if we
are (ongoing) also planning to split the deployment dependencies of
Scheduler, Worker, Webserver etc?
-->
https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes#Airflow3Devcall:MeetingNotes-19September2024
Jens
On 16.09.24 16:54, Vincent Beck wrote:
On one side it makes sense to me, and I actually like the thinking "providers should
only be for DAG authors". That makes it simple to figure whether something should
belong to providers. If we go that way then FAB would no longer be a provider but a
plugin which would be one step closer to not install providers on the webserver.
On the other side, I can see some concerns/questions:
- Where would these plugins be in the codebase? And more importantly, how would
they be released? As part of Airflow? As a separate release management like
apache-airflow-plugins?
- Some executors (e.g. AwsEcsExecutor), depend on hooks (AWS hook). If we move
executors outside of providers, then we need the plugin to depend on some
providers?
PS: we diverged from the original topic, we might want to create a separate
thread for that.
On 2024/09/14 12:52:25 Ash Berlin-Taylor wrote:
Provider package name: "edge", so gets to package "airflow.providers.edge"
I'm not so sure about this name. I have no problems with the"edge" part, I am
mostly questioning if this should be a provider, or should it be another kind of module.
For instance it won't have an Operator, Sensor or Hook, nor need anything else provided
by the Provider Manager.
Similarly now I think about it for Celery executor, it's not got anything for
use in dags does it?
My thinking here is: if a dag author uses it directly -> "provider"
If it's only used by the deployment manager -> something else, possibly
"plugin"?
Things that I think should fall in this second category would be the FAB,
Celery and Edge executors. cncf.kubernetes is mixed as it has the executor and
an operator.
What do others think?
On 8 September 2024 14:05:51 BST, Jens Scheffler <j_scheff...@gmx.de.INVALID>
wrote:
Hi Devs,
as we had a naming discussion for AIP-69 in
https://lists.apache.org/thread/br1jfoc8p1wjzk74c09srjgr29spytfy and PRs
are ready, Elad proposed to have at least a lazy consensus to close the
naming before merge.
Not having real counts leave "Distributed" and "Edge" close-by with most
positive feedback. As nobody objected and the term "Edge" is shorter...
That means:
- Provider package name: "edge", so gets to package "airflow.providers.edge"
- Executor class: "EdgeExecutor"
- Remotely running worker name: "Edge Worker" - called via `airflow edge
worker [options]` (similar like celery)
If somebody wants to have a look to the implementation, several PRs are
split for better review:
- "Mothership" PR with the complete implementation if you want to have a
test drive: https://github.com/apache/airflow/pull/40900 (=3800 LoC)
- Split 1: (Full) Edge provider package:
https://github.com/apache/airflow/pull/41729 (3700 LoC)
- Split 1.1: (Empty) Provider package (Empty boilerplate)
https://github.com/apache/airflow/pull/42046 (450 LoC)
- Split 1.2: DB Models for Edge Provider
https://github.com/apache/airflow/pull/42047 (1.1 + 950 LoC)
- Split 1.3: EdgeExecutor
https://github.com/apache/airflow/pull/42048 (1.1+1.2 + 650 LoC)
- Split 1.4: Rest API Endpoint
https://github.com/apache/airflow/pull/42049 (1.1+1.2 + 1050 LoC)
- Split 1.5: Worker CLI
https://github.com/apache/airflow/pull/42050 (1.1+1.2 + 650 LoC)
- Split 1.6: Leftover glue to bring all together
https://github.com/apache/airflow/pull/42051 (1.1-1.5 + 25 LoC)
- Split 2: (Small) Adjustments in Core:
https://github.com/apache/airflow/pull/41730 (<10 LoC)
- Split 3: Breeze adjustments to develop/start via CLI
https://github.com/apache/airflow/pull/41731 (<200 LoC)
If anyone objects to the consensus here, let me/devlist know - otherwise this
will be effective by Thursday 12th of September 2024 18.00 am PST (which
coincidentally is when
Airflow Summit is completed in San Francisco) - Also looking forward for
approvals on the codebase / PRs. PRs are ready to have something working.
Jens
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
For additional commands, e-mail: dev-h...@airflow.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
For additional commands, e-mail: dev-h...@airflow.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org
For additional commands, e-mail: dev-h...@airflow.apache.org
