A sortof third option... Seems there could be good reason to release executor separately
e.g. `apache-airflow-providers-aws-executors` Separate from the dag authoring stuff. So e.g. with cncf you could stick with old dag authoring stuff but stay on latest k8s executor. On Mon, Sep 16, 2024 at 7:54 AM Vincent Beck <vincb...@apache.org> wrote: > On one side it makes sense to me, and I actually like the thinking > "providers should only be for DAG authors". That makes it simple to figure > whether something should belong to providers. If we go that way then FAB > would no longer be a provider but a plugin which would be one step closer > to not install providers on the webserver. > > On the other side, I can see some concerns/questions: > > - Where would these plugins be in the codebase? And more importantly, how > would they be released? As part of Airflow? As a separate release > management like apache-airflow-plugins? > > - Some executors (e.g. AwsEcsExecutor), depend on hooks (AWS hook). If we > move executors outside of providers, then we need the plugin to depend on > some providers? > > PS: we diverged from the original topic, we might want to create a > separate thread for that. > > On 2024/09/14 12:52:25 Ash Berlin-Taylor wrote: > > > Provider package name: "edge", so gets to package > "airflow.providers.edge" > > > > I'm not so sure about this name. I have no problems with the"edge" part, > I am mostly questioning if this should be a provider, or should it be > another kind of module. For instance it won't have an Operator, Sensor or > Hook, nor need anything else provided by the Provider Manager. > > > > Similarly now I think about it for Celery executor, it's not got > anything for use in dags does it? > > > > My thinking here is: if a dag author uses it directly -> "provider" > > If it's only used by the deployment manager -> something else, possibly > "plugin"? > > > > Things that I think should fall in this second category would be the > FAB, Celery and Edge executors. cncf.kubernetes is mixed as it has the > executor and an operator. > > > > What do others think? > > > > On 8 September 2024 14:05:51 BST, Jens Scheffler > <j_scheff...@gmx.de.INVALID> wrote: > > >Hi Devs, > > > > > >as we had a naming discussion for AIP-69 in > > >https://lists.apache.org/thread/br1jfoc8p1wjzk74c09srjgr29spytfy and > PRs > > >are ready, Elad proposed to have at least a lazy consensus to close the > > >naming before merge. > > > > > >Not having real counts leave "Distributed" and "Edge" close-by with most > > >positive feedback. As nobody objected and the term "Edge" is shorter... > > >That means: > > > > > >- Provider package name: "edge", so gets to package > "airflow.providers.edge" > > > > > >- Executor class: "EdgeExecutor" > > > > > >- Remotely running worker name: "Edge Worker" - called via `airflow edge > > >worker [options]` (similar like celery) > > > > > > > > >If somebody wants to have a look to the implementation, several PRs are > > >split for better review: > > > > > >- "Mothership" PR with the complete implementation if you want to have a > > >test drive: https://github.com/apache/airflow/pull/40900 (=3800 LoC) > > > > > > - Split 1: (Full) Edge provider package: > > >https://github.com/apache/airflow/pull/41729 (3700 LoC) > > > > > > - Split 1.1: (Empty) Provider package (Empty boilerplate) > > >https://github.com/apache/airflow/pull/42046 (450 LoC) > > > > > > - Split 1.2: DB Models for Edge Provider > > >https://github.com/apache/airflow/pull/42047 (1.1 + 950 LoC) > > > > > > - Split 1.3: EdgeExecutor > > >https://github.com/apache/airflow/pull/42048 (1.1+1.2 + 650 LoC) > > > > > > - Split 1.4: Rest API Endpoint > > >https://github.com/apache/airflow/pull/42049 (1.1+1.2 + 1050 LoC) > > > > > > - Split 1.5: Worker CLI > > >https://github.com/apache/airflow/pull/42050 (1.1+1.2 + 650 LoC) > > > > > > - Split 1.6: Leftover glue to bring all together > > >https://github.com/apache/airflow/pull/42051 (1.1-1.5 + 25 LoC) > > > > > > - Split 2: (Small) Adjustments in Core: > > >https://github.com/apache/airflow/pull/41730 (<10 LoC) > > > > > > - Split 3: Breeze adjustments to develop/start via CLI > > >https://github.com/apache/airflow/pull/41731 (<200 LoC) > > > > > > > > >If anyone objects to the consensus here, let me/devlist know - > otherwise this will be effective by Thursday 12th of September 2024 18.00 > am PST (which coincidentally is when > > >Airflow Summit is completed in San Francisco) - Also looking forward > for approvals on the codebase / PRs. PRs are ready to have something > working. > > > > > >Jens > > > > > > > > >--------------------------------------------------------------------- > > >To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > >For additional commands, e-mail: dev-h...@airflow.apache.org > > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > For additional commands, e-mail: dev-h...@airflow.apache.org > >
