Peace, I hope to follow this discussion. I was pointed this way from Slack by Jarek Potiuk.
I'll copy my question and his reply here in hopes that any nuanced clarifications are sure to show up in any Airflow 3 documentation. My question as as follows: "What is the differene between a provider and a plugin? They both seem to be able to extend Airflow. Is it that providers are usually made by the community or 3rd-parties to communciate with their systems and plugins are for a company's dev team to make tooling to communicate for their internal systems? Might it be a matter of all providers being plugins, but not all plugins being providers? Or is it that all providers are plugins that usually handle communication with external systems but could do more, while plugins could do what providers do, but usually do more than communicate with external systems? Is there any article that goes into this? Is this something idiomatic that someone who's been with the community a long time would just get?" Reply from Jarek Potiuk: "This will get clarified with Airflow 3 - as there will be likely different kinds of extensions and the old plugin/provider distinction will be more nuanced - there will be more "types" likely (UI , Auth , logging extensions etc. ). Historically there is some overlap (that will disappear in Airflow 3). But generally you can think of providers as a way to extend Airflow to communicate with particular "External system" - while plugin is more about extending Airflow "Core" capabilties - for example scheduling, listeners, etc. "Main difference is about security of the "core" components (webserver and scheduler) - plugin is the only way that you can extend "scheduler" and "webserver") , where provider is where you want to interact with "another software/database/service" during dag parsing, worker and triggerer execution. "This will get clarified and "organized" better in Airflow 3 and we have - exactly now - discussion about it at the devlist with rough proposal on how we should split things out in Airflow 3 - you can take part in that discussion, read the doc, comment etc. https://lists.apache.org/thread/gm2rj7s4vsnyt95bxbh1pjkht49zvb3t"; On 2024/10/25 18:29:56 Vikram Koka wrote: > Hi everyone, > > Following up from the Providers packaging discussion in the dev call two > weeks ago, we have been working on a written proposal to share. > > While working on that proposal, it became obvious that it would be useful > to have a document detailing the Core Airflow 3 packaging and distribution > < https://docs.google.com/document/d/1isjpMKYdgdYpEGQDtN2hKrJ-oq8qFWU9kqootRUHiX8/ > > mechanism post AIP-72. Here is a draft proposal around the same. Ash, Elad, > Jarek, Jens, Kaxil, and I have all put some thought into this, and would > appreciate your input on the same. > https://docs.google.com/document/d/1isjpMKYdgdYpEGQDtN2hKrJ-oq8qFWU9kqootRUHiX8 > > The Providers Packaging proposal will be based on this and will follow > soon. Please consider this as a pre-read. > > Best regards, > Vikram >
