Going forward with Airflow 3 I also think we should re-consider what we version/depend upon from providers.
If we move all providers (and likley also utility functions!) out of airflow core then the thing that a given provider needs to depend upon is the shared library/provider versions and the "Task SDK” that we are introducing with AIP-72, and the version of Airflow core doesn’t matter anymore. This doesn’t take away the need to think about how long we continue to support Airflow 2 in our providers, but that going forward I think we need to subtly shift how we think about this. The first “key use case” we mention in AIP 72 is this: > > 1. Ensure that this interface reduces the interaction between the code > running within the Task and the rest of Airflow. This is to address > unintended ripple effects from core Airflow changes which has caused numerous > Airflow upgrade issues, because Task (i.e. DAG) code relied on Core Airflow > abstractions. This has been a common problem pointed out by numerous Airflow > users including early adopters. This proposal would enable Airflow users to > upgrade Airflow system components (Scheduler, et al), without impacting DAG > user code. I don’t yet have a clear proposal of what it should be, but I do want to surface this. -ash > On 15 Aug 2024, at 12:05, Amogh Desai <amoghdesai....@gmail.com> wrote: > > Nice points to ponder upon, Jarek. Both topics are essential for ensuring > that users have a clear path forwars. > >> We currently guarantee that the minimum Airflow version supported by a > provider is the release date of the next minor version, plus 12 months. > You’re asking if this should be adjusted for Airflow 3, correct? > When we say adjust, what are we looking at? Are we thinking of extending > the 12 month period, to lets say 18 months > or something else? > > Few thoughts that come to my mind when I read this email: > - A potential adjustment would be to extend the window to 18 months post > the first Airflow 3 release, > allowing users more time to transition without feeling > rushed. Communicating this timeline clearly > to the community will be key here. We should *emphasize* that while they > extra time for the transition, > they will eventually need to migrate to Airflow 3 to access the newer > features and bug fixes. > - We can introduce a policy (and document it well, maybe in > https://github.com/apache/airflow/blob/main/PROVIDERS.rst#upgrading-minimum-supported-version-of-airflow > ) > where providers are guaranteed to work with all Airflow versions released > within 12 months (or 18) > of the provider release. After that, users *should* expect that older > providers may not be fully compatible > with newer Airflow versions, and any issues arising from this won't be > treated as bugs. > > We should also document these rules clearly to provide users with the > context and rationale behind these decisions. > > Thanks & Regards, > Amogh Desai > > > On Wed, Aug 14, 2024 at 3:37 PM Jarek Potiuk <ja...@potiuk.com> wrote: > >>> We currently guarantee that the minimum Airflow version supported by a >> provider is the release date of the next minor version, plus 12 months. >> You’re asking if this should be adjusted for Airflow 3, correct? >> >> Yes >> >>> Are you asking for guarantees the other way around, so Airflow version -> >> guaranteed provider package min version support? >> >> Yes. >> >> On Wed, Aug 14, 2024 at 12:04 PM Bas Harenslak <b...@astronomer.io.invalid> >> wrote: >> >>> Sorry I’m a bit lost in the long text. Is my understanding of the 2 >>> questions correct? >>> >>> We currently guarantee that the minimum Airflow version supported by a >>> provider is the release date of the next minor version, plus 12 months. >>> You’re asking if this should be adjusted for Airflow 3, correct? >>> Are you asking for guarantees the other way around, so Airflow version -> >>> guaranteed provider package min version support? >>> >>> Bas >>> >>>> On 14 Aug 2024, at 11:22, Jarek Potiuk <ja...@potiuk.com> wrote: >>>> >>>> Hello everyone, >>>> >>>> I have two important topics to discuss regarding Provider <> Airflow >>>> version support rules. >>>> >>>> *1) What is the "min Airflow version" support while we are >> transitioning >>> to >>>> Airflow 3? * >>>> >>>> So far we had the [1] rule: >>>> >>>>> The default support timespan for the minimum version of Airflow (there >>>> could be justified exceptions) is that we increase the minimum Airflow >>>> version to the next MINOR release, when 12 months passed since the >> first >>>> release for the MINOR version of Airflow. >>>> >>>>> For example this means that by default we upgrade the minimum version >> of >>>> Airflow supported by providers to 2.8.0 in the first Provider's release >>>> after 18th of December 2024. 18th of December 2023 is the date when the >>>> first PATCHLEVEL of 2.8 (2.8.0) has been released. >>>> >>>> The next cutoff time (2.9.0 + 12 months) will be in April 2025, And the >>>> next one - 2.10.0 - assuming we release 2.10.0 in August - for min >>> airflow >>>> version = 2.10.0 will be in August 2025. If we have "bridge" 2.11.0 >>> release >>>> (very likely) - say in December, then min_version = 2.11.0 will be in >>>> December 2025. >>>> >>>> Assuming that we release Airflow 3.0.0 in March 2025, if we follow this >>>> rule, we will be able to drop Airflow 2 support in providers in March >>> 2026 >>>> - so we give about a year to Airflow 2 users to migrate to Airflow 3 - >>> if >>>> they will want new features and bug-fixes from new providers. Also it >>> means >>>> that we have 1.5 year of supporting both Airflow 2 and Airflow 3 in >>>> providers (that for example includes the old Templated Field >>>> back-compatibility support assuming that we will want to go ahead with >>> the >>>> change) >>>> >>>> That provides (as was the original idea of this rolling min-version >>>> support) additional incentive for users to migrate (want new features >>> from >>>> providers - migrate to latest Airflow). With Airflow 3 that will be a >>> more >>>> difficult decision to migrate, so I wonder if this rule should be >>> somewhat >>>> adapted. >>>> >>>> *2) Should we introduce a rule for a "rolling" min version Provider's >>>> support in Airflow 3?* >>>> >>>> In Airflow 2 we never had a rule describing "how old" providers are >> still >>>> supported in newer versions of Airflow. For example we never said you >>> need >>>> to have at least "2.10.0" version of a google provider. This was never >> a >>>> huge problem, because vast majority of users were upgrading providers >>> with >>>> Airflow and only occasionally downgraded providers or kept them with >> old >>>> versions because of some breaking changes.That had never caused a big >>>> problem as far as I know and remember - and we can safely assume that >>>> "really old" providers might not work with "new" airflow even if we >> had >>> no >>>> formal rule for it. >>>> >>>> However that required some back-compatibility code sometimes - like >> this >>>> code removal that started breaking our main after removing old >> "imports" >>>> [2]. This means that some very old SB providers will not work with the >>> new >>>> common.sql provider when released (and they also will not work with >>> Airflow >>>> 3). I think it will never cause a "real" serious issue - but would be >>> great >>>> to formalize it to be able to set expectations of our users and have >> some >>>> ways to respond to potential issues they might raise. >>>> >>>> Should we formalize it somehow? For example add a rule that providers >>> will >>>> be guaranteed to support Airflow versions released up to 12 months >> after >>>> the provider was released? We could potentially add tests to check it >> in >>>> the future (though it would be rather complex and time consuming). But >> I >>>> think we should start with the rule first. >>>> >>>> Example to illustrate it (hypothetical): >>>> >>>> * Google provider 11.0.4 is released in December 2024 - just before >>>> Airflow 2.11.0 >>>> * Airflow 3.0 - 3.4 are released between December 2024 and December >> 2025 >>> - >>>> all of them support google provider 2.11.0 (and if not - this is a bug >>> that >>>> needs to be fixed) >>>> * Airflow 3.5 is released in January 2026 - and if someone wants to >>> install >>>> Airflow 3.5 and use Google provider 11.0.4 or below - they might, or >>> might >>>> not work. But if they don't we do not treat it as a bug and we >> recommend >>>> the users to upgrade to a newer Google provider version. >>>> >>>> I think we should not make "intentional" changes to break those >>> providers, >>>> unless we have good reasons, and it's mostly about some weird edge >> cases, >>>> where people have new Airflow and old providers, but I think it's worth >>>> introducing and documenting some rules about it. >>>> >>>> >>>> Let me know what you think on both accounts. >>>> >>>> J. >>>> >>>> [1] >>>> >>> >> https://github.com/apache/airflow/blob/main/PROVIDERS.rst#upgrading-minimum-supported-version-of-airflow >>>> [2] https://github.com/apache/airflow/pull/41461 >>> >>> >> --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org For additional commands, e-mail: dev-h...@airflow.apache.org
