Yes. Agree with Ash that when we have "sdk" that will be all-but-guaranteed (or at least intended to be) backwards compatible and no DB access from workers, or DAG processor, then yes we can change the rules for Airflow 3 going forward, but Airflow 2 support question remains.
Basically it boils down to "How long are we going to support Airflow 2 in providers released while Airflow 3 is already out?". 12 months after Airflow 3.0 is released is if we use the current rule. No strong opinions of that but that looks good. J On Thu, Aug 15, 2024 at 1:57 PM Ash Berlin-Taylor <a...@apache.org> wrote: > Going forward with Airflow 3 I also think we should re-consider what we > version/depend upon from providers. > > If we move all providers (and likley also utility functions!) out of > airflow core then the thing that a given provider needs to depend upon is > the shared library/provider versions and the "Task SDK” that we are > introducing with AIP-72, and the version of Airflow core doesn’t matter > anymore. > > This doesn’t take away the need to think about how long we continue to > support Airflow 2 in our providers, but that going forward I think we need > to subtly shift how we think about this. The first “key use case” we > mention in AIP 72 is this: > > > > 1. Ensure that this interface reduces the interaction between the code > running within the Task and the rest of Airflow. This is to address > unintended ripple effects from core Airflow changes which has caused > numerous Airflow upgrade issues, because Task (i.e. DAG) code relied on > Core Airflow abstractions. This has been a common problem pointed out by > numerous Airflow users including early adopters. This proposal would enable > Airflow users to upgrade Airflow system components (Scheduler, et al), > without impacting DAG user code. > > I don’t yet have a clear proposal of what it should be, but I do want to > surface this. > > -ash > > > On 15 Aug 2024, at 12:05, Amogh Desai <amoghdesai....@gmail.com> wrote: > > > > Nice points to ponder upon, Jarek. Both topics are essential for ensuring > > that users have a clear path forwars. > > > >> We currently guarantee that the minimum Airflow version supported by a > > provider is the release date of the next minor version, plus 12 months. > > You’re asking if this should be adjusted for Airflow 3, correct? > > When we say adjust, what are we looking at? Are we thinking of extending > > the 12 month period, to lets say 18 months > > or something else? > > > > Few thoughts that come to my mind when I read this email: > > - A potential adjustment would be to extend the window to 18 months post > > the first Airflow 3 release, > > allowing users more time to transition without feeling > > rushed. Communicating this timeline clearly > > to the community will be key here. We should *emphasize* that while they > > extra time for the transition, > > they will eventually need to migrate to Airflow 3 to access the newer > > features and bug fixes. > > - We can introduce a policy (and document it well, maybe in > > > https://github.com/apache/airflow/blob/main/PROVIDERS.rst#upgrading-minimum-supported-version-of-airflow > > ) > > where providers are guaranteed to work with all Airflow versions released > > within 12 months (or 18) > > of the provider release. After that, users *should* expect that older > > providers may not be fully compatible > > with newer Airflow versions, and any issues arising from this won't be > > treated as bugs. > > > > We should also document these rules clearly to provide users with the > > context and rationale behind these decisions. > > > > Thanks & Regards, > > Amogh Desai > > > > > > On Wed, Aug 14, 2024 at 3:37 PM Jarek Potiuk <ja...@potiuk.com> wrote: > > > >>> We currently guarantee that the minimum Airflow version supported by a > >> provider is the release date of the next minor version, plus 12 months. > >> You’re asking if this should be adjusted for Airflow 3, correct? > >> > >> Yes > >> > >>> Are you asking for guarantees the other way around, so Airflow version > -> > >> guaranteed provider package min version support? > >> > >> Yes. > >> > >> On Wed, Aug 14, 2024 at 12:04 PM Bas Harenslak > <b...@astronomer.io.invalid> > >> wrote: > >> > >>> Sorry I’m a bit lost in the long text. Is my understanding of the 2 > >>> questions correct? > >>> > >>> We currently guarantee that the minimum Airflow version supported by a > >>> provider is the release date of the next minor version, plus 12 months. > >>> You’re asking if this should be adjusted for Airflow 3, correct? > >>> Are you asking for guarantees the other way around, so Airflow version > -> > >>> guaranteed provider package min version support? > >>> > >>> Bas > >>> > >>>> On 14 Aug 2024, at 11:22, Jarek Potiuk <ja...@potiuk.com> wrote: > >>>> > >>>> Hello everyone, > >>>> > >>>> I have two important topics to discuss regarding Provider <> Airflow > >>>> version support rules. > >>>> > >>>> *1) What is the "min Airflow version" support while we are > >> transitioning > >>> to > >>>> Airflow 3? * > >>>> > >>>> So far we had the [1] rule: > >>>> > >>>>> The default support timespan for the minimum version of Airflow > (there > >>>> could be justified exceptions) is that we increase the minimum Airflow > >>>> version to the next MINOR release, when 12 months passed since the > >> first > >>>> release for the MINOR version of Airflow. > >>>> > >>>>> For example this means that by default we upgrade the minimum version > >> of > >>>> Airflow supported by providers to 2.8.0 in the first Provider's > release > >>>> after 18th of December 2024. 18th of December 2023 is the date when > the > >>>> first PATCHLEVEL of 2.8 (2.8.0) has been released. > >>>> > >>>> The next cutoff time (2.9.0 + 12 months) will be in April 2025, And > the > >>>> next one - 2.10.0 - assuming we release 2.10.0 in August - for min > >>> airflow > >>>> version = 2.10.0 will be in August 2025. If we have "bridge" 2.11.0 > >>> release > >>>> (very likely) - say in December, then min_version = 2.11.0 will be in > >>>> December 2025. > >>>> > >>>> Assuming that we release Airflow 3.0.0 in March 2025, if we follow > this > >>>> rule, we will be able to drop Airflow 2 support in providers in March > >>> 2026 > >>>> - so we give about a year to Airflow 2 users to migrate to Airflow 3 > - > >>> if > >>>> they will want new features and bug-fixes from new providers. Also it > >>> means > >>>> that we have 1.5 year of supporting both Airflow 2 and Airflow 3 in > >>>> providers (that for example includes the old Templated Field > >>>> back-compatibility support assuming that we will want to go ahead with > >>> the > >>>> change) > >>>> > >>>> That provides (as was the original idea of this rolling min-version > >>>> support) additional incentive for users to migrate (want new features > >>> from > >>>> providers - migrate to latest Airflow). With Airflow 3 that will be a > >>> more > >>>> difficult decision to migrate, so I wonder if this rule should be > >>> somewhat > >>>> adapted. > >>>> > >>>> *2) Should we introduce a rule for a "rolling" min version Provider's > >>>> support in Airflow 3?* > >>>> > >>>> In Airflow 2 we never had a rule describing "how old" providers are > >> still > >>>> supported in newer versions of Airflow. For example we never said you > >>> need > >>>> to have at least "2.10.0" version of a google provider. This was never > >> a > >>>> huge problem, because vast majority of users were upgrading providers > >>> with > >>>> Airflow and only occasionally downgraded providers or kept them with > >> old > >>>> versions because of some breaking changes.That had never caused a big > >>>> problem as far as I know and remember - and we can safely assume that > >>>> "really old" providers might not work with "new" airflow even if we > >> had > >>> no > >>>> formal rule for it. > >>>> > >>>> However that required some back-compatibility code sometimes - like > >> this > >>>> code removal that started breaking our main after removing old > >> "imports" > >>>> [2]. This means that some very old SB providers will not work with the > >>> new > >>>> common.sql provider when released (and they also will not work with > >>> Airflow > >>>> 3). I think it will never cause a "real" serious issue - but would be > >>> great > >>>> to formalize it to be able to set expectations of our users and have > >> some > >>>> ways to respond to potential issues they might raise. > >>>> > >>>> Should we formalize it somehow? For example add a rule that providers > >>> will > >>>> be guaranteed to support Airflow versions released up to 12 months > >> after > >>>> the provider was released? We could potentially add tests to check it > >> in > >>>> the future (though it would be rather complex and time consuming). But > >> I > >>>> think we should start with the rule first. > >>>> > >>>> Example to illustrate it (hypothetical): > >>>> > >>>> * Google provider 11.0.4 is released in December 2024 - just before > >>>> Airflow 2.11.0 > >>>> * Airflow 3.0 - 3.4 are released between December 2024 and December > >> 2025 > >>> - > >>>> all of them support google provider 2.11.0 (and if not - this is a bug > >>> that > >>>> needs to be fixed) > >>>> * Airflow 3.5 is released in January 2026 - and if someone wants to > >>> install > >>>> Airflow 3.5 and use Google provider 11.0.4 or below - they might, or > >>> might > >>>> not work. But if they don't we do not treat it as a bug and we > >> recommend > >>>> the users to upgrade to a newer Google provider version. > >>>> > >>>> I think we should not make "intentional" changes to break those > >>> providers, > >>>> unless we have good reasons, and it's mostly about some weird edge > >> cases, > >>>> where people have new Airflow and old providers, but I think it's > worth > >>>> introducing and documenting some rules about it. > >>>> > >>>> > >>>> Let me know what you think on both accounts. > >>>> > >>>> J. > >>>> > >>>> [1] > >>>> > >>> > >> > https://github.com/apache/airflow/blob/main/PROVIDERS.rst#upgrading-minimum-supported-version-of-airflow > >>>> [2] https://github.com/apache/airflow/pull/41461 > >>> > >>> > >> > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > For additional commands, e-mail: dev-h...@airflow.apache.org > >
