That looks quite good to me because as per the below quote from you: > Assuming that we release Airflow 3.0.0 in March 2025, if we follow this rule, we will be able to drop Airflow 2 support in providers in March 2026 - so we give about a year to Airflow 2 users to migrate to Airflow 3 - if they will want new features and bug-fixes from new providers. Also it means that we have 1.5 year of supporting both Airflow 2 and Airflow 3 in providers (that for example includes the old Templated Field back-compatibility support assuming that we will want to go ahead with the change) It seems like there still will be 18 months for users to comfortably migrate.
Thanks & Regards, Amogh Desai On Fri, Aug 16, 2024 at 7:36 PM Jarek Potiuk <ja...@potiuk.com> wrote: > Yes. Agree with Ash that when we have "sdk" that will be all-but-guaranteed > (or at least intended to be) backwards compatible and no DB access from > workers, or DAG processor, then yes we can change the rules for Airflow 3 > going forward, but Airflow 2 support question remains. > > Basically it boils down to "How long are we going to support Airflow 2 in > providers released while Airflow 3 is already out?". 12 months > after Airflow 3.0 is released is if we use the current rule. > > No strong opinions of that but that looks good. > > J > > > > On Thu, Aug 15, 2024 at 1:57 PM Ash Berlin-Taylor <a...@apache.org> wrote: > > > Going forward with Airflow 3 I also think we should re-consider what we > > version/depend upon from providers. > > > > If we move all providers (and likley also utility functions!) out of > > airflow core then the thing that a given provider needs to depend upon is > > the shared library/provider versions and the "Task SDK” that we are > > introducing with AIP-72, and the version of Airflow core doesn’t matter > > anymore. > > > > This doesn’t take away the need to think about how long we continue to > > support Airflow 2 in our providers, but that going forward I think we > need > > to subtly shift how we think about this. The first “key use case” we > > mention in AIP 72 is this: > > > > > > 1. Ensure that this interface reduces the interaction between the code > > running within the Task and the rest of Airflow. This is to address > > unintended ripple effects from core Airflow changes which has caused > > numerous Airflow upgrade issues, because Task (i.e. DAG) code relied on > > Core Airflow abstractions. This has been a common problem pointed out by > > numerous Airflow users including early adopters. This proposal would > enable > > Airflow users to upgrade Airflow system components (Scheduler, et al), > > without impacting DAG user code. > > > > I don’t yet have a clear proposal of what it should be, but I do want to > > surface this. > > > > -ash > > > > > On 15 Aug 2024, at 12:05, Amogh Desai <amoghdesai....@gmail.com> > wrote: > > > > > > Nice points to ponder upon, Jarek. Both topics are essential for > ensuring > > > that users have a clear path forwars. > > > > > >> We currently guarantee that the minimum Airflow version supported by a > > > provider is the release date of the next minor version, plus 12 months. > > > You’re asking if this should be adjusted for Airflow 3, correct? > > > When we say adjust, what are we looking at? Are we thinking of > extending > > > the 12 month period, to lets say 18 months > > > or something else? > > > > > > Few thoughts that come to my mind when I read this email: > > > - A potential adjustment would be to extend the window to 18 months > post > > > the first Airflow 3 release, > > > allowing users more time to transition without feeling > > > rushed. Communicating this timeline clearly > > > to the community will be key here. We should *emphasize* that while > they > > > extra time for the transition, > > > they will eventually need to migrate to Airflow 3 to access the newer > > > features and bug fixes. > > > - We can introduce a policy (and document it well, maybe in > > > > > > https://github.com/apache/airflow/blob/main/PROVIDERS.rst#upgrading-minimum-supported-version-of-airflow > > > ) > > > where providers are guaranteed to work with all Airflow versions > released > > > within 12 months (or 18) > > > of the provider release. After that, users *should* expect that older > > > providers may not be fully compatible > > > with newer Airflow versions, and any issues arising from this won't be > > > treated as bugs. > > > > > > We should also document these rules clearly to provide users with the > > > context and rationale behind these decisions. > > > > > > Thanks & Regards, > > > Amogh Desai > > > > > > > > > On Wed, Aug 14, 2024 at 3:37 PM Jarek Potiuk <ja...@potiuk.com> wrote: > > > > > >>> We currently guarantee that the minimum Airflow version supported by > a > > >> provider is the release date of the next minor version, plus 12 > months. > > >> You’re asking if this should be adjusted for Airflow 3, correct? > > >> > > >> Yes > > >> > > >>> Are you asking for guarantees the other way around, so Airflow > version > > -> > > >> guaranteed provider package min version support? > > >> > > >> Yes. > > >> > > >> On Wed, Aug 14, 2024 at 12:04 PM Bas Harenslak > > <b...@astronomer.io.invalid> > > >> wrote: > > >> > > >>> Sorry I’m a bit lost in the long text. Is my understanding of the 2 > > >>> questions correct? > > >>> > > >>> We currently guarantee that the minimum Airflow version supported by > a > > >>> provider is the release date of the next minor version, plus 12 > months. > > >>> You’re asking if this should be adjusted for Airflow 3, correct? > > >>> Are you asking for guarantees the other way around, so Airflow > version > > -> > > >>> guaranteed provider package min version support? > > >>> > > >>> Bas > > >>> > > >>>> On 14 Aug 2024, at 11:22, Jarek Potiuk <ja...@potiuk.com> wrote: > > >>>> > > >>>> Hello everyone, > > >>>> > > >>>> I have two important topics to discuss regarding Provider <> Airflow > > >>>> version support rules. > > >>>> > > >>>> *1) What is the "min Airflow version" support while we are > > >> transitioning > > >>> to > > >>>> Airflow 3? * > > >>>> > > >>>> So far we had the [1] rule: > > >>>> > > >>>>> The default support timespan for the minimum version of Airflow > > (there > > >>>> could be justified exceptions) is that we increase the minimum > Airflow > > >>>> version to the next MINOR release, when 12 months passed since the > > >> first > > >>>> release for the MINOR version of Airflow. > > >>>> > > >>>>> For example this means that by default we upgrade the minimum > version > > >> of > > >>>> Airflow supported by providers to 2.8.0 in the first Provider's > > release > > >>>> after 18th of December 2024. 18th of December 2023 is the date when > > the > > >>>> first PATCHLEVEL of 2.8 (2.8.0) has been released. > > >>>> > > >>>> The next cutoff time (2.9.0 + 12 months) will be in April 2025, And > > the > > >>>> next one - 2.10.0 - assuming we release 2.10.0 in August - for min > > >>> airflow > > >>>> version = 2.10.0 will be in August 2025. If we have "bridge" 2.11.0 > > >>> release > > >>>> (very likely) - say in December, then min_version = 2.11.0 will be > in > > >>>> December 2025. > > >>>> > > >>>> Assuming that we release Airflow 3.0.0 in March 2025, if we follow > > this > > >>>> rule, we will be able to drop Airflow 2 support in providers in > March > > >>> 2026 > > >>>> - so we give about a year to Airflow 2 users to migrate to Airflow 3 > > - > > >>> if > > >>>> they will want new features and bug-fixes from new providers. Also > it > > >>> means > > >>>> that we have 1.5 year of supporting both Airflow 2 and Airflow 3 in > > >>>> providers (that for example includes the old Templated Field > > >>>> back-compatibility support assuming that we will want to go ahead > with > > >>> the > > >>>> change) > > >>>> > > >>>> That provides (as was the original idea of this rolling min-version > > >>>> support) additional incentive for users to migrate (want new > features > > >>> from > > >>>> providers - migrate to latest Airflow). With Airflow 3 that will be > a > > >>> more > > >>>> difficult decision to migrate, so I wonder if this rule should be > > >>> somewhat > > >>>> adapted. > > >>>> > > >>>> *2) Should we introduce a rule for a "rolling" min version > Provider's > > >>>> support in Airflow 3?* > > >>>> > > >>>> In Airflow 2 we never had a rule describing "how old" providers are > > >> still > > >>>> supported in newer versions of Airflow. For example we never said > you > > >>> need > > >>>> to have at least "2.10.0" version of a google provider. This was > never > > >> a > > >>>> huge problem, because vast majority of users were upgrading > providers > > >>> with > > >>>> Airflow and only occasionally downgraded providers or kept them with > > >> old > > >>>> versions because of some breaking changes.That had never caused a > big > > >>>> problem as far as I know and remember - and we can safely assume > that > > >>>> "really old" providers might not work with "new" airflow even if we > > >> had > > >>> no > > >>>> formal rule for it. > > >>>> > > >>>> However that required some back-compatibility code sometimes - like > > >> this > > >>>> code removal that started breaking our main after removing old > > >> "imports" > > >>>> [2]. This means that some very old SB providers will not work with > the > > >>> new > > >>>> common.sql provider when released (and they also will not work with > > >>> Airflow > > >>>> 3). I think it will never cause a "real" serious issue - but would > be > > >>> great > > >>>> to formalize it to be able to set expectations of our users and have > > >> some > > >>>> ways to respond to potential issues they might raise. > > >>>> > > >>>> Should we formalize it somehow? For example add a rule that > providers > > >>> will > > >>>> be guaranteed to support Airflow versions released up to 12 months > > >> after > > >>>> the provider was released? We could potentially add tests to check > it > > >> in > > >>>> the future (though it would be rather complex and time consuming). > But > > >> I > > >>>> think we should start with the rule first. > > >>>> > > >>>> Example to illustrate it (hypothetical): > > >>>> > > >>>> * Google provider 11.0.4 is released in December 2024 - just before > > >>>> Airflow 2.11.0 > > >>>> * Airflow 3.0 - 3.4 are released between December 2024 and December > > >> 2025 > > >>> - > > >>>> all of them support google provider 2.11.0 (and if not - this is a > bug > > >>> that > > >>>> needs to be fixed) > > >>>> * Airflow 3.5 is released in January 2026 - and if someone wants to > > >>> install > > >>>> Airflow 3.5 and use Google provider 11.0.4 or below - they might, or > > >>> might > > >>>> not work. But if they don't we do not treat it as a bug and we > > >> recommend > > >>>> the users to upgrade to a newer Google provider version. > > >>>> > > >>>> I think we should not make "intentional" changes to break those > > >>> providers, > > >>>> unless we have good reasons, and it's mostly about some weird edge > > >> cases, > > >>>> where people have new Airflow and old providers, but I think it's > > worth > > >>>> introducing and documenting some rules about it. > > >>>> > > >>>> > > >>>> Let me know what you think on both accounts. > > >>>> > > >>>> J. > > >>>> > > >>>> [1] > > >>>> > > >>> > > >> > > > https://github.com/apache/airflow/blob/main/PROVIDERS.rst#upgrading-minimum-supported-version-of-airflow > > >>>> [2] https://github.com/apache/airflow/pull/41461 > > >>> > > >>> > > >> > > > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > > For additional commands, e-mail: dev-h...@airflow.apache.org > > > > >
