Nice points to ponder upon, Jarek. Both topics are essential for ensuring that users have a clear path forwars.
> We currently guarantee that the minimum Airflow version supported by a provider is the release date of the next minor version, plus 12 months. You’re asking if this should be adjusted for Airflow 3, correct? When we say adjust, what are we looking at? Are we thinking of extending the 12 month period, to lets say 18 months or something else? Few thoughts that come to my mind when I read this email: - A potential adjustment would be to extend the window to 18 months post the first Airflow 3 release, allowing users more time to transition without feeling rushed. Communicating this timeline clearly to the community will be key here. We should *emphasize* that while they extra time for the transition, they will eventually need to migrate to Airflow 3 to access the newer features and bug fixes. - We can introduce a policy (and document it well, maybe in https://github.com/apache/airflow/blob/main/PROVIDERS.rst#upgrading-minimum-supported-version-of-airflow ) where providers are guaranteed to work with all Airflow versions released within 12 months (or 18) of the provider release. After that, users *should* expect that older providers may not be fully compatible with newer Airflow versions, and any issues arising from this won't be treated as bugs. We should also document these rules clearly to provide users with the context and rationale behind these decisions. Thanks & Regards, Amogh Desai On Wed, Aug 14, 2024 at 3:37 PM Jarek Potiuk <ja...@potiuk.com> wrote: > > We currently guarantee that the minimum Airflow version supported by a > provider is the release date of the next minor version, plus 12 months. > You’re asking if this should be adjusted for Airflow 3, correct? > > Yes > > > Are you asking for guarantees the other way around, so Airflow version -> > guaranteed provider package min version support? > > Yes. > > On Wed, Aug 14, 2024 at 12:04 PM Bas Harenslak <b...@astronomer.io.invalid> > wrote: > > > Sorry I’m a bit lost in the long text. Is my understanding of the 2 > > questions correct? > > > > We currently guarantee that the minimum Airflow version supported by a > > provider is the release date of the next minor version, plus 12 months. > > You’re asking if this should be adjusted for Airflow 3, correct? > > Are you asking for guarantees the other way around, so Airflow version -> > > guaranteed provider package min version support? > > > > Bas > > > > > On 14 Aug 2024, at 11:22, Jarek Potiuk <ja...@potiuk.com> wrote: > > > > > > Hello everyone, > > > > > > I have two important topics to discuss regarding Provider <> Airflow > > > version support rules. > > > > > > *1) What is the "min Airflow version" support while we are > transitioning > > to > > > Airflow 3? * > > > > > > So far we had the [1] rule: > > > > > >> The default support timespan for the minimum version of Airflow (there > > > could be justified exceptions) is that we increase the minimum Airflow > > > version to the next MINOR release, when 12 months passed since the > first > > > release for the MINOR version of Airflow. > > > > > >> For example this means that by default we upgrade the minimum version > of > > > Airflow supported by providers to 2.8.0 in the first Provider's release > > > after 18th of December 2024. 18th of December 2023 is the date when the > > > first PATCHLEVEL of 2.8 (2.8.0) has been released. > > > > > > The next cutoff time (2.9.0 + 12 months) will be in April 2025, And the > > > next one - 2.10.0 - assuming we release 2.10.0 in August - for min > > airflow > > > version = 2.10.0 will be in August 2025. If we have "bridge" 2.11.0 > > release > > > (very likely) - say in December, then min_version = 2.11.0 will be in > > > December 2025. > > > > > > Assuming that we release Airflow 3.0.0 in March 2025, if we follow this > > > rule, we will be able to drop Airflow 2 support in providers in March > > 2026 > > > - so we give about a year to Airflow 2 users to migrate to Airflow 3 - > > if > > > they will want new features and bug-fixes from new providers. Also it > > means > > > that we have 1.5 year of supporting both Airflow 2 and Airflow 3 in > > > providers (that for example includes the old Templated Field > > > back-compatibility support assuming that we will want to go ahead with > > the > > > change) > > > > > > That provides (as was the original idea of this rolling min-version > > > support) additional incentive for users to migrate (want new features > > from > > > providers - migrate to latest Airflow). With Airflow 3 that will be a > > more > > > difficult decision to migrate, so I wonder if this rule should be > > somewhat > > > adapted. > > > > > > *2) Should we introduce a rule for a "rolling" min version Provider's > > > support in Airflow 3?* > > > > > > In Airflow 2 we never had a rule describing "how old" providers are > still > > > supported in newer versions of Airflow. For example we never said you > > need > > > to have at least "2.10.0" version of a google provider. This was never > a > > > huge problem, because vast majority of users were upgrading providers > > with > > > Airflow and only occasionally downgraded providers or kept them with > old > > > versions because of some breaking changes.That had never caused a big > > > problem as far as I know and remember - and we can safely assume that > > > "really old" providers might not work with "new" airflow even if we > had > > no > > > formal rule for it. > > > > > > However that required some back-compatibility code sometimes - like > this > > > code removal that started breaking our main after removing old > "imports" > > > [2]. This means that some very old SB providers will not work with the > > new > > > common.sql provider when released (and they also will not work with > > Airflow > > > 3). I think it will never cause a "real" serious issue - but would be > > great > > > to formalize it to be able to set expectations of our users and have > some > > > ways to respond to potential issues they might raise. > > > > > > Should we formalize it somehow? For example add a rule that providers > > will > > > be guaranteed to support Airflow versions released up to 12 months > after > > > the provider was released? We could potentially add tests to check it > in > > > the future (though it would be rather complex and time consuming). But > I > > > think we should start with the rule first. > > > > > > Example to illustrate it (hypothetical): > > > > > > * Google provider 11.0.4 is released in December 2024 - just before > > > Airflow 2.11.0 > > > * Airflow 3.0 - 3.4 are released between December 2024 and December > 2025 > > - > > > all of them support google provider 2.11.0 (and if not - this is a bug > > that > > > needs to be fixed) > > > * Airflow 3.5 is released in January 2026 - and if someone wants to > > install > > > Airflow 3.5 and use Google provider 11.0.4 or below - they might, or > > might > > > not work. But if they don't we do not treat it as a bug and we > recommend > > > the users to upgrade to a newer Google provider version. > > > > > > I think we should not make "intentional" changes to break those > > providers, > > > unless we have good reasons, and it's mostly about some weird edge > cases, > > > where people have new Airflow and old providers, but I think it's worth > > > introducing and documenting some rules about it. > > > > > > > > > Let me know what you think on both accounts. > > > > > > J. > > > > > > [1] > > > > > > https://github.com/apache/airflow/blob/main/PROVIDERS.rst#upgrading-minimum-supported-version-of-airflow > > > [2] https://github.com/apache/airflow/pull/41461 > > > > >
