Hello everyone, I wanted to separate one topic into a separate side discussion. Maybe not useful with Dask discussion but might be something that we might use for the future.
> This means that this change might potentially break existing > setups for some users when they upgrade Airflow version, which contradicts > the Airflow deprecation policy also there was another commend from Daniel: > As a user, your concerns and expectations regarding semver are about > essentially how the code works, e.g. are you going to have to refactor all > of your 500 dags. In other words the API.. But to me this is a lower > evel thing that doesn't seem should strictly be forbidden. It's like... > when we change the dependencies in airflow, inevitably that will break > someone's cluster if they blindly deploy to prod. But this is not a > prudent thing to do. So I think it's not an unreasonable expectation that a > user verify that their cluster will spin up. It's something you can verify > without running any dags at all (i.e. if you set dask executor and you > start airflow, i think airflow will not start). And then users can rely on > semver about changing behavior, which is not something you necessarily can > tell without running dags. I tend to agree with Daniel that technically speaking SemVer does not tell anything about breaking upgrades. It does not really say "you might continue upgrades the same way as before and things will work" but "when you upgrade <CORRECTLY> things will work the way they worked". This is the promise of https://semver.org/ . I re-read it again and could not find any promise that "your process of upgrade is covered by SemVer promises". Interestingly I also found out (I did not know) that our rule "breaking changes in our dependencies is not breaking our compatibility" is explained there in FAQ: > That would be considered compatible since it does not affect > the public API. Software that explicitly depends on the same > dependencies as your package should have their own > dependency specifications and the author will > notice any conflicts. Determining whether the change > is a patch level or minor level modification depends on > whether you updated your dependencies in order to fix a > bug or introduce new functionality. We would usually expect > additional code for the latter instance, in which case > it’s obviously a minor level increment. We do not make a promise that our upgrade process should look the same between versions - it largely is the same, but IMHO technically we can throw-in a new step or add a note in the upgrade process telling the user to do things differently to upgrade. IMHO it is perfectly fine if in the release notes/significant section we tell the user "If you used XX " you need to make sure you install airflow with [xx] extra. Yes. It might break some installation scripts and CI workflows, but upgrading to new version of Airflow should ALWAYS be a deliberate action - with migration of the DB and reading release notes before doing the upgrade - and failing such automated workflow of upgrade is at most a signal for whoever maintains it to make some fixes to their process. I also think that guaranteed upgrades work is rather difficult - ys we have our "reproducible" installation with constraints (which is guaranteed to work), but there are many other ways and tools people might install airflow - for example we have poetry or bazel installation - and for example we already broke basel installation by adding back limits on pre-installed providers, because basel expects dependency graph to be DAG and for us (and pip/PyPI) having circles in dependency graph are quite legitimate use case - see https://github.com/bazelbuild/rules_python/pull/1166 - so technically speaking we are not really ablet to guarantee much when it comes "upgradeability". But I would love to hear what others think about it. I would love it if we get to a consensus on that and write it down in the "Public Airflow Interface/SamVer" promise of ours so that we can communicate it better rather than allow different people to have different assumptions here. I think we have two choices if we were to "disambiguate what Semver + upgrades mean": * either we agree that breaking upgrade flow is occasionally OK and release notes should explain it * or we document that the users can 100% rely on it working always seamlessly when they upgrade - but I think this one is going to be difficult to describe in the way that will include all the exclusions such as the "basel" case. J. --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org For additional commands, e-mail: dev-h...@airflow.apache.org
