I think it's OK to conclude the release strategy for clients: So the first two digits of the client's release number will be borrowed from the first two digits of Airflow's release. The last and third digit would be independent of the Airflow release and will denote any patches done on the client side.
For example, the first release of the Python client would be *2.0.0* which would be based on *Airflow 2.0.1*. On Mon, Apr 5, 2021 at 5:46 PM Sumit Maheshwari <sumeet.ma...@gmail.com> wrote: > I think I'm fine with the approach you suggested Jarek. This approach > mitigates the need to release clients with each patch release of Airflow, > and that does make sense. > > I hope that others are fine as well with the approach, where we've to > release clients with every major or minor release of Airflow. > > On Mon, Apr 5, 2021 at 5:31 PM Jarek Potiuk <ja...@potiuk.com> wrote: > >> Hmm. I think it's a great direction, but 4 digits is a bit too much :) >> keeping three-digit SemVer is good for clarity/tools. And itroducing 4th >> digit is not really needed i think. >> >> Your proposal Sumit made me think, and I asked myself a few questions: >> >> 1) will we ever release API client new feature without releasing that >> feature in airflow ? I think not. The API client should provide API calls >> for all the endpoints available and any change in those endpoints or adding >> new one - by definition - should mean a bump in airflow minor version at >> least. >> >> 2) is it likely we release a bugfix to a single API client only without >> releasing Airflow new bugfix? I think so. We might upgrade generator for >> one client only to fix a particular problem we find for that language. >> >> 3) is it likely we release airflow bugfix without the need of releasing >> any of the clients? Yeah it's very likely the client will work without any >> changes with the new Airflow bugfix version. >> >> ... thinking .... >> >> So, how about a hybrid solution where we also include the minor version >> of Airflow in the client version >> >> Say airflow 2.1.x will work only with client 2.1.n but x and n are >> independent ? >> >> Example: >> >> We release 2.1.0 of Airflow and 2.1.0 of the Python and Java clients. >> >> Then we release 2.1.1, 2.1.2, 2.1.3 of Airflow. No release of clients. >> >> Then we find a bug in the Python generator, upgrade it and fix it in >> 2.1.1 version of the Python client. But the Java client might still be at >> 2.1.0 as the generator for Java worked fine. >> >> At this point everyone who uses 2.1 airflow SHOULD upgrade to 2.1.3 of >> Airflow and 2.1.1 of the Python client and 2.1.0 of the Java client. If >> anyone uses any other combination, they should first upgrade to those >> latest 2.1 ones before 'complaining' that there are some problems. >> >> Next - whenever we release 2.2.0 of Airflow we also release 2.2.0 of all >> clients. And our client check if Airflow is the same major/minor version >> and fails if it is not. So you cannot use 2.1.1 Python client with 2.0 >> Airflow - you must switch to 2.2.0. >> >> I think that might be the nicest approach for both users and maintainers. >> In this case we do not even have to maintain the 'compatibility' matrix i >> mentioned before. And we keep 100% Semver semantics. And we can add >> automated tests for all the 'last X.Y client + airflow' combination as >> there is always only one 'blessed' combination airflow + clients that is >> 'supposed' to work. >> >> WDYT? >> >> J. >> >> >> >> >> pon., 5 kwi 2021, 08:09 użytkownik Sumit Maheshwari < >> sumeet.ma...@gmail.com> napisał: >> >>> Thanks, everyone for the inputs. >>> >>> I think it's clear that we want to go with the hybrid way of versioning. >>> Given that we can automate the client releases easily, how if we release a >>> client automatically with each Airflow release, and go for a four numbered >>> release, where the first three would be the Airflow's release number and >>> the fourth one would be specific to the client, something like this >>> >>> *Airflow 2.0.1 <=> Client 2.0.1.0* -- Initial release >>> *Airflow 2.0.1 <=> Client 2.0.1.1* -- Some fix done in client (treating >>> major or minor same for client, ideally there won't be any major patches in >>> client) >>> >>> The upside of this process is that we can automate it & the client >>> version will be in sync with the Airflow version always. The downside that >>> we would be releasing more packages than required. >>> >>> About Vikram's question of not supporting any APIs in the client, AFAIK >>> there are none. >>> >>> On Mon, Apr 5, 2021 at 2:51 AM Vikram Koka <vik...@astronomer.io.invalid> >>> wrote: >>> >>>> Sumit, >>>> Thank you for bringing up this discussion. >>>> >>>> Reading through the thread, I am definitely far more comfortable with >>>> the hybrid approach proposed by Kaxil, than the first approach, primarily >>>> because of the same concerns articulated by Jarek regarding end user >>>> experience. Prior to that, I was leaning towards the second approach. >>>> >>>> Having said that, I think we need to be sure to follow certain >>>> guidelines with Core Airflow to make sure this approach works consistently, >>>> with a minimum of documentation needed and documentation to be read by >>>> Airflow users. Since there are generated clients, I am specifically >>>> referring to changes within Airflow regarding the API. Most importantly, I >>>> think we need to have any API additions, whether a new endpoint or an >>>> enhancement to an existing endpoint, only be available as part of a >>>> "feature release", rather than a "patch release". And that, this should >>>> trigger an update to the generated clients, triggering a new version of the >>>> same. This may be obvious, because we agreed on following semantic >>>> versioning strictly, but I thought it was worth reiterating. >>>> >>>> Having said that, I think it is worth asking the question if that is >>>> actually true, or if there are any APIs which are not (or should not be) >>>> supported in the clients? >>>> I didn't think so, but wanted to make sure. >>>> >>>> Best regards, >>>> Vikram >>>> >>>> >>>> On Sat, Apr 3, 2021 at 4:11 AM Jarek Potiuk <ja...@potiuk.com> wrote: >>>> >>>>> I also like the hybrid approach proposed by Kaxil. It is not as >>>>> simplenfor dependency management as 1-1 version, but IMHO it is >>>>> simple-enough. As long as we strictly follow SemVer, a number of questions >>>>> i listed above from the user, are answered: >>>>> >>>>> >>>>> 1) Will my client be compatible with the version of Airflow ? >>>>> >>>>> Easy: 2.x.y will be compatible with 2.n.m of Airflow and so on.. Since >>>>> we will not have breaking changes, the 'broad compatibility' level should >>>>> be fine (with some feature -level differences but no general compatibility >>>>> problems) >>>>> >>>>> 2) Will feature (API endpoint) x available in this version of client >>>>> be working with the version of Airflow I have? >>>>> >>>>> We have to make sure that features are documented in the APIs at the >>>>> level of '@since' if we keep it updated the form 'This feature is >>>>> available >>>>> in 'Airflow 2.3.*' and 'API 2.1.*' - we are covered. Since there will be >>>>> no breaking changes in neither API nor Airflow - this will be enough. >>>>> >>>>> >>>>> 3) Which version of Airflow should I install in order to use feature X >>>>> that I have in this new version of client ? >>>>> >>>>> I think this might be possible to generate such compatibility matrix >>>>> for all new API features added since Airflow 2.0 - it might be >>>>> automatically generated from the API specification in this case. If we >>>>> parse the API definition and have the right 'since' definition for both >>>>> Airflow and API, it should be rather easy to generate such table: >>>>> >>>>> Feature | Min API version | Min Airflow version >>>>> MoonShot feature X | 2.3 | 2.2 >>>>> >>>>> Having such compatibility matric would, I think, be a great help for >>>>> the users i think when they would plan their Airflow/API releases >>>>> >>>>> >>>>> J. >>>>> >>>>> sob., 3 kwi 2021, 12:06 użytkownik .... <dzaku...@gmail.com> napisał: >>>>> >>>>>> +1 for Kaxil proposal >>>>>> >>>>>> I think, attempting to assign a specific client version to the library >>>>>> application makes no sense in creating an API that is based on a >>>>>> strict specification. >>>>>> >>>>>> One of the goals of this API was to make it easier to manage multiple >>>>>> instances, and it's not often that you have an identically homogeneous >>>>>> environment. It's natural to have different versions of Airflow in one >>>>>> organization. If we want users to use a specific version of the >>>>>> library on the client-side, it will probably not be possible in many >>>>>> cases. >>>>>> >>>>>> pt., 2 kwi 2021 o 23:35 Kaxil Naik <kaxiln...@gmail.com> napisał(a): >>>>>> > >>>>>> > Thanks, Sumit, for bringing this up. >>>>>> > >>>>>> > re: versioning for Python client: >>>>>> > >>>>>> > I favour a bit of a hybrid of 1 and 2. Initially, I favored just >>>>>> (1) when we talked but after seeing how Elastic-Search does it I liked >>>>>> their versioning strategy. >>>>>> > How their python-client is compatible with ES itself is documented >>>>>> at https://elasticsearch-py.readthedocs.io/en/v7.12.0/#compatibility. >>>>>> > >>>>>> > Since Airflow from 2.0 strictly follows SemVer we can continue with >>>>>> the assumption that we (Airflow) won't break backwards-compatibility (to >>>>>> the best of our ability). >>>>>> > So we can have M.x.y as our versioning for our client where M is >>>>>> the Major Version of the client that matches the Major version of >>>>>> Airflow; >>>>>> x is the minor version for the client & y is the patch release. Here x & >>>>>> y >>>>>> are independent of Airflow version. Our goal and message to user would be >>>>>> that "You have to use a version of client library matching the major >>>>>> version of Airflow" >>>>>> > >>>>>> > re: the release of different client libraries: >>>>>> > >>>>>> > Client for each language should be versioned & released separately. >>>>>> > >>>>>> > Regards, >>>>>> > Kaxil >>>>>> > >>>>>> > On Wed, Mar 31, 2021 at 7:01 PM Jarek Potiuk <ja...@potiuk.com> >>>>>> wrote: >>>>>> >> >>>>>> >> I think there are more questions we need to answer when it comes >>>>>> to different versions. If we are going to different versioning for agent >>>>>> and Airflow, we will have to accept the fact that over time people will >>>>>> have a "matrix of versions" - for both Airflow and the client. And this >>>>>> introduces "dependency mess" if we do not handle it well. >>>>>> >> >>>>>> >> I think in this we should give the user much clearer indication on >>>>>> which version of Airflow the client should work on, and (more important) >>>>>> which version it does NOT work on and which features will NOT be >>>>>> available. >>>>>> Both cases are possible. >>>>>> >> >>>>>> >> There should be some way of knowing for the user, (in case there >>>>>> is a backwards incompatibility or in case some features are not >>>>>> available) >>>>>> that a given version of the client does not work with the version of >>>>>> Airflow that the user has. >>>>>> >> >>>>>> >> I think users looking at the documentation should be able to be >>>>>> able to figure out: >>>>>> >> >>>>>> >> 1) Will my client be compatible with the version of Airflow ? >>>>>> >> 2) Will feature (API endpoint) x available in this version of >>>>>> client be working with the version of Airflow I have? >>>>>> >> 3) Which version of Airflow should I install in order to use >>>>>> feature X that I have in this new version of client ? >>>>>> >> >>>>>> >> Similarly proper error messages should be displayed if there is >>>>>> incompatibility or when a feature X is not available when you try to use >>>>>> a >>>>>> version of client that supports it with Airflow version of Agent that >>>>>> does >>>>>> not have it. >>>>>> >> >>>>>> >> So I'd say - separate versioning is fine as long as we handle the >>>>>> above. >>>>>> >> >>>>>> >> >>>>>> >> Saying all that - there is a beautiful "maintenance" simplicity if >>>>>> we actually synchronize the version and we say "Always use SAME version >>>>>> of >>>>>> client as Airflow" (and we fail the client if the versions is different). >>>>>> This is way more simple in communicating to the users, and with most >>>>>> deployments, I think upgrading airflow and clients at the same time >>>>>> should >>>>>> be no problem. >>>>>> >> >>>>>> >> I think over time as the matrix of potential Airflow/Client >>>>>> combinations will grow this might mean bigger number of issues that we >>>>>> have >>>>>> to handle ("Why ma client does not do this and this (and of course it >>>>>> will >>>>>> not have neither client nor airflow version specified and we will have to >>>>>> ask for clarification). >>>>>> >> >>>>>> >> So as I see it - maybe release-wise it's a little redundant and >>>>>> you'd have to release a bugfix version of Airflow if you find a critical >>>>>> problem in Client (which should be rare as agent is mostly automatically >>>>>> generated), but synchronizing versions is very tempting from the >>>>>> "supportability" point of view. >>>>>> >> >>>>>> >> Those are my 2 cents, I was not involved too much in the client >>>>>> part, so I do not feel like I have a strong voice in there, but if I >>>>>> would >>>>>> vote between the two - I'd choose synchronizing the releases. >>>>>> >> >>>>>> >> BTW. In providers we have a very different story - we keep >>>>>> dependency to airflow via PIP (which we can set it, if we find a provider >>>>>> stops being compatible with a version of Airflow). Also we automatically >>>>>> test in CI if the provider installs and imports on Airflow 2.0.0 to >>>>>> verify >>>>>> that it is backwards compatible (at least to the level of importability). >>>>>> This has already saved us from "accidental" incompatibilities (wit yaml >>>>>> package import change recently). So there, separate versioning makes much >>>>>> more sense - also because the version of the provider relates exclusively >>>>>> to the feature that the provider version has, not Airflow itself). I >>>>>> think >>>>>> this is quite a different story from the client, which is very closely >>>>>> coupled with Airflow. >>>>>> >> >>>>>> >> J. >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> On Wed, Mar 31, 2021 at 11:08 AM Deng Xiaodong < >>>>>> xd.den...@gmail.com> wrote: >>>>>> >>> >>>>>> >>> Yep, if we can, I would prefer to explicitly mention something >>>>>> like 2.0.X :) >>>>>> >>> >>>>>> >>> On Wed, Mar 31, 2021 at 10:54 AM Sumit Maheshwari < >>>>>> sumeet.ma...@gmail.com> wrote: >>>>>> >>>> >>>>>> >>>> Yeah totally agrees with you XD about the range vs the fixed >>>>>> version of the main software. And with Airflow 2 dot & beyond, we would >>>>>> be >>>>>> following the semantic versioning more judiciously, so with that can we >>>>>> assume that users would be cognizant of the related Airflow version >>>>>> implicitly, or do you think we should mention something like 2.0.X in the >>>>>> changelog explicitly? >>>>>> >>>> >>>>>> >>>> On Wed, Mar 31, 2021 at 1:38 PM Deng Xiaodong < >>>>>> xd.den...@gmail.com> wrote: >>>>>> >>>>> >>>>>> >>>>> Thanks, Sumit, for bringing this up. >>>>>> >>>>> >>>>>> >>>>> - Regarding versioning for Python client: I favour the 1st way >>>>>> as well. The only minor comment I have is: instead of "mention the >>>>>> release >>>>>> version of the main software", possibly what users would expect is a >>>>>> "range >>>>>> of compatible versions of main software". If we follow semver for Airflow >>>>>> itself relatively strictly, this should not be hard. >>>>>> >>>>> >>>>>> >>>>> - Regarding the release of different client libraries: I agree >>>>>> with you. They should be versioned & released separately. >>>>>> >>>>> >>>>>> >>>>> >>>>>> >>>>> Regards, >>>>>> >>>>> XD >>>>>> >>>>> >>>>>> >>>>> On Wed, Mar 31, 2021 at 9:21 AM Sumit Maheshwari < >>>>>> sumeet.ma...@gmail.com> wrote: >>>>>> >>>>>> >>>>>> >>>>>> Thanks QP for your feedback. I think the latest release of K8s >>>>>> client (v17.x.y) is moving in another direction. >>>>>> >>>>>> >>>>>> >>>>>> On Wed, Mar 31, 2021 at 12:13 AM QP Hou <q...@scribd.com.invalid> >>>>>> wrote: >>>>>> >>>>>>> >>>>>> >>>>>>> Thanks Sumit for kicking off this discussion! >>>>>> >>>>>>> >>>>>> >>>>>>> I am in favor of approach #1 as well. Airflow software >>>>>> version can >>>>>> >>>>>>> change due to none API related changes, but the client should >>>>>> only >>>>>> >>>>>>> care about the public API contract. I believe this is also >>>>>> the same >>>>>> >>>>>>> approach that the k8s python client took as well? For >>>>>> example, the k8s >>>>>> >>>>>>> python client version 12.0.1 targets k8s api version 1.16.15. >>>>>> >>>>>>> >>>>>> >>>>>>> I am also with you that clients for different languages >>>>>> should have >>>>>> >>>>>>> their own release cycles. Other than testing burdens, every >>>>>> client >>>>>> >>>>>>> could introduce their own bug fixes and breaking changes >>>>>> without >>>>>> >>>>>>> having to trigger releases for all other clients. It would be >>>>>> a waste >>>>>> >>>>>>> of work for us to bump the python client version just for a >>>>>> bug fix in >>>>>> >>>>>>> the go client. >>>>>> >>>>>>> >>>>>> >>>>>>> Thanks, >>>>>> >>>>>>> QP >>>>>> >>>>>>> >>>>>> >>>>>>> On Tue, Mar 30, 2021 at 1:56 AM Sumit Maheshwari < >>>>>> msu...@apache.org> wrote: >>>>>> >>>>>>> > >>>>>> >>>>>>> > Hello Airflow dev community >>>>>> >>>>>>> > >>>>>> >>>>>>> > I'm writing this mail in regards to seek feedback on the >>>>>> first release of Apache Airflow's Python client. This is an icebreaker >>>>>> email only and we will follow the Apache process of releasing the version >>>>>> later on. As of now, I've raised a small PR to add the missing files in >>>>>> the >>>>>> repo, which are required to do the release of the client lib. >>>>>> >>>>>>> > >>>>>> >>>>>>> > The main question I've in mind about the versioning. I see >>>>>> that there are mainly two theories related to maintain the release >>>>>> versions >>>>>> of clients: >>>>>> >>>>>>> > >>>>>> >>>>>>> > The first is about following the semantic release of the >>>>>> clients on its own. The release version of the client doesn't include the >>>>>> version of the software itself. Something like we are doing with >>>>>> Airflow's >>>>>> provider packages. However, in each release, we mention the release >>>>>> version >>>>>> of the main software (Airflow in this case). This is the simple most way >>>>>> of >>>>>> maintaining releases and followed majorly. >>>>>> >>>>>>> > The second way is to include the software's version as well >>>>>> in its client's versioning, something like it's been done in the K8s >>>>>> Python >>>>>> client. In this approach, the client version provides more implicit >>>>>> knowledge about the compatible software, however it creates an extra >>>>>> burden >>>>>> of releasing the client with every minor release of the software, >>>>>> otherwise >>>>>> the versioning would fall apart. Also, there is a good chance that most >>>>>> of >>>>>> these minor or even a major release has no changes in the APIs, but the >>>>>> client would be needed to re-released nevertheless, just to keep the >>>>>> versioning in check. >>>>>> >>>>>>> > >>>>>> >>>>>>> > Personally, I'm in favour of the 1st way, as it's simple to >>>>>> manage and doesn't tightly couple client with its software. >>>>>> >>>>>>> > >>>>>> >>>>>>> > Another unrelated question I have is about the release of >>>>>> different client libraries. As of now, Airflow has Python and Go clients, >>>>>> so should we try to release them together or they should follow their own >>>>>> paths? In general, I'm against batching them together, cause the person >>>>>> who >>>>>> is releasing them may not be able to test all the different clients in >>>>>> the >>>>>> future. >>>>>> >>>>>>> > >>>>>> >>>>>>> > Let me know your thoughts on these 2 points. >>>>>> >>>>>>> > >>>>>> >>>>>>> > Thanks, >>>>>> >>>>>>> > Sumit Maheshwari >>>>>> >>>>>>> > PMC Apache Airflow >>>>>> >> >>>>>> >> >>>>>> >> >>>>>> >> -- >>>>>> >> +48 660 796 129 >>>>>> >>>>>>
