Thanks, everyone for the inputs. I think it's clear that we want to go with the hybrid way of versioning. Given that we can automate the client releases easily, how if we release a client automatically with each Airflow release, and go for a four numbered release, where the first three would be the Airflow's release number and the fourth one would be specific to the client, something like this
*Airflow 2.0.1 <=> Client 2.0.1.0* -- Initial release *Airflow 2.0.1 <=> Client 2.0.1.1* -- Some fix done in client (treating major or minor same for client, ideally there won't be any major patches in client) The upside of this process is that we can automate it & the client version will be in sync with the Airflow version always. The downside that we would be releasing more packages than required. About Vikram's question of not supporting any APIs in the client, AFAIK there are none. On Mon, Apr 5, 2021 at 2:51 AM Vikram Koka <vik...@astronomer.io.invalid> wrote: > Sumit, > Thank you for bringing up this discussion. > > Reading through the thread, I am definitely far more comfortable with the > hybrid approach proposed by Kaxil, than the first approach, primarily > because of the same concerns articulated by Jarek regarding end user > experience. Prior to that, I was leaning towards the second approach. > > Having said that, I think we need to be sure to follow certain guidelines > with Core Airflow to make sure this approach works consistently, with a > minimum of documentation needed and documentation to be read by Airflow > users. Since there are generated clients, I am specifically referring to > changes within Airflow regarding the API. Most importantly, I think we need > to have any API additions, whether a new endpoint or an enhancement to an > existing endpoint, only be available as part of a "feature release", rather > than a "patch release". And that, this should trigger an update to the > generated clients, triggering a new version of the same. This may be > obvious, because we agreed on following semantic versioning strictly, but I > thought it was worth reiterating. > > Having said that, I think it is worth asking the question if that is > actually true, or if there are any APIs which are not (or should not be) > supported in the clients? > I didn't think so, but wanted to make sure. > > Best regards, > Vikram > > > On Sat, Apr 3, 2021 at 4:11 AM Jarek Potiuk <ja...@potiuk.com> wrote: > >> I also like the hybrid approach proposed by Kaxil. It is not as >> simplenfor dependency management as 1-1 version, but IMHO it is >> simple-enough. As long as we strictly follow SemVer, a number of questions >> i listed above from the user, are answered: >> >> >> 1) Will my client be compatible with the version of Airflow ? >> >> Easy: 2.x.y will be compatible with 2.n.m of Airflow and so on.. Since we >> will not have breaking changes, the 'broad compatibility' level should be >> fine (with some feature -level differences but no general compatibility >> problems) >> >> 2) Will feature (API endpoint) x available in this version of client be >> working with the version of Airflow I have? >> >> We have to make sure that features are documented in the APIs at the >> level of '@since' if we keep it updated the form 'This feature is available >> in 'Airflow 2.3.*' and 'API 2.1.*' - we are covered. Since there will be >> no breaking changes in neither API nor Airflow - this will be enough. >> >> >> 3) Which version of Airflow should I install in order to use feature X >> that I have in this new version of client ? >> >> I think this might be possible to generate such compatibility matrix for >> all new API features added since Airflow 2.0 - it might be automatically >> generated from the API specification in this case. If we parse the API >> definition and have the right 'since' definition for both Airflow and API, >> it should be rather easy to generate such table: >> >> Feature | Min API version | Min Airflow version >> MoonShot feature X | 2.3 | 2.2 >> >> Having such compatibility matric would, I think, be a great help for the >> users i think when they would plan their Airflow/API releases >> >> >> J. >> >> sob., 3 kwi 2021, 12:06 użytkownik .... <dzaku...@gmail.com> napisał: >> >>> +1 for Kaxil proposal >>> >>> I think, attempting to assign a specific client version to the library >>> application makes no sense in creating an API that is based on a >>> strict specification. >>> >>> One of the goals of this API was to make it easier to manage multiple >>> instances, and it's not often that you have an identically homogeneous >>> environment. It's natural to have different versions of Airflow in one >>> organization. If we want users to use a specific version of the >>> library on the client-side, it will probably not be possible in many >>> cases. >>> >>> pt., 2 kwi 2021 o 23:35 Kaxil Naik <kaxiln...@gmail.com> napisał(a): >>> > >>> > Thanks, Sumit, for bringing this up. >>> > >>> > re: versioning for Python client: >>> > >>> > I favour a bit of a hybrid of 1 and 2. Initially, I favored just (1) >>> when we talked but after seeing how Elastic-Search does it I liked their >>> versioning strategy. >>> > How their python-client is compatible with ES itself is documented at >>> https://elasticsearch-py.readthedocs.io/en/v7.12.0/#compatibility. >>> > >>> > Since Airflow from 2.0 strictly follows SemVer we can continue with >>> the assumption that we (Airflow) won't break backwards-compatibility (to >>> the best of our ability). >>> > So we can have M.x.y as our versioning for our client where M is the >>> Major Version of the client that matches the Major version of Airflow; x is >>> the minor version for the client & y is the patch release. Here x & y are >>> independent of Airflow version. Our goal and message to user would be that >>> "You have to use a version of client library matching the major version of >>> Airflow" >>> > >>> > re: the release of different client libraries: >>> > >>> > Client for each language should be versioned & released separately. >>> > >>> > Regards, >>> > Kaxil >>> > >>> > On Wed, Mar 31, 2021 at 7:01 PM Jarek Potiuk <ja...@potiuk.com> wrote: >>> >> >>> >> I think there are more questions we need to answer when it comes to >>> different versions. If we are going to different versioning for agent and >>> Airflow, we will have to accept the fact that over time people will have a >>> "matrix of versions" - for both Airflow and the client. And this introduces >>> "dependency mess" if we do not handle it well. >>> >> >>> >> I think in this we should give the user much clearer indication on >>> which version of Airflow the client should work on, and (more important) >>> which version it does NOT work on and which features will NOT be available. >>> Both cases are possible. >>> >> >>> >> There should be some way of knowing for the user, (in case there is a >>> backwards incompatibility or in case some features are not available) that >>> a given version of the client does not work with the version of Airflow >>> that the user has. >>> >> >>> >> I think users looking at the documentation should be able to be able >>> to figure out: >>> >> >>> >> 1) Will my client be compatible with the version of Airflow ? >>> >> 2) Will feature (API endpoint) x available in this version of client >>> be working with the version of Airflow I have? >>> >> 3) Which version of Airflow should I install in order to use feature >>> X that I have in this new version of client ? >>> >> >>> >> Similarly proper error messages should be displayed if there is >>> incompatibility or when a feature X is not available when you try to use a >>> version of client that supports it with Airflow version of Agent that does >>> not have it. >>> >> >>> >> So I'd say - separate versioning is fine as long as we handle the >>> above. >>> >> >>> >> >>> >> Saying all that - there is a beautiful "maintenance" simplicity if we >>> actually synchronize the version and we say "Always use SAME version of >>> client as Airflow" (and we fail the client if the versions is different). >>> This is way more simple in communicating to the users, and with most >>> deployments, I think upgrading airflow and clients at the same time should >>> be no problem. >>> >> >>> >> I think over time as the matrix of potential Airflow/Client >>> combinations will grow this might mean bigger number of issues that we have >>> to handle ("Why ma client does not do this and this (and of course it will >>> not have neither client nor airflow version specified and we will have to >>> ask for clarification). >>> >> >>> >> So as I see it - maybe release-wise it's a little redundant and you'd >>> have to release a bugfix version of Airflow if you find a critical problem >>> in Client (which should be rare as agent is mostly automatically >>> generated), but synchronizing versions is very tempting from the >>> "supportability" point of view. >>> >> >>> >> Those are my 2 cents, I was not involved too much in the client part, >>> so I do not feel like I have a strong voice in there, but if I would vote >>> between the two - I'd choose synchronizing the releases. >>> >> >>> >> BTW. In providers we have a very different story - we keep dependency >>> to airflow via PIP (which we can set it, if we find a provider stops being >>> compatible with a version of Airflow). Also we automatically test in CI if >>> the provider installs and imports on Airflow 2.0.0 to verify that it is >>> backwards compatible (at least to the level of importability). This has >>> already saved us from "accidental" incompatibilities (wit yaml package >>> import change recently). So there, separate versioning makes much more >>> sense - also because the version of the provider relates exclusively to the >>> feature that the provider version has, not Airflow itself). I think this is >>> quite a different story from the client, which is very closely coupled with >>> Airflow. >>> >> >>> >> J. >>> >> >>> >> >>> >> >>> >> On Wed, Mar 31, 2021 at 11:08 AM Deng Xiaodong <xd.den...@gmail.com> >>> wrote: >>> >>> >>> >>> Yep, if we can, I would prefer to explicitly mention something like >>> 2.0.X :) >>> >>> >>> >>> On Wed, Mar 31, 2021 at 10:54 AM Sumit Maheshwari < >>> sumeet.ma...@gmail.com> wrote: >>> >>>> >>> >>>> Yeah totally agrees with you XD about the range vs the fixed >>> version of the main software. And with Airflow 2 dot & beyond, we would be >>> following the semantic versioning more judiciously, so with that can we >>> assume that users would be cognizant of the related Airflow version >>> implicitly, or do you think we should mention something like 2.0.X in the >>> changelog explicitly? >>> >>>> >>> >>>> On Wed, Mar 31, 2021 at 1:38 PM Deng Xiaodong <xd.den...@gmail.com> >>> wrote: >>> >>>>> >>> >>>>> Thanks, Sumit, for bringing this up. >>> >>>>> >>> >>>>> - Regarding versioning for Python client: I favour the 1st way as >>> well. The only minor comment I have is: instead of "mention the release >>> version of the main software", possibly what users would expect is a "range >>> of compatible versions of main software". If we follow semver for Airflow >>> itself relatively strictly, this should not be hard. >>> >>>>> >>> >>>>> - Regarding the release of different client libraries: I agree >>> with you. They should be versioned & released separately. >>> >>>>> >>> >>>>> >>> >>>>> Regards, >>> >>>>> XD >>> >>>>> >>> >>>>> On Wed, Mar 31, 2021 at 9:21 AM Sumit Maheshwari < >>> sumeet.ma...@gmail.com> wrote: >>> >>>>>> >>> >>>>>> Thanks QP for your feedback. I think the latest release of K8s >>> client (v17.x.y) is moving in another direction. >>> >>>>>> >>> >>>>>> On Wed, Mar 31, 2021 at 12:13 AM QP Hou <q...@scribd.com.invalid> >>> wrote: >>> >>>>>>> >>> >>>>>>> Thanks Sumit for kicking off this discussion! >>> >>>>>>> >>> >>>>>>> I am in favor of approach #1 as well. Airflow software version >>> can >>> >>>>>>> change due to none API related changes, but the client should >>> only >>> >>>>>>> care about the public API contract. I believe this is also the >>> same >>> >>>>>>> approach that the k8s python client took as well? For example, >>> the k8s >>> >>>>>>> python client version 12.0.1 targets k8s api version 1.16.15. >>> >>>>>>> >>> >>>>>>> I am also with you that clients for different languages should >>> have >>> >>>>>>> their own release cycles. Other than testing burdens, every >>> client >>> >>>>>>> could introduce their own bug fixes and breaking changes without >>> >>>>>>> having to trigger releases for all other clients. It would be a >>> waste >>> >>>>>>> of work for us to bump the python client version just for a bug >>> fix in >>> >>>>>>> the go client. >>> >>>>>>> >>> >>>>>>> Thanks, >>> >>>>>>> QP >>> >>>>>>> >>> >>>>>>> On Tue, Mar 30, 2021 at 1:56 AM Sumit Maheshwari < >>> msu...@apache.org> wrote: >>> >>>>>>> > >>> >>>>>>> > Hello Airflow dev community >>> >>>>>>> > >>> >>>>>>> > I'm writing this mail in regards to seek feedback on the first >>> release of Apache Airflow's Python client. This is an icebreaker email only >>> and we will follow the Apache process of releasing the version later on. As >>> of now, I've raised a small PR to add the missing files in the repo, which >>> are required to do the release of the client lib. >>> >>>>>>> > >>> >>>>>>> > The main question I've in mind about the versioning. I see >>> that there are mainly two theories related to maintain the release versions >>> of clients: >>> >>>>>>> > >>> >>>>>>> > The first is about following the semantic release of the >>> clients on its own. The release version of the client doesn't include the >>> version of the software itself. Something like we are doing with Airflow's >>> provider packages. However, in each release, we mention the release version >>> of the main software (Airflow in this case). This is the simple most way of >>> maintaining releases and followed majorly. >>> >>>>>>> > The second way is to include the software's version as well in >>> its client's versioning, something like it's been done in the K8s Python >>> client. In this approach, the client version provides more implicit >>> knowledge about the compatible software, however it creates an extra burden >>> of releasing the client with every minor release of the software, otherwise >>> the versioning would fall apart. Also, there is a good chance that most of >>> these minor or even a major release has no changes in the APIs, but the >>> client would be needed to re-released nevertheless, just to keep the >>> versioning in check. >>> >>>>>>> > >>> >>>>>>> > Personally, I'm in favour of the 1st way, as it's simple to >>> manage and doesn't tightly couple client with its software. >>> >>>>>>> > >>> >>>>>>> > Another unrelated question I have is about the release of >>> different client libraries. As of now, Airflow has Python and Go clients, >>> so should we try to release them together or they should follow their own >>> paths? In general, I'm against batching them together, cause the person who >>> is releasing them may not be able to test all the different clients in the >>> future. >>> >>>>>>> > >>> >>>>>>> > Let me know your thoughts on these 2 points. >>> >>>>>>> > >>> >>>>>>> > Thanks, >>> >>>>>>> > Sumit Maheshwari >>> >>>>>>> > PMC Apache Airflow >>> >> >>> >> >>> >> >>> >> -- >>> >> +48 660 796 129 >>> >>
