I think I'm fine with the approach you suggested Jarek. This approach mitigates the need to release clients with each patch release of Airflow, and that does make sense.
I hope that others are fine as well with the approach, where we've to release clients with every major or minor release of Airflow. On Mon, Apr 5, 2021 at 5:31 PM Jarek Potiuk <ja...@potiuk.com> wrote: > Hmm. I think it's a great direction, but 4 digits is a bit too much :) > keeping three-digit SemVer is good for clarity/tools. And itroducing 4th > digit is not really needed i think. > > Your proposal Sumit made me think, and I asked myself a few questions: > > 1) will we ever release API client new feature without releasing that > feature in airflow ? I think not. The API client should provide API calls > for all the endpoints available and any change in those endpoints or adding > new one - by definition - should mean a bump in airflow minor version at > least. > > 2) is it likely we release a bugfix to a single API client only without > releasing Airflow new bugfix? I think so. We might upgrade generator for > one client only to fix a particular problem we find for that language. > > 3) is it likely we release airflow bugfix without the need of releasing > any of the clients? Yeah it's very likely the client will work without any > changes with the new Airflow bugfix version. > > ... thinking .... > > So, how about a hybrid solution where we also include the minor version of > Airflow in the client version > > Say airflow 2.1.x will work only with client 2.1.n but x and n are > independent ? > > Example: > > We release 2.1.0 of Airflow and 2.1.0 of the Python and Java clients. > > Then we release 2.1.1, 2.1.2, 2.1.3 of Airflow. No release of clients. > > Then we find a bug in the Python generator, upgrade it and fix it in 2.1.1 > version of the Python client. But the Java client might still be at 2.1.0 > as the generator for Java worked fine. > > At this point everyone who uses 2.1 airflow SHOULD upgrade to 2.1.3 of > Airflow and 2.1.1 of the Python client and 2.1.0 of the Java client. If > anyone uses any other combination, they should first upgrade to those > latest 2.1 ones before 'complaining' that there are some problems. > > Next - whenever we release 2.2.0 of Airflow we also release 2.2.0 of all > clients. And our client check if Airflow is the same major/minor version > and fails if it is not. So you cannot use 2.1.1 Python client with 2.0 > Airflow - you must switch to 2.2.0. > > I think that might be the nicest approach for both users and maintainers. > In this case we do not even have to maintain the 'compatibility' matrix i > mentioned before. And we keep 100% Semver semantics. And we can add > automated tests for all the 'last X.Y client + airflow' combination as > there is always only one 'blessed' combination airflow + clients that is > 'supposed' to work. > > WDYT? > > J. > > > > > pon., 5 kwi 2021, 08:09 użytkownik Sumit Maheshwari < > sumeet.ma...@gmail.com> napisał: > >> Thanks, everyone for the inputs. >> >> I think it's clear that we want to go with the hybrid way of versioning. >> Given that we can automate the client releases easily, how if we release a >> client automatically with each Airflow release, and go for a four numbered >> release, where the first three would be the Airflow's release number and >> the fourth one would be specific to the client, something like this >> >> *Airflow 2.0.1 <=> Client 2.0.1.0* -- Initial release >> *Airflow 2.0.1 <=> Client 2.0.1.1* -- Some fix done in client (treating >> major or minor same for client, ideally there won't be any major patches in >> client) >> >> The upside of this process is that we can automate it & the client >> version will be in sync with the Airflow version always. The downside that >> we would be releasing more packages than required. >> >> About Vikram's question of not supporting any APIs in the client, AFAIK >> there are none. >> >> On Mon, Apr 5, 2021 at 2:51 AM Vikram Koka <vik...@astronomer.io.invalid> >> wrote: >> >>> Sumit, >>> Thank you for bringing up this discussion. >>> >>> Reading through the thread, I am definitely far more comfortable with >>> the hybrid approach proposed by Kaxil, than the first approach, primarily >>> because of the same concerns articulated by Jarek regarding end user >>> experience. Prior to that, I was leaning towards the second approach. >>> >>> Having said that, I think we need to be sure to follow certain >>> guidelines with Core Airflow to make sure this approach works consistently, >>> with a minimum of documentation needed and documentation to be read by >>> Airflow users. Since there are generated clients, I am specifically >>> referring to changes within Airflow regarding the API. Most importantly, I >>> think we need to have any API additions, whether a new endpoint or an >>> enhancement to an existing endpoint, only be available as part of a >>> "feature release", rather than a "patch release". And that, this should >>> trigger an update to the generated clients, triggering a new version of the >>> same. This may be obvious, because we agreed on following semantic >>> versioning strictly, but I thought it was worth reiterating. >>> >>> Having said that, I think it is worth asking the question if that is >>> actually true, or if there are any APIs which are not (or should not be) >>> supported in the clients? >>> I didn't think so, but wanted to make sure. >>> >>> Best regards, >>> Vikram >>> >>> >>> On Sat, Apr 3, 2021 at 4:11 AM Jarek Potiuk <ja...@potiuk.com> wrote: >>> >>>> I also like the hybrid approach proposed by Kaxil. It is not as >>>> simplenfor dependency management as 1-1 version, but IMHO it is >>>> simple-enough. As long as we strictly follow SemVer, a number of questions >>>> i listed above from the user, are answered: >>>> >>>> >>>> 1) Will my client be compatible with the version of Airflow ? >>>> >>>> Easy: 2.x.y will be compatible with 2.n.m of Airflow and so on.. Since >>>> we will not have breaking changes, the 'broad compatibility' level should >>>> be fine (with some feature -level differences but no general compatibility >>>> problems) >>>> >>>> 2) Will feature (API endpoint) x available in this version of client be >>>> working with the version of Airflow I have? >>>> >>>> We have to make sure that features are documented in the APIs at the >>>> level of '@since' if we keep it updated the form 'This feature is available >>>> in 'Airflow 2.3.*' and 'API 2.1.*' - we are covered. Since there will be >>>> no breaking changes in neither API nor Airflow - this will be enough. >>>> >>>> >>>> 3) Which version of Airflow should I install in order to use feature X >>>> that I have in this new version of client ? >>>> >>>> I think this might be possible to generate such compatibility matrix >>>> for all new API features added since Airflow 2.0 - it might be >>>> automatically generated from the API specification in this case. If we >>>> parse the API definition and have the right 'since' definition for both >>>> Airflow and API, it should be rather easy to generate such table: >>>> >>>> Feature | Min API version | Min Airflow version >>>> MoonShot feature X | 2.3 | 2.2 >>>> >>>> Having such compatibility matric would, I think, be a great help for >>>> the users i think when they would plan their Airflow/API releases >>>> >>>> >>>> J. >>>> >>>> sob., 3 kwi 2021, 12:06 użytkownik .... <dzaku...@gmail.com> napisał: >>>> >>>>> +1 for Kaxil proposal >>>>> >>>>> I think, attempting to assign a specific client version to the library >>>>> application makes no sense in creating an API that is based on a >>>>> strict specification. >>>>> >>>>> One of the goals of this API was to make it easier to manage multiple >>>>> instances, and it's not often that you have an identically homogeneous >>>>> environment. It's natural to have different versions of Airflow in one >>>>> organization. If we want users to use a specific version of the >>>>> library on the client-side, it will probably not be possible in many >>>>> cases. >>>>> >>>>> pt., 2 kwi 2021 o 23:35 Kaxil Naik <kaxiln...@gmail.com> napisał(a): >>>>> > >>>>> > Thanks, Sumit, for bringing this up. >>>>> > >>>>> > re: versioning for Python client: >>>>> > >>>>> > I favour a bit of a hybrid of 1 and 2. Initially, I favored just (1) >>>>> when we talked but after seeing how Elastic-Search does it I liked their >>>>> versioning strategy. >>>>> > How their python-client is compatible with ES itself is documented >>>>> at https://elasticsearch-py.readthedocs.io/en/v7.12.0/#compatibility. >>>>> > >>>>> > Since Airflow from 2.0 strictly follows SemVer we can continue with >>>>> the assumption that we (Airflow) won't break backwards-compatibility (to >>>>> the best of our ability). >>>>> > So we can have M.x.y as our versioning for our client where M is the >>>>> Major Version of the client that matches the Major version of Airflow; x >>>>> is >>>>> the minor version for the client & y is the patch release. Here x & y are >>>>> independent of Airflow version. Our goal and message to user would be that >>>>> "You have to use a version of client library matching the major version of >>>>> Airflow" >>>>> > >>>>> > re: the release of different client libraries: >>>>> > >>>>> > Client for each language should be versioned & released separately. >>>>> > >>>>> > Regards, >>>>> > Kaxil >>>>> > >>>>> > On Wed, Mar 31, 2021 at 7:01 PM Jarek Potiuk <ja...@potiuk.com> >>>>> wrote: >>>>> >> >>>>> >> I think there are more questions we need to answer when it comes to >>>>> different versions. If we are going to different versioning for agent and >>>>> Airflow, we will have to accept the fact that over time people will have a >>>>> "matrix of versions" - for both Airflow and the client. And this >>>>> introduces >>>>> "dependency mess" if we do not handle it well. >>>>> >> >>>>> >> I think in this we should give the user much clearer indication on >>>>> which version of Airflow the client should work on, and (more important) >>>>> which version it does NOT work on and which features will NOT be >>>>> available. >>>>> Both cases are possible. >>>>> >> >>>>> >> There should be some way of knowing for the user, (in case there is >>>>> a backwards incompatibility or in case some features are not available) >>>>> that a given version of the client does not work with the version of >>>>> Airflow that the user has. >>>>> >> >>>>> >> I think users looking at the documentation should be able to be >>>>> able to figure out: >>>>> >> >>>>> >> 1) Will my client be compatible with the version of Airflow ? >>>>> >> 2) Will feature (API endpoint) x available in this version of >>>>> client be working with the version of Airflow I have? >>>>> >> 3) Which version of Airflow should I install in order to use >>>>> feature X that I have in this new version of client ? >>>>> >> >>>>> >> Similarly proper error messages should be displayed if there is >>>>> incompatibility or when a feature X is not available when you try to use a >>>>> version of client that supports it with Airflow version of Agent that does >>>>> not have it. >>>>> >> >>>>> >> So I'd say - separate versioning is fine as long as we handle the >>>>> above. >>>>> >> >>>>> >> >>>>> >> Saying all that - there is a beautiful "maintenance" simplicity if >>>>> we actually synchronize the version and we say "Always use SAME version of >>>>> client as Airflow" (and we fail the client if the versions is different). >>>>> This is way more simple in communicating to the users, and with most >>>>> deployments, I think upgrading airflow and clients at the same time should >>>>> be no problem. >>>>> >> >>>>> >> I think over time as the matrix of potential Airflow/Client >>>>> combinations will grow this might mean bigger number of issues that we >>>>> have >>>>> to handle ("Why ma client does not do this and this (and of course it will >>>>> not have neither client nor airflow version specified and we will have to >>>>> ask for clarification). >>>>> >> >>>>> >> So as I see it - maybe release-wise it's a little redundant and >>>>> you'd have to release a bugfix version of Airflow if you find a critical >>>>> problem in Client (which should be rare as agent is mostly automatically >>>>> generated), but synchronizing versions is very tempting from the >>>>> "supportability" point of view. >>>>> >> >>>>> >> Those are my 2 cents, I was not involved too much in the client >>>>> part, so I do not feel like I have a strong voice in there, but if I would >>>>> vote between the two - I'd choose synchronizing the releases. >>>>> >> >>>>> >> BTW. In providers we have a very different story - we keep >>>>> dependency to airflow via PIP (which we can set it, if we find a provider >>>>> stops being compatible with a version of Airflow). Also we automatically >>>>> test in CI if the provider installs and imports on Airflow 2.0.0 to verify >>>>> that it is backwards compatible (at least to the level of importability). >>>>> This has already saved us from "accidental" incompatibilities (wit yaml >>>>> package import change recently). So there, separate versioning makes much >>>>> more sense - also because the version of the provider relates exclusively >>>>> to the feature that the provider version has, not Airflow itself). I think >>>>> this is quite a different story from the client, which is very closely >>>>> coupled with Airflow. >>>>> >> >>>>> >> J. >>>>> >> >>>>> >> >>>>> >> >>>>> >> On Wed, Mar 31, 2021 at 11:08 AM Deng Xiaodong <xd.den...@gmail.com> >>>>> wrote: >>>>> >>> >>>>> >>> Yep, if we can, I would prefer to explicitly mention something >>>>> like 2.0.X :) >>>>> >>> >>>>> >>> On Wed, Mar 31, 2021 at 10:54 AM Sumit Maheshwari < >>>>> sumeet.ma...@gmail.com> wrote: >>>>> >>>> >>>>> >>>> Yeah totally agrees with you XD about the range vs the fixed >>>>> version of the main software. And with Airflow 2 dot & beyond, we would be >>>>> following the semantic versioning more judiciously, so with that can we >>>>> assume that users would be cognizant of the related Airflow version >>>>> implicitly, or do you think we should mention something like 2.0.X in the >>>>> changelog explicitly? >>>>> >>>> >>>>> >>>> On Wed, Mar 31, 2021 at 1:38 PM Deng Xiaodong < >>>>> xd.den...@gmail.com> wrote: >>>>> >>>>> >>>>> >>>>> Thanks, Sumit, for bringing this up. >>>>> >>>>> >>>>> >>>>> - Regarding versioning for Python client: I favour the 1st way >>>>> as well. The only minor comment I have is: instead of "mention the release >>>>> version of the main software", possibly what users would expect is a >>>>> "range >>>>> of compatible versions of main software". If we follow semver for Airflow >>>>> itself relatively strictly, this should not be hard. >>>>> >>>>> >>>>> >>>>> - Regarding the release of different client libraries: I agree >>>>> with you. They should be versioned & released separately. >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> Regards, >>>>> >>>>> XD >>>>> >>>>> >>>>> >>>>> On Wed, Mar 31, 2021 at 9:21 AM Sumit Maheshwari < >>>>> sumeet.ma...@gmail.com> wrote: >>>>> >>>>>> >>>>> >>>>>> Thanks QP for your feedback. I think the latest release of K8s >>>>> client (v17.x.y) is moving in another direction. >>>>> >>>>>> >>>>> >>>>>> On Wed, Mar 31, 2021 at 12:13 AM QP Hou <q...@scribd.com.invalid> >>>>> wrote: >>>>> >>>>>>> >>>>> >>>>>>> Thanks Sumit for kicking off this discussion! >>>>> >>>>>>> >>>>> >>>>>>> I am in favor of approach #1 as well. Airflow software version >>>>> can >>>>> >>>>>>> change due to none API related changes, but the client should >>>>> only >>>>> >>>>>>> care about the public API contract. I believe this is also the >>>>> same >>>>> >>>>>>> approach that the k8s python client took as well? For example, >>>>> the k8s >>>>> >>>>>>> python client version 12.0.1 targets k8s api version 1.16.15. >>>>> >>>>>>> >>>>> >>>>>>> I am also with you that clients for different languages should >>>>> have >>>>> >>>>>>> their own release cycles. Other than testing burdens, every >>>>> client >>>>> >>>>>>> could introduce their own bug fixes and breaking changes >>>>> without >>>>> >>>>>>> having to trigger releases for all other clients. It would be >>>>> a waste >>>>> >>>>>>> of work for us to bump the python client version just for a >>>>> bug fix in >>>>> >>>>>>> the go client. >>>>> >>>>>>> >>>>> >>>>>>> Thanks, >>>>> >>>>>>> QP >>>>> >>>>>>> >>>>> >>>>>>> On Tue, Mar 30, 2021 at 1:56 AM Sumit Maheshwari < >>>>> msu...@apache.org> wrote: >>>>> >>>>>>> > >>>>> >>>>>>> > Hello Airflow dev community >>>>> >>>>>>> > >>>>> >>>>>>> > I'm writing this mail in regards to seek feedback on the >>>>> first release of Apache Airflow's Python client. This is an icebreaker >>>>> email only and we will follow the Apache process of releasing the version >>>>> later on. As of now, I've raised a small PR to add the missing files in >>>>> the >>>>> repo, which are required to do the release of the client lib. >>>>> >>>>>>> > >>>>> >>>>>>> > The main question I've in mind about the versioning. I see >>>>> that there are mainly two theories related to maintain the release >>>>> versions >>>>> of clients: >>>>> >>>>>>> > >>>>> >>>>>>> > The first is about following the semantic release of the >>>>> clients on its own. The release version of the client doesn't include the >>>>> version of the software itself. Something like we are doing with Airflow's >>>>> provider packages. However, in each release, we mention the release >>>>> version >>>>> of the main software (Airflow in this case). This is the simple most way >>>>> of >>>>> maintaining releases and followed majorly. >>>>> >>>>>>> > The second way is to include the software's version as well >>>>> in its client's versioning, something like it's been done in the K8s >>>>> Python >>>>> client. In this approach, the client version provides more implicit >>>>> knowledge about the compatible software, however it creates an extra >>>>> burden >>>>> of releasing the client with every minor release of the software, >>>>> otherwise >>>>> the versioning would fall apart. Also, there is a good chance that most of >>>>> these minor or even a major release has no changes in the APIs, but the >>>>> client would be needed to re-released nevertheless, just to keep the >>>>> versioning in check. >>>>> >>>>>>> > >>>>> >>>>>>> > Personally, I'm in favour of the 1st way, as it's simple to >>>>> manage and doesn't tightly couple client with its software. >>>>> >>>>>>> > >>>>> >>>>>>> > Another unrelated question I have is about the release of >>>>> different client libraries. As of now, Airflow has Python and Go clients, >>>>> so should we try to release them together or they should follow their own >>>>> paths? In general, I'm against batching them together, cause the person >>>>> who >>>>> is releasing them may not be able to test all the different clients in the >>>>> future. >>>>> >>>>>>> > >>>>> >>>>>>> > Let me know your thoughts on these 2 points. >>>>> >>>>>>> > >>>>> >>>>>>> > Thanks, >>>>> >>>>>>> > Sumit Maheshwari >>>>> >>>>>>> > PMC Apache Airflow >>>>> >> >>>>> >> >>>>> >> >>>>> >> -- >>>>> >> +48 660 796 129 >>>>> >>>>>
