Hello, Since I have not heard any further comments till now. I will close this discussion and formally name our next major release (this coming Oct.) 1.0.0; will update the wiki pages accordingly.
Sönke, Thanks for volunteering in documenting the release logistics. I'll leave it to Ismael to to sync up with you (probably over JIRA) since he mentioned he will create a ticket for capturing all the aspects. Guozhang On Mon, Jul 24, 2017 at 12:28 PM, Guozhang Wang <wangg...@gmail.com> wrote: > Thanks for everyone who has shared their thoughts and feedbacks to this > discussion. Here is my conclusion: > > 1. So far everyone has agreed on naming the next major release version to > 1.0; if I did not hear any other opinions by the end of today PDT time I > will name the next release as 1.0.0 and update the release wiki page > correspondingly. > > 2. People have also expressed expectations to some of the release > logistics while we are releasing 1.0.0, including old version support > period, upgrade support period, bug fix / security fix backport policies > etc. Ismael has a detailed summary on these in this email thread and we > will look for someone to drive the documentation of these in parallel with > the 1.0.0 release process. > > > Guozhang > > > On Mon, Jul 24, 2017 at 11:45 AM, Becket Qin <becket....@gmail.com> wrote: > >> +1 on 1.0. I think it makes a lot of sense given the point Kafka is now. >> To >> me Kafka has absolutely reached 1.0 in terms of features/functions. That >> said, I do share the same opinion with others that the usablity and >> policies might still need some improvement to meet the 1.0 standard. >> >> On Sat, Jul 22, 2017 at 7:21 AM, Matthias J. Sax <matth...@confluent.io> >> wrote: >> >> > I agree with Ismael. If we go for 1.0 (what I do support), we need to >> > meet people's expectations and should document all policies and >> > guarantees explicitly. We might also consider to support older releases >> > longer and do bug fix release for older releases, too. >> > >> > Other projects (like Flink) do a fantastic job with this regard and we >> > should learn from them. >> > >> > -Matthias >> > >> > On 7/21/17 9:50 PM, Guozhang Wang wrote: >> > > That's fair enough too. >> > > >> > > Guozhang >> > > >> > > On Fri, Jul 21, 2017 at 12:13 PM, Ismael Juma <isma...@gmail.com> >> wrote: >> > > >> > >> Yes, I agree that the choice of version number can be done >> separately. >> > >> That's why I said I'd file a separate JIRA for the documentation >> > >> improvements. Having said that, there are some expectations that >> people >> > >> have for projects that have reached 1.0.0 and we should try to >> allocate >> > >> time for the important ones. >> > >> >> > >> Ismael >> > >> >> > >> On 21 Jul 2017 8:07 pm, "Guozhang Wang" <wangg...@gmail.com> wrote: >> > >> >> > >>> Thanks Ismael. I agree with you on all these points, and for some of >> > >> these >> > >>> points like 3) we never have a written-down policy though in >> practice >> > we >> > >>> tend to follow some patterns. >> > >>> >> > >>> To me deciding what's the version number of the next major release >> does >> > >> not >> > >>> necessarily mean we need now to change any of these or to set the >> hard >> > >>> rules along with it; I'd like to keep them as two separate >> discussions >> > as >> > >>> they seem semi-orthogonal to me. >> > >>> >> > >>> >> > >>> Guozhang >> > >>> >> > >>> >> > >>> On Fri, Jul 21, 2017 at 8:44 AM, Ismael Juma <ism...@juma.me.uk> >> > wrote: >> > >>> >> > >>>> On the topic of documentation, we should also document which >> releases >> > >> are >> > >>>> still supported and which are not. There a few factors to consider: >> > >>>> >> > >>>> 1. Which branches receive bug fixes. We typically backport fixes to >> > the >> > >>> two >> > >>>> most recent stable branches (the most recent stable branch >> typically >> > >> gets >> > >>>> more backports than the older one). >> > >>>> >> > >>>> 2. Which branches receive security fixes. This could be the same as >> > >> `1`, >> > >>>> but we could attempt to backport more aggressively for security >> fixes >> > >> as >> > >>>> they tend to be rare (so far at least) and the impact could be >> severe. >> > >>>> >> > >>>> 3. The release policy for stable branches. We tend to stop >> releasing >> > >>> from a >> > >>>> given stable branch before we stop backporting fixes. Maybe that's >> OK, >> > >>> but >> > >>>> it would be good to document how we decide that a bug fix release >> is >> > >>>> needed. >> > >>>> >> > >>>> 4. How long are direct upgrades supported for. During the >> time-based >> > >>>> releases discussion, we agreed to support direct upgrades for 2 >> years. >> > >> As >> > >>>> it happens, direct upgrades from 0.8.2 to 1.0.0 would not be >> supported >> > >> if >> > >>>> we follow this strictly. Not clear if we want to do that. >> > >>>> >> > >>>> 5. How long are older clients supported for. Current brokers >> support >> > >>>> clients all the way to 0.8.x. >> > >>>> >> > >>>> 6. How long are older brokers supported for. Current clients >> support >> > >>> 0.10.x >> > >>>> and newer brokers. >> > >>>> >> > >>>> 7. How long are message formats supported for. We never discussed >> > >> this. I >> > >>>> think 5 years would probably be the minimum. >> > >>>> >> > >>>> Ismael >> > >>>> >> > >>>> P.S. I'll file a JIRA to capture this information. >> > >>>> >> > >>>> On Wed, Jul 19, 2017 at 10:12 AM, Ismael Juma <ism...@juma.me.uk> >> > >> wrote: >> > >>>> >> > >>>>> Hi Stevo, >> > >>>>> >> > >>>>> Thanks for your feedback. We should definitely do a better job of >> > >>>>> documenting things. We basically follow semantic versioning, but >> it's >> > >>>>> currently a bit confusing because: >> > >>>>> >> > >>>>> 1. There are 4 segments in the version. The "0." part should be >> > >> ignored >> > >>>>> when deciding what is major, minor and patch at the moment, but >> many >> > >>>> people >> > >>>>> don't know this. Once we move to 1.0.0, that problem goes away. >> > >>>>> >> > >>>>> 2. To know what is a public API, you must check the Javadoc ( >> > >>>>> https://kafka.apache.org/0110/javadoc/index.html?org/ >> > >>>>> apache/kafka/clients/consumer/KafkaConsumer.html). If it's not >> > >> listed >> > >>>>> there, it's not public API. Ideally, it would be obvious from the >> > >>> package >> > >>>>> name (i.e. there would be "internals" in the name), but we are not >> > >>> there >> > >>>>> yet. The exception are the old Scala APIs, but they have all been >> > >>>>> deprecated and they will be removed eventually (the old Scala >> > >> consumers >> > >>>>> won't be removed until the June 2018 release at the earliest in >> order >> > >>> to >> > >>>>> give people time to migrate). >> > >>>>> >> > >>>>> 3. Even though we are following reasonably common practices, we >> > >> haven't >> > >>>>> documented them all in one place. It would be great to do it >> during >> > >> the >> > >>>>> next release cycle. >> > >>>>> >> > >>>>> A few comments below. >> > >>>>> >> > >>>>> On Wed, Jul 19, 2017 at 1:31 AM, Stevo Slavić <ssla...@gmail.com> >> > >>> wrote: >> > >>>>> >> > >>>>>> - APIs not labeled or labeled as stable >> > >>>>>> -- change in major version is only one that can break backward >> > >>>>>> compatibility (client APIs or behavior) >> > >>>>>> >> > >>>>> >> > >>>>> To clarify, stable APIs should not be changed in an incompatible >> way >> > >>>>> without a deprecation cycle. Independently of whether it's a major >> > >>>> release >> > >>>>> or not. >> > >>>>> >> > >>>>> >> > >>>>>> -- change in minor version can introduce new features, but not >> break >> > >>>>>> backward compatibility >> > >>>>>> -- change in patch version, is for bug fixes only. >> > >>>>>> >> > >>>>> >> > >>>>> Right, this has been the case for a while already. Also see >> > >> annotations >> > >>>>> below. >> > >>>>> >> > >>>>> >> > >>>>>> - APIs labeled as evolving can be broken in backward incompatible >> > >> way >> > >>> in >> > >>>>>> any release, but are assumed less likely to be broken compared to >> > >>>> unstable >> > >>>>>> APIs >> > >>>>>> - APIs labeled as unstable can be broken in backward incompatible >> > >> way >> > >>> in >> > >>>>>> any release, major, minor or patch >> > >>>>>> >> > >>>>> >> > >>>>> The relevant annotations do explain this: >> > >>>>> >> > >>>>> https://kafka.apache.org/0110/javadoc/org/apache/kafka/ >> > >>>> common/annotation/ >> > >>>>> InterfaceStability.html >> > >>>>> https://kafka.apache.org/0110/javadoc/org/apache/kafka/ >> > >>>> common/annotation/ >> > >>>>> InterfaceStability.Stable.html >> > >>>>> https://kafka.apache.org/0110/javadoc/org/apache/kafka/ >> > >>>> common/annotation/ >> > >>>>> InterfaceStability.Evolving.html >> > >>>>> https://kafka.apache.org/0110/javadoc/org/apache/kafka/ >> > >>>> common/annotation/ >> > >>>>> InterfaceStability.Unstable.html >> > >>>>> >> > >>>>> But we should have a section in our documentation as well. >> > >>>>> >> > >>>>> >> > >>>>>> - deprecated stable APIs are treated as any stable APIs, they >> can be >> > >>>>>> removed only in major release, are not allowed to be changed in >> > >>> backward >> > >>>>>> incompatible way in either patch or minor version release >> > >>>>>> >> > >>>>> >> > >>>>> Right, but note that stable non-deprecated APIs provide stronger >> > >>>>> guarantees in major releases (they can't be changed in an >> > >> incompatible >> > >>>> way). >> > >>>>> >> > >>>>>> >> > >>>>>> This means one should be able to upgrade server and >> recompile/deploy >> > >>>> apps >> > >>>>>> with clients to new minor.patch release with dependency version >> > >> change >> > >>>>>> being only change needed and there would be no drama. >> > >>>>>> >> > >>>>> >> > >>>>> That should have been the case for a while as long as you are >> using >> > >>>> stable >> > >>>>> public APIs. >> > >>>>> >> > >>>>>> >> > >>>>>> Practice/"features" like protocol version being a parameter, and >> > >>>>>> defaulting >> > >>>>>> to latest so auto updated with dependency update which introduces >> > >> new >> > >>>>>> protocol/behavior should not be used in public client APIs. To >> > >> switch >> > >>>>>> between backward incompatible APIs (contract and behaviors), >> ideally >> > >>>> user >> > >>>>>> should explicitly have to change code and not dependency only, >> but >> > >> at >> > >>>>>> least >> > >>>>>> it should be clearly communicated that there are breaking >> changes to >> > >>>>>> expect >> > >>>>>> even with just dependency update by e.g. giving major version >> > >> release >> > >>>>>> clear >> > >>>>>> meaning. If app dependency on Kafka client library minor.patch on >> > >> same >> > >>>>>> major is updated, and if there's a change in behavior or API >> > >> requiring >> > >>>> app >> > >>>>>> code change - it's a bug. >> > >>>>>> >> > >>>>> >> > >>>>> Hmm, if the protocol bump provides improved behaviour, that is >> not a >> > >>>>> backwards incompatible change though. So, I don't think I agree >> with >> > >>>> this. >> > >>>>> Of course, >> > >>>>> it does mean that _downgrading_ may cause loss of functionality. >> > >> That's >> > >>>>> OK, in my opinion. >> > >>>>> >> > >>>>> Change introduced contrary to the SLO, is OK to be reported as >> bug. >> > >>>>>> Everything else is improvement or feature request. >> > >>>>>> >> > >>>>>> If this was the case, and 1.0.0 was released today with APIs as >> they >> > >>> are >> > >>>>>> now, Scala client APIs even though deprecated would not break and >> > >>>> require >> > >>>>>> refactoring with every 1.* minor/patch release, and would only be >> > >>>> allowed >> > >>>>>> to be broken or removed in future major release, like 2.0.0 >> > >>>>>> >> > >>>>> >> > >>>>> Yes, that is the plan for any _public_ Scala client APIs that are >> > >> still >> > >>>>> present in 1.0.0. The public Scala client APIs are the producer >> and >> > >>>>> consumer, basically. Again, we should make this clear in our >> > >>>> documentation. >> > >>>>> Note that we have made an effort to keep those APIs compatible for >> > >>> quite >> > >>>> a >> > >>>>> while. It sounds like you have had some issues, were they related >> to >> > >>>> usage >> > >>>>> of internal Admin APIs by any chance (since we didn't have a >> public >> > >>>>> AdminClient API until very recently)? >> > >>>>> >> > >>>>>> >> > >>>>>> It should be also clear how long is each version supported - >> e.g. if >> > >>>>>> minor.patch had meaning that there are no backward incompatible >> > >>> changes, >> > >>>>>> it's OK to file a bug only for current major.minor.patch; >> previous >> > >>> major >> > >>>>>> and its last minor.patch can only have patches released up to >> some >> > >>> time >> > >>>>>> like 1 up to 3 months. >> > >>>>>> >> > >>>>> >> > >>>>> I am not sure I understood this point correctly. Can you please >> > >>> clarify? >> > >>>>> >> > >>>>> If there are changes in release cadence with new versioning, it >> > >> should >> > >>> be >> > >>>>>> clear too. >> > >>>>>> >> > >>>>> >> > >>>>> No changes are planned. We have started time-based releases less >> > >> than a >> > >>>>> year ago and they seem to be going well. >> > >>>>> >> > >>>>> Ismael >> > >>>>> >> > >>>> >> > >>> >> > >>> >> > >>> >> > >>> -- >> > >>> -- Guozhang >> > >>> >> > >> >> > > >> > > >> > > >> > >> > >> > > > > -- > -- Guozhang > -- -- Guozhang
