I am happy to volunteer for working on documenting the release logistics and related topics if someone is needed for that part.
Best regards, Sönke Am 24.07.2017 9:28 nachm. schrieb "Guozhang Wang" <wangg...@gmail.com>: > Thanks for everyone who has shared their thoughts and feedbacks to this > discussion. Here is my conclusion: > > 1. So far everyone has agreed on naming the next major release version to > 1.0; if I did not hear any other opinions by the end of today PDT time I > will name the next release as 1.0.0 and update the release wiki page > correspondingly. > > 2. People have also expressed expectations to some of the release logistics > while we are releasing 1.0.0, including old version support period, upgrade > support period, bug fix / security fix backport policies etc. Ismael has a > detailed summary on these in this email thread and we will look for someone > to drive the documentation of these in parallel with the 1.0.0 release > process. > > > Guozhang > > > On Mon, Jul 24, 2017 at 11:45 AM, Becket Qin <becket....@gmail.com> wrote: > > > +1 on 1.0. I think it makes a lot of sense given the point Kafka is now. > To > > me Kafka has absolutely reached 1.0 in terms of features/functions. That > > said, I do share the same opinion with others that the usablity and > > policies might still need some improvement to meet the 1.0 standard. > > > > On Sat, Jul 22, 2017 at 7:21 AM, Matthias J. Sax <matth...@confluent.io> > > wrote: > > > > > I agree with Ismael. If we go for 1.0 (what I do support), we need to > > > meet people's expectations and should document all policies and > > > guarantees explicitly. We might also consider to support older releases > > > longer and do bug fix release for older releases, too. > > > > > > Other projects (like Flink) do a fantastic job with this regard and we > > > should learn from them. > > > > > > -Matthias > > > > > > On 7/21/17 9:50 PM, Guozhang Wang wrote: > > > > That's fair enough too. > > > > > > > > Guozhang > > > > > > > > On Fri, Jul 21, 2017 at 12:13 PM, Ismael Juma <isma...@gmail.com> > > wrote: > > > > > > > >> Yes, I agree that the choice of version number can be done > separately. > > > >> That's why I said I'd file a separate JIRA for the documentation > > > >> improvements. Having said that, there are some expectations that > > people > > > >> have for projects that have reached 1.0.0 and we should try to > > allocate > > > >> time for the important ones. > > > >> > > > >> Ismael > > > >> > > > >> On 21 Jul 2017 8:07 pm, "Guozhang Wang" <wangg...@gmail.com> wrote: > > > >> > > > >>> Thanks Ismael. I agree with you on all these points, and for some > of > > > >> these > > > >>> points like 3) we never have a written-down policy though in > practice > > > we > > > >>> tend to follow some patterns. > > > >>> > > > >>> To me deciding what's the version number of the next major release > > does > > > >> not > > > >>> necessarily mean we need now to change any of these or to set the > > hard > > > >>> rules along with it; I'd like to keep them as two separate > > discussions > > > as > > > >>> they seem semi-orthogonal to me. > > > >>> > > > >>> > > > >>> Guozhang > > > >>> > > > >>> > > > >>> On Fri, Jul 21, 2017 at 8:44 AM, Ismael Juma <ism...@juma.me.uk> > > > wrote: > > > >>> > > > >>>> On the topic of documentation, we should also document which > > releases > > > >> are > > > >>>> still supported and which are not. There a few factors to > consider: > > > >>>> > > > >>>> 1. Which branches receive bug fixes. We typically backport fixes > to > > > the > > > >>> two > > > >>>> most recent stable branches (the most recent stable branch > typically > > > >> gets > > > >>>> more backports than the older one). > > > >>>> > > > >>>> 2. Which branches receive security fixes. This could be the same > as > > > >> `1`, > > > >>>> but we could attempt to backport more aggressively for security > > fixes > > > >> as > > > >>>> they tend to be rare (so far at least) and the impact could be > > severe. > > > >>>> > > > >>>> 3. The release policy for stable branches. We tend to stop > releasing > > > >>> from a > > > >>>> given stable branch before we stop backporting fixes. Maybe that's > > OK, > > > >>> but > > > >>>> it would be good to document how we decide that a bug fix release > is > > > >>>> needed. > > > >>>> > > > >>>> 4. How long are direct upgrades supported for. During the > time-based > > > >>>> releases discussion, we agreed to support direct upgrades for 2 > > years. > > > >> As > > > >>>> it happens, direct upgrades from 0.8.2 to 1.0.0 would not be > > supported > > > >> if > > > >>>> we follow this strictly. Not clear if we want to do that. > > > >>>> > > > >>>> 5. How long are older clients supported for. Current brokers > support > > > >>>> clients all the way to 0.8.x. > > > >>>> > > > >>>> 6. How long are older brokers supported for. Current clients > support > > > >>> 0.10.x > > > >>>> and newer brokers. > > > >>>> > > > >>>> 7. How long are message formats supported for. We never discussed > > > >> this. I > > > >>>> think 5 years would probably be the minimum. > > > >>>> > > > >>>> Ismael > > > >>>> > > > >>>> P.S. I'll file a JIRA to capture this information. > > > >>>> > > > >>>> On Wed, Jul 19, 2017 at 10:12 AM, Ismael Juma <ism...@juma.me.uk> > > > >> wrote: > > > >>>> > > > >>>>> Hi Stevo, > > > >>>>> > > > >>>>> Thanks for your feedback. We should definitely do a better job of > > > >>>>> documenting things. We basically follow semantic versioning, but > > it's > > > >>>>> currently a bit confusing because: > > > >>>>> > > > >>>>> 1. There are 4 segments in the version. The "0." part should be > > > >> ignored > > > >>>>> when deciding what is major, minor and patch at the moment, but > > many > > > >>>> people > > > >>>>> don't know this. Once we move to 1.0.0, that problem goes away. > > > >>>>> > > > >>>>> 2. To know what is a public API, you must check the Javadoc ( > > > >>>>> https://kafka.apache.org/0110/javadoc/index.html?org/ > > > >>>>> apache/kafka/clients/consumer/KafkaConsumer.html). If it's not > > > >> listed > > > >>>>> there, it's not public API. Ideally, it would be obvious from the > > > >>> package > > > >>>>> name (i.e. there would be "internals" in the name), but we are > not > > > >>> there > > > >>>>> yet. The exception are the old Scala APIs, but they have all been > > > >>>>> deprecated and they will be removed eventually (the old Scala > > > >> consumers > > > >>>>> won't be removed until the June 2018 release at the earliest in > > order > > > >>> to > > > >>>>> give people time to migrate). > > > >>>>> > > > >>>>> 3. Even though we are following reasonably common practices, we > > > >> haven't > > > >>>>> documented them all in one place. It would be great to do it > during > > > >> the > > > >>>>> next release cycle. > > > >>>>> > > > >>>>> A few comments below. > > > >>>>> > > > >>>>> On Wed, Jul 19, 2017 at 1:31 AM, Stevo Slavić <ssla...@gmail.com > > > > > >>> wrote: > > > >>>>> > > > >>>>>> - APIs not labeled or labeled as stable > > > >>>>>> -- change in major version is only one that can break backward > > > >>>>>> compatibility (client APIs or behavior) > > > >>>>>> > > > >>>>> > > > >>>>> To clarify, stable APIs should not be changed in an incompatible > > way > > > >>>>> without a deprecation cycle. Independently of whether it's a > major > > > >>>> release > > > >>>>> or not. > > > >>>>> > > > >>>>> > > > >>>>>> -- change in minor version can introduce new features, but not > > break > > > >>>>>> backward compatibility > > > >>>>>> -- change in patch version, is for bug fixes only. > > > >>>>>> > > > >>>>> > > > >>>>> Right, this has been the case for a while already. Also see > > > >> annotations > > > >>>>> below. > > > >>>>> > > > >>>>> > > > >>>>>> - APIs labeled as evolving can be broken in backward > incompatible > > > >> way > > > >>> in > > > >>>>>> any release, but are assumed less likely to be broken compared > to > > > >>>> unstable > > > >>>>>> APIs > > > >>>>>> - APIs labeled as unstable can be broken in backward > incompatible > > > >> way > > > >>> in > > > >>>>>> any release, major, minor or patch > > > >>>>>> > > > >>>>> > > > >>>>> The relevant annotations do explain this: > > > >>>>> > > > >>>>> https://kafka.apache.org/0110/javadoc/org/apache/kafka/ > > > >>>> common/annotation/ > > > >>>>> InterfaceStability.html > > > >>>>> https://kafka.apache.org/0110/javadoc/org/apache/kafka/ > > > >>>> common/annotation/ > > > >>>>> InterfaceStability.Stable.html > > > >>>>> https://kafka.apache.org/0110/javadoc/org/apache/kafka/ > > > >>>> common/annotation/ > > > >>>>> InterfaceStability.Evolving.html > > > >>>>> https://kafka.apache.org/0110/javadoc/org/apache/kafka/ > > > >>>> common/annotation/ > > > >>>>> InterfaceStability.Unstable.html > > > >>>>> > > > >>>>> But we should have a section in our documentation as well. > > > >>>>> > > > >>>>> > > > >>>>>> - deprecated stable APIs are treated as any stable APIs, they > can > > be > > > >>>>>> removed only in major release, are not allowed to be changed in > > > >>> backward > > > >>>>>> incompatible way in either patch or minor version release > > > >>>>>> > > > >>>>> > > > >>>>> Right, but note that stable non-deprecated APIs provide stronger > > > >>>>> guarantees in major releases (they can't be changed in an > > > >> incompatible > > > >>>> way). > > > >>>>> > > > >>>>>> > > > >>>>>> This means one should be able to upgrade server and > > recompile/deploy > > > >>>> apps > > > >>>>>> with clients to new minor.patch release with dependency version > > > >> change > > > >>>>>> being only change needed and there would be no drama. > > > >>>>>> > > > >>>>> > > > >>>>> That should have been the case for a while as long as you are > using > > > >>>> stable > > > >>>>> public APIs. > > > >>>>> > > > >>>>>> > > > >>>>>> Practice/"features" like protocol version being a parameter, and > > > >>>>>> defaulting > > > >>>>>> to latest so auto updated with dependency update which > introduces > > > >> new > > > >>>>>> protocol/behavior should not be used in public client APIs. To > > > >> switch > > > >>>>>> between backward incompatible APIs (contract and behaviors), > > ideally > > > >>>> user > > > >>>>>> should explicitly have to change code and not dependency only, > but > > > >> at > > > >>>>>> least > > > >>>>>> it should be clearly communicated that there are breaking > changes > > to > > > >>>>>> expect > > > >>>>>> even with just dependency update by e.g. giving major version > > > >> release > > > >>>>>> clear > > > >>>>>> meaning. If app dependency on Kafka client library minor.patch > on > > > >> same > > > >>>>>> major is updated, and if there's a change in behavior or API > > > >> requiring > > > >>>> app > > > >>>>>> code change - it's a bug. > > > >>>>>> > > > >>>>> > > > >>>>> Hmm, if the protocol bump provides improved behaviour, that is > not > > a > > > >>>>> backwards incompatible change though. So, I don't think I agree > > with > > > >>>> this. > > > >>>>> Of course, > > > >>>>> it does mean that _downgrading_ may cause loss of functionality. > > > >> That's > > > >>>>> OK, in my opinion. > > > >>>>> > > > >>>>> Change introduced contrary to the SLO, is OK to be reported as > bug. > > > >>>>>> Everything else is improvement or feature request. > > > >>>>>> > > > >>>>>> If this was the case, and 1.0.0 was released today with APIs as > > they > > > >>> are > > > >>>>>> now, Scala client APIs even though deprecated would not break > and > > > >>>> require > > > >>>>>> refactoring with every 1.* minor/patch release, and would only > be > > > >>>> allowed > > > >>>>>> to be broken or removed in future major release, like 2.0.0 > > > >>>>>> > > > >>>>> > > > >>>>> Yes, that is the plan for any _public_ Scala client APIs that are > > > >> still > > > >>>>> present in 1.0.0. The public Scala client APIs are the producer > and > > > >>>>> consumer, basically. Again, we should make this clear in our > > > >>>> documentation. > > > >>>>> Note that we have made an effort to keep those APIs compatible > for > > > >>> quite > > > >>>> a > > > >>>>> while. It sounds like you have had some issues, were they related > > to > > > >>>> usage > > > >>>>> of internal Admin APIs by any chance (since we didn't have a > public > > > >>>>> AdminClient API until very recently)? > > > >>>>> > > > >>>>>> > > > >>>>>> It should be also clear how long is each version supported - > e.g. > > if > > > >>>>>> minor.patch had meaning that there are no backward incompatible > > > >>> changes, > > > >>>>>> it's OK to file a bug only for current major.minor.patch; > previous > > > >>> major > > > >>>>>> and its last minor.patch can only have patches released up to > some > > > >>> time > > > >>>>>> like 1 up to 3 months. > > > >>>>>> > > > >>>>> > > > >>>>> I am not sure I understood this point correctly. Can you please > > > >>> clarify? > > > >>>>> > > > >>>>> If there are changes in release cadence with new versioning, it > > > >> should > > > >>> be > > > >>>>>> clear too. > > > >>>>>> > > > >>>>> > > > >>>>> No changes are planned. We have started time-based releases less > > > >> than a > > > >>>>> year ago and they seem to be going well. > > > >>>>> > > > >>>>> Ismael > > > >>>>> > > > >>>> > > > >>> > > > >>> > > > >>> > > > >>> -- > > > >>> -- Guozhang > > > >>> > > > >> > > > > > > > > > > > > > > > > > > > > > > > > -- > -- Guozhang >
