On the topic of documentation, we should also document which releases are still supported and which are not. There a few factors to consider:
1. Which branches receive bug fixes. We typically backport fixes to the two most recent stable branches (the most recent stable branch typically gets more backports than the older one). 2. Which branches receive security fixes. This could be the same as `1`, but we could attempt to backport more aggressively for security fixes as they tend to be rare (so far at least) and the impact could be severe. 3. The release policy for stable branches. We tend to stop releasing from a given stable branch before we stop backporting fixes. Maybe that's OK, but it would be good to document how we decide that a bug fix release is needed. 4. How long are direct upgrades supported for. During the time-based releases discussion, we agreed to support direct upgrades for 2 years. As it happens, direct upgrades from 0.8.2 to 1.0.0 would not be supported if we follow this strictly. Not clear if we want to do that. 5. How long are older clients supported for. Current brokers support clients all the way to 0.8.x. 6. How long are older brokers supported for. Current clients support 0.10.x and newer brokers. 7. How long are message formats supported for. We never discussed this. I think 5 years would probably be the minimum. Ismael P.S. I'll file a JIRA to capture this information. On Wed, Jul 19, 2017 at 10:12 AM, Ismael Juma <ism...@juma.me.uk> wrote: > Hi Stevo, > > Thanks for your feedback. We should definitely do a better job of > documenting things. We basically follow semantic versioning, but it's > currently a bit confusing because: > > 1. There are 4 segments in the version. The "0." part should be ignored > when deciding what is major, minor and patch at the moment, but many people > don't know this. Once we move to 1.0.0, that problem goes away. > > 2. To know what is a public API, you must check the Javadoc ( > https://kafka.apache.org/0110/javadoc/index.html?org/ > apache/kafka/clients/consumer/KafkaConsumer.html). If it's not listed > there, it's not public API. Ideally, it would be obvious from the package > name (i.e. there would be "internals" in the name), but we are not there > yet. The exception are the old Scala APIs, but they have all been > deprecated and they will be removed eventually (the old Scala consumers > won't be removed until the June 2018 release at the earliest in order to > give people time to migrate). > > 3. Even though we are following reasonably common practices, we haven't > documented them all in one place. It would be great to do it during the > next release cycle. > > A few comments below. > > On Wed, Jul 19, 2017 at 1:31 AM, Stevo Slavić <ssla...@gmail.com> wrote: > >> - APIs not labeled or labeled as stable >> -- change in major version is only one that can break backward >> compatibility (client APIs or behavior) >> > > To clarify, stable APIs should not be changed in an incompatible way > without a deprecation cycle. Independently of whether it's a major release > or not. > > >> -- change in minor version can introduce new features, but not break >> backward compatibility >> -- change in patch version, is for bug fixes only. >> > > Right, this has been the case for a while already. Also see annotations > below. > > >> - APIs labeled as evolving can be broken in backward incompatible way in >> any release, but are assumed less likely to be broken compared to unstable >> APIs >> - APIs labeled as unstable can be broken in backward incompatible way in >> any release, major, minor or patch >> > > The relevant annotations do explain this: > > https://kafka.apache.org/0110/javadoc/org/apache/kafka/common/annotation/ > InterfaceStability.html > https://kafka.apache.org/0110/javadoc/org/apache/kafka/common/annotation/ > InterfaceStability.Stable.html > https://kafka.apache.org/0110/javadoc/org/apache/kafka/common/annotation/ > InterfaceStability.Evolving.html > https://kafka.apache.org/0110/javadoc/org/apache/kafka/common/annotation/ > InterfaceStability.Unstable.html > > But we should have a section in our documentation as well. > > >> - deprecated stable APIs are treated as any stable APIs, they can be >> removed only in major release, are not allowed to be changed in backward >> incompatible way in either patch or minor version release >> > > Right, but note that stable non-deprecated APIs provide stronger > guarantees in major releases (they can't be changed in an incompatible way). > >> >> This means one should be able to upgrade server and recompile/deploy apps >> with clients to new minor.patch release with dependency version change >> being only change needed and there would be no drama. >> > > That should have been the case for a while as long as you are using stable > public APIs. > >> >> Practice/"features" like protocol version being a parameter, and >> defaulting >> to latest so auto updated with dependency update which introduces new >> protocol/behavior should not be used in public client APIs. To switch >> between backward incompatible APIs (contract and behaviors), ideally user >> should explicitly have to change code and not dependency only, but at >> least >> it should be clearly communicated that there are breaking changes to >> expect >> even with just dependency update by e.g. giving major version release >> clear >> meaning. If app dependency on Kafka client library minor.patch on same >> major is updated, and if there's a change in behavior or API requiring app >> code change - it's a bug. >> > > Hmm, if the protocol bump provides improved behaviour, that is not a > backwards incompatible change though. So, I don't think I agree with this. > Of course, > it does mean that _downgrading_ may cause loss of functionality. That's > OK, in my opinion. > > Change introduced contrary to the SLO, is OK to be reported as bug. >> Everything else is improvement or feature request. >> >> If this was the case, and 1.0.0 was released today with APIs as they are >> now, Scala client APIs even though deprecated would not break and require >> refactoring with every 1.* minor/patch release, and would only be allowed >> to be broken or removed in future major release, like 2.0.0 >> > > Yes, that is the plan for any _public_ Scala client APIs that are still > present in 1.0.0. The public Scala client APIs are the producer and > consumer, basically. Again, we should make this clear in our documentation. > Note that we have made an effort to keep those APIs compatible for quite a > while. It sounds like you have had some issues, were they related to usage > of internal Admin APIs by any chance (since we didn't have a public > AdminClient API until very recently)? > >> >> It should be also clear how long is each version supported - e.g. if >> minor.patch had meaning that there are no backward incompatible changes, >> it's OK to file a bug only for current major.minor.patch; previous major >> and its last minor.patch can only have patches released up to some time >> like 1 up to 3 months. >> > > I am not sure I understood this point correctly. Can you please clarify? > > If there are changes in release cadence with new versioning, it should be >> clear too. >> > > No changes are planned. We have started time-based releases less than a > year ago and they seem to be going well. > > Ismael >
