Those are great suggestions, thank you. We will continue this discussion forward in a separate KIP for release plan for Tiered Storage.
On Thu 27. Jul 2023 at 21:46, Ismael Juma <m...@ismaeljuma.com> wrote: > Hi Divij, > > I think the points you bring up for discussion are all good. My main > feedback is that they should be discussed in the context of KIPs vs the > release template. That's why we have a backwards compatibility section for > every KIP, it's precisely to ensure we think carefully about some of the > points you're bringing up. When it comes to defining the meaning of early > access, we have two options: > > 1. Have a KIP specifically for tiered storage. > 2. Have a KIP to define general guidelines for what early access means. > > Does this make sense? > > Ismael > > On Thu, Jul 27, 2023 at 6:38 PM Divij Vaidya <divijvaidy...@gmail.com> > wrote: > > > Thank you for the response, Ismael. > > > > 1. Specifically in context of 3.6, I wanted this compatibility > > guarantee point to encourage a discussion on > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-952%3A+Regenerate+segment-aligned+producer+snapshots+when+upgrading+to+a+Kafka+version+supporting+Tiered+Storage > > . > > Due to lack of producer snapshots in <2.8 versions, a customer may not > > be able to upgrade to 3.6 and use TS on a topic which was created when > > the cluster was on <2.8 version (see motivation for details). We can > > discuss and agree that it does not break compatibility, which is fine. > > But I want to ensure that we have a discussion soon on this to reach a > > conclusion. > > > > 2. I will start a KIP on this for further discussion. > > > > 3. In the context of 3.6, this would mean that there should be > > no-regression, if a user does "not" turn-on remote storage (early > > access feature) at a cluster level. We have some known cases (such as > > https://issues.apache.org/jira/browse/KAFKA-15189) which violate this > > compatibility requirement. Having this guarantee mentioned in the > > release plan will ensure that we are all in agreement with which cases > > are truly blockers and which aren't. > > > > 4. Fair, instead of a general goal, let me put it specifically in the > > context of 3.6. Let me know if this is not the right forum for this > > discussion. > > Once a user "turns on" tiered storage (TS) at a cluster level, I am > > proposing that they should have the ability to turn it off as well at > > a cluster level. Since this is a topic level feature, folks may not > > spin up a separate cluster to try this feature, hence, we need to > > ensure that we provide them with the ability to try tiered storage for > > a topic which could be deleted and featured turned-off, so that rest > > of the production cases are not impacted. > > > > 5. Agree on not making public interface change as a requirement but we > > should define what "early access" means in that case. Users may not be > > aware that "early access" public APIs may change (unless I am missing > > some documentation somewhere completely, in which case I apologize for > > bringing this naive point). > > > > -- > > Divij Vaidya > > > > On Thu, Jul 27, 2023 at 2:27 PM Ismael Juma <m...@ismaeljuma.com> wrote: > > > > > > Hi Divij, > > > > > > Some of these are launch checklist items (not really goals) and some > are > > > compatibility guarantees. More below. > > > > > > On Thu, Jul 27, 2023, 12:10 PM Divij Vaidya <divijvaidy...@gmail.com> > > wrote: > > > > > > > Hey Satish > > > > > > > > Could we consider adding "launch goals" in the release plan. While > > > > some of these may be implicit, it would be nice to list them down in > > > > the release plan. For this release, our launch requirements would be: > > > > 1. Users should be able to upgrade from any prior Kafka version to > this > > > > version. > > > > > > > > > > This is part of the compatibility guarantees. The upgrade notes mention > > > this already. If there is a change in a given release, it should > > definitely > > > be highlighted. > > > > > > 2. On release, this version (or it's dependencies) would not have any > > > > known MEDIUM/HIGH CVE. > > > > > > > > > > This is a new policy and the details should be discussed. In > particular, > > > the threshold (medium or high). > > > > > > 3. Presence of any "early access"/"beta" feature should not impact > > > > other production features when it is not enabled. > > > > > > > > > > This is a general guideline for early access features and not specific > to > > > this release. It would be good to have a page that talks about these > > things. > > > > > > 4. Once enabled, users should have an option to disable any "early > > > > access"/"beta" feature and resume normal production features, i.e. > > > > impact of beta features should be reversible. > > > > > > > > > > This needs discussion and I don't think it's reasonable as a general > > rule. > > > For example, Kraft early access wasn't reversible and it was not > feasible > > > for it to be. > > > > > > 5. KIP-405 will be available in "early access"/"beta" mode. Early > > > > access/beta means that the public facing interfaces won't change in > > > > future but the implementation is not recommended to be used in > > > > production. > > > > > > > > > I don't think it's ok to make this a requirement. Early access is a way > > to > > > get early feedback and all types of changes should be on the table. > They > > > would be discussed via KIPs as usual. I believe there were some > > > incompatible changes for Kraft during the early access period although > > the > > > team aimed to minimize work required during upgrades. I have mentioned > > > Kraft a couple of times since it's a good example of a large feature > that > > > went through this process. > > > > > > Ismael > > > -- Divij Vaidya
