Re: [DISCUSS] KIP-584: Versioning scheme for features

Hi Kowshik, Thanks for the KIP, this is exciting! The KIP includes examples on how operators could use the command line utility, etc. It would be great to add some high-level details on how the upgrade workflow changes overall with the addition of feature versions. - Dhruvil On Wed, Apr 15, 202

Re: [DISCUSS] KIP-584: Versioning scheme for features

Hi Jun, Thanks for the feedback! I have addressed the comments in the KIP. > 200. In the validation section, there is still the text "*from* > {"max_version_level": > X} *to* {"max_version_level": X’}". It seems that it should say "from X to > Y"? (Kowshik): Done. I have reworded it a bit to ma

Re: [DISCUSS] KIP-584: Versioning scheme for features

Hi Jun, Sorry the links were broken in my last response, here are the right links: 200. https://cwiki.apache.org/confluence/display/KAFKA/KIP-584%3A+Versioning+scheme+for+features#KIP-584:Versioning Scheme For Features-Validations

Re: [DISCUSS] KIP-584: Versioning scheme for features

Hi, Kowshik, Looks good to me now. Just a couple of minor things below. 200. In the validation section, there is still the text "*from* {"max_version_level": X} *to* {"max_version_level": X’}". It seems that it should say "from X to Y"? 110. Could we add that we need to document the bumped vers

Re: [DISCUSS] KIP-584: Versioning scheme for features

Hi all, Thank you very much for all the insightful feedback! How do you feel about the KIP? Does the scope and the write up look OK to you, and is it time to call a vote? Cheers, Kowshik On Wed, Apr 15, 2020 at 1:08 PM Kowshik Prakasam wrote: > Hi Jun, > > Thank you for the suggestion! I have

Re: [DISCUSS] KIP-584: Versioning scheme for features

Hi Jun, Thank you for the suggestion! I have updated the KIP, please find my response below. > 200. I guess you are saying only when the allowDowngrade field is set, the > finalized feature version can go backward. Otherwise, it can only go up. > That makes sense. It would be useful to make that

Re: [DISCUSS] KIP-584: Versioning scheme for features

Hi, Kowshik, 200. I guess you are saying only when the allowDowngrade field is set, the finalized feature version can go backward. Otherwise, it can only go up. That makes sense. It would be useful to make that clear when explaining the usage of the allowDowngrade field. In the validation section,

Re: [DISCUSS] KIP-584: Versioning scheme for features

Hi Jun, Great question! Please find my response below. > 200. My understanding is that If the CLI tool passes the > '--allow-downgrade' flag when updating a specific feature, then a future > downgrade is possible. Otherwise, the feature is now downgradable. If so, I > was wondering how the contro

Re: [DISCUSS] KIP-584: Versioning scheme for features

Hi, Kowshik, Thanks for the reply. Makes sense. Just one more question. 200. My understanding is that If the CLI tool passes the '--allow-downgrade' flag when updating a specific feature, then a future downgrade is possible. Otherwise, the feature is now downgradable. If so, I was wondering how t

Re: [DISCUSS] KIP-584: Versioning scheme for features

Hi Jun, Thanks a lot for the feedback and the questions! Please find my response below. > 200. The UpdateFeaturesRequest includes an AllowDowngrade field. It seems > that field needs to be persisted somewhere in ZK? (Kowshik): Great question! Below is my explanation. Please help me understand, i

Re: [DISCUSS] KIP-584: Versioning scheme for features

Hi, Kowshik, Thanks for addressing those comments. Just a few more minor comments. 200. The UpdateFeaturesRequest includes an AllowDowngrade field. It seems that field needs to be persisted somewhere in ZK? 201. UpdateFeaturesResponse has the following top level fields. Should those fields be pe

Re: [DISCUSS] KIP-584: Versioning scheme for features

Hi Jun, Thanks for the feedback! I have updated the KIP-584 addressing your comments. Please find my response below. > 100.6 You can look for the sentence "This operation requires ALTER on > CLUSTER." in KIP-455. Also, you can check its usage in > KafkaApis.authorize(). (Kowshik): Done. Great po

Re: [DISCUSS] KIP-584: Versioning scheme for features

Hi Guozhang, Thanks for the explanation! This is a very good point. I have updated the KIP incorporating the proposed idea. We now maintain/serve MAX as well as MIN version levels of finalized features. So, the client will get to know both these values in the ApiVersionsResponse. This serves as a

Re: [DISCUSS] KIP-584: Versioning scheme for features

Hi, Kowshik, Thanks for the reply. A few more comments. 110. Keeping the feature version as int is probably fine. I just felt that for some of the common user interactions, it's more convenient to relate that to a release version. For example, if a user wants to downgrade to a release 2.5, it's e

Re: [DISCUSS] KIP-584: Versioning scheme for features

Hi Jun, I have updated the KIP for the item 111. I'm in the process of addressing 100.6, and will provide an update soon. I think item 110 is still under discussion given we are now providing a way to finalize all features to their latest version levels. In any case, please let us know how you fee

Re: [DISCUSS] KIP-584: Versioning scheme for features

Hi Colin, Thanks for the feedback, and suggestions! It is a great idea to provide a `--finalize-latest` flag. I agree it's a burden to ask the user to manually upgrade each feature to the latest version, after a release. I have now updated the KIP adding this idea. > What about a simple solution

Re: [DISCUSS] KIP-584: Versioning scheme for features

Hi Jun, I agree that asking the user to manually upgrade all features to the latest version is a burden. Then the user has to know what the latest version of every feature is when upgrading. What about a simple solution to problem this where we add a flag to the command-line tool like --enabl

Re: [DISCUSS] KIP-584: Versioning scheme for features

Hello Kowshik, For 2) above, my motivations is more from the flexibility on client side instead of version deprecation: let's say a client talks to the cluster learned that the cluster-wide version for feature is X, while the client itself only knows how to execute the feature up to version Y ( <

Re: [DISCUSS] KIP-584: Versioning scheme for features

Hi, Kowshik, Thanks for the reply. A few more replies below. 100.6 You can look for the sentence "This operation requires ALTER on CLUSTER." in KIP-455. Also, you can check its usage in KafkaApis.authorize(). 110. From the external client/tooling perspective, it's more natural to use the release

Re: [DISCUSS] KIP-584: Versioning scheme for features

Hi Guozhang, Thanks for the insightful feedback and questions! I have updated the KIP in response to some of the suggestions. Please find my response below. > 1. Could you explain a bit what would the "the set of features supported by > a broker" information, beyond the cluster-level finalized fe

Re: [DISCUSS] KIP-584: Versioning scheme for features

Hi Colin, Thanks a lot for the explanation! I've updated the KIP based on your suggestions. Please find my response to your comments below. > If you can just treat "not present" as version level 0, you can have just checks like the second one. This should lead to simpler code. (Kowshik): Good i

Re: [DISCUSS] KIP-584: Versioning scheme for features

Hello Kowshik, Thanks for the great write-up, overall it reads great to me already. Adding a few meta comments here: 1. Could you explain a bit what would the "the set of features supported by a broker" information, beyond the cluster-level finalized features, be used by the client? I think that

Re: [DISCUSS] KIP-584: Versioning scheme for features

On Fri, Apr 3, 2020, at 11:24, Jun Rao wrote: > Hi, Kowshik, > > Thanks for the reply. A few more comments below. > > 100.6 For every new request, the admin needs to control who is allowed > to issue that request if security is enabled. So, we need to assign the new > request a ResourceType and

Re: [DISCUSS] KIP-584: Versioning scheme for features

On Fri, Apr 3, 2020, at 20:32, Kowshik Prakasam wrote: > > Colin wrote: > > It would be simpler to just say that a feature flag which doesn't appear > > in the znode is considered to be at version level 0. This will also > > simplify the code a lot, I think, since you won't have to keep track of >

Re: [DISCUSS] KIP-584: Versioning scheme for features

Hi Colin, Thanks for the feedback! I have updated the KIP based on your feedback. Please find my response below. > The discussion on ZooKeeper reads versus writes makes sense to me. The important thing to keep in mind here is that in the bridge release, > all brokers can read from ZooKeeper, but

Re: [DISCUSS] KIP-584: Versioning scheme for features

Hi Jun, Thanks for the feedback and suggestions. Please find my response below. > 100.6 For every new request, the admin needs to control who is allowed to > issue that request if security is enabled. So, we need to assign the new > request a ResourceType and possible AclOperations. See > https:/

Re: [DISCUSS] KIP-584: Versioning scheme for features

Hi Kowshik, The discussion on ZooKeeper reads versus writes makes sense to me. The important thing to keep in mind here is that in the bridge release, all brokers can read from ZooKeeper, but only the controller writes. Why do we need both UpdateFeaturesRequest and DeleteFeaturesRequest? It s

Re: [DISCUSS] KIP-584: Versioning scheme for features

Hey Boyang, Thanks for the feedback! I've updated the KIP. Please find below my response. > 1. Do you mind updating the non-goal section as we are introducing a > --feature-force-downgrade to address downgrade concern? (Kowshik): This was already mentioned. Look for non-goal: 1-b. > 2. For the

Re: [DISCUSS] KIP-584: Versioning scheme for features

Hi, Kowshik, Thanks for the reply. A few more comments below. 100.6 For every new request, the admin needs to control who is allowed to issue that request if security is enabled. So, we need to assign the new request a ResourceType and possible AclOperations. See https://cwiki.apache.org/confluen

Re: [DISCUSS] KIP-584: Versioning scheme for features

Hey Kowshik, thanks for getting the KIP updated. The Zookeeper routing approach makes sense and simplifies the changes. Some follow-ups: 1. Do you mind updating the non-goal section as we are introducing a --feature-force-downgrade to address downgrade concern? 2. For the flags `--feature` seems

Re: [DISCUSS] KIP-584: Versioning scheme for features

Hi all, Any other feedback on this KIP before we start the vote? Cheers, Kowshik On Fri, Apr 3, 2020 at 1:27 AM Kowshik Prakasam wrote: > Hey Jun, > > Thanks a lot for the great feedback! Please note that the design > has changed a little bit on the KIP, and we now propagate the finalized > f

Re: [DISCUSS] KIP-584: Versioning scheme for features

Hey Jun, Thanks a lot for the great feedback! Please note that the design has changed a little bit on the KIP, and we now propagate the finalized features metadata only via ZK watches (instead of UpdateMetadataRequest from the controller). Please find below my response to your questions/feedback,

Re: [DISCUSS] KIP-584: Versioning scheme for features

Hey Boyang, Great point! You are right, thanks for the suggestion! Yes, we can just use ZK watches to propagate finalized features information. I have updated the KIP write up with this change. As a result, I feel the design is simpler as we have also eliminated the changes to UpdateMetadataReques

Re: [DISCUSS] KIP-584: Versioning scheme for features

Thanks for the reply. The only remaining question is the propagation path. KIP-500 only restricts `write access` to the controller, in a sense that brokers in the pre-KIP-500 world could still listen to Zookeeper notifications. Thus, we are open to discuss the engineering effort to go through Zooke

Re: [DISCUSS] KIP-584: Versioning scheme for features

Hey Boyang, Thanks for the feedback! Please find below my response to your latest comments. I have modified the KIP wherever possible to address the comments. > My point is that during a bootstrapping stage of a cluster, we could not > pick the desired feature version as no controller is actively

Re: [DISCUSS] KIP-584: Versioning scheme for features

Thanks Kowshik, the answers are making sense. Some follow-ups: On Tue, Mar 31, 2020 at 6:51 PM Jun Rao wrote: > Hi, Kowshik, > > Thanks for the KIP. Looks good overall. A few comments below. > > 100. UpdateFeaturesRequest/UpdateFeaturesResponse > 100.1 Since this request waits for responses from

Re: [DISCUSS] KIP-584: Versioning scheme for features

Hi, Kowshik, Thanks for the KIP. Looks good overall. A few comments below. 100. UpdateFeaturesRequest/UpdateFeaturesResponse 100.1 Since this request waits for responses from brokers, should we add a timeout in the request (like createTopicRequest)? 100.2 The response schema is a bit weird. Typic

Re: [DISCUSS] KIP-584: Versioning scheme for features

Hey Boyang, Thanks for the great feedback! I have updated the KIP based on your feedback. Please find my response below for your comments, look for sentences starting with "(Kowshik)" below. > 1. "When is it safe for the brokers to begin handling EOS traffic" could be > converted as "When is it

Re: [DISCUSS] KIP-584: Versioning scheme for features

Hi Colin, Thanks for the suggestions. It is a good idea to refer to the '__data_version__' as just 'epoch', to avoid any confusions. However note that this is not the same as broker epoch. The main distinction is that this epoch is bumped by the controller whenever a modification made to the final

Re: [DISCUSS] KIP-584: Versioning scheme for features

Hey Kowshik, thanks for the revised KIP. Got a couple of questions: 1. "When is it safe for the brokers to begin handling EOS traffic" could be converted as "When is it safe for the brokers to start serving new Exactly-Once(EOS) semantics" since EOS is not explained earlier in the context. 2. In

Re: [DISCUSS] KIP-584: Versioning scheme for features

On Thu, Mar 26, 2020, at 19:24, Kowshik Prakasam wrote: > Hi Colin, > > Thanks for the feedback! I've changed the KIP to address your > suggestions. > Please find below my explanation. Here is a link to KIP 584: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-584%3A+Versioning+scheme+for+

Re: [DISCUSS] KIP-584: Versioning scheme for features

Hi Colin, Once again, thanks a lot for the feedback! Regarding the first point about using a single version field, you are correct. Recently I understood the idea and realized that both the '__data_version__' and '__schema_version__' can be folded into a single field. This can be bumped anytime wh

Re: [DISCUSS] KIP-584: Versioning scheme for features

Hi Colin, Thanks for the feedback! I've changed the KIP to address your suggestions. Please find below my explanation. Here is a link to KIP 584: https://cwiki.apache.org/confluence/display/KAFKA/KIP-584%3A+Versioning+scheme+for+features . 1. '__data_version__' is the version of the finalized fea

Re: [DISCUSS] KIP-584: Versioning scheme for features

Thanks, Kowshik, this looks good. In the "Schema" section, do we really need both __schema_version__ and __data_version__? Can we just have a single version field here? Shouldn't the Admin(Client) function have some way to get the min and max information that we're exposing as well? I guess w

Re: [DISCUSS] KIP-584: Versioning scheme for features

Hi Boyang, Great catch, thanks! I have fixed this now. Please have a look, and let me if you have any questions. Cheers, Kowshik On Tue, Mar 24, 2020 at 11:06 PM Boyang Chen wrote: > Nice KIP Kowshik! This is a long due feature for the ease of both client > side and server side upgrade in gen

Re: [DISCUSS] KIP-584: Versioning scheme for features

Nice KIP Kowshik! This is a long due feature for the ease of both client side and server side upgrade in general. One meta comment: it seems the KIP's font is a bit weirdly rendered: [image: image.png] Could you try to remove all the rectangle blocks? It looks inconsistent with most KIPs. Thanks,

Re: [DISCUSS] KIP-584: Versioning scheme for features

Hi all, Apologies, please ignore the reference to https://issues.apache.org/jira/browse/KIP-584. It was included accidentally. But the link to the KIP is the right one. Cheers, Kowshik On Tue, Mar 24, 2020 at 5:08 PM Kowshik Prakasam wrote: > Hi all, > > I've opened KIP-584

[DISCUSS] KIP-584: Versioning scheme for features

Hi all, I've opened KIP-584 which is intended to provide a versioning scheme for features. I'd like to use this thread to discuss the same. I'd appreciate any feedback on this. Here is a link to KIP-584: https://cwiki.apache.org/confluence/display/KA