hi All, I noticed that while the discussion seemed to include the functionality to retrieve supported features from a specific broker, the final implementation appears to fetch this information from a "random" broker. I have filed KAFKA-18786 to address this discrepancy, but I would appreciate it if I have misunderstood any aspect of the implementation.
thanks. Chia-Ping On 2020/03/31 22:50:41 Kowshik Prakasam wrote: > Hi Colin, > > Thanks for the suggestions. It is a good idea to refer to the > '__data_version__' as just 'epoch', to avoid any confusions. However note > that this is not the same as broker epoch. The main distinction is that > this epoch is bumped by the controller whenever a modification made to the > finalized feature versions is persisted into ZK. > > I have updated the KIP to use the new schema for the ‘/features’ ZK node: > > - We use 2 separate fields ‘epoch’ and ‘version’. The latter describing > changes to the overall schema of the data that is written to ZooKeeper in > the '/features' node. > - We don’t have a header and a data section separately, I have clubbed > these so that we have just 1 dictionary containing both. > > > Here is a link to the updated section: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-584 > <https://issues.apache.org/jira/browse/KIP-584> > %3A+Versioning+scheme+for+features#KIP-584 > <https://issues.apache.org/jira/browse/KIP-584>:Versioningschemeforfeatures-Persistenceoffinalizedfeatureversions > . > > Please feel free to let me know if you have any questions or concerns. > > > Cheers, > Kowshik > > > On Mon, Mar 30, 2020 at 4:53 PM Colin McCabe <cmcc...@apache.org> wrote: > > > On Thu, Mar 26, 2020, at 19:24, Kowshik Prakasam wrote: > > > Hi Colin, > > > > > > Thanks for the feedback! I've changed the KIP to address your > > > suggestions. > > > Please find below my explanation. Here is a link to KIP 584: > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-584%3A+Versioning+scheme+for+features > > > . > > > > > > 1. '__data_version__' is the version of the finalized feature metadata > > > (i.e. actual ZK node contents), while the '__schema_version__' is the > > > version of the schema of the data persisted in ZK. These serve different > > > purposes. '__data_version__' is is useful mainly to clients during reads, > > > to differentiate between the 2 versions of eventually consistent > > 'finalized > > > features' metadata (i.e. larger metadata version is more recent). > > > '__schema_version__' provides an additional degree of flexibility, where > > if > > > we decide to change the schema for '/features' node in ZK (in the > > future), > > > then we can manage broker roll outs suitably (i.e. > > > serialization/deserialization of the ZK data can be handled safely). > > > > Hi Kowshik, > > > > If you're talking about a number that lets you know if data is more or > > less recent, we would typically call that an epoch, and not a version. For > > the ZK data structures, the word "version" is typically reserved for > > describing changes to the overall schema of the data that is written to > > ZooKeeper. We don't even really change the "version" of those schemas that > > much, since most changes are backwards-compatible. But we do include that > > version field just in case. > > > > I don't think we really need an epoch here, though, since we can just look > > at the broker epoch. Whenever the broker registers, its epoch will be > > greater than the previous broker epoch. And the newly registered data will > > take priority. This will be a lot simpler than adding a separate epoch > > system, I think. > > > > > > > > 2. Regarding admin client needing min and max information - you are > > right! > > > I've changed the KIP such that the Admin API also allows the user to read > > > 'supported features' from a specific broker. Please look at the section > > > "Admin API changes". > > > > Thanks. > > > > > > > > 3. Regarding the use of `long` vs `Long` - it was not deliberate. I've > > > improved the KIP to just use `long` at all places. > > > > Sounds good. > > > > > > > > 4. Regarding kafka.admin.FeatureCommand tool - you are right! I've > > updated > > > the KIP sketching the functionality provided by this tool, with some > > > examples. Please look at the section "Tooling support examples". > > > > > > Thank you! > > > > > > Thanks, Kowshik. > > > > cheers, > > Colin > > > > > > > > > > > Cheers, > > > Kowshik > > > > > > On Wed, Mar 25, 2020 at 11:31 PM Colin McCabe <cmcc...@apache.org> > > wrote: > > > > > > > Thanks, Kowshik, this looks good. > > > > > > > > In the "Schema" section, do we really need both __schema_version__ and > > > > __data_version__? Can we just have a single version field here? > > > > > > > > Shouldn't the Admin(Client) function have some way to get the min and > > max > > > > information that we're exposing as well? I guess we could have min, > > max, > > > > and current. Unrelated: is the use of Long rather than long deliberate > > > > here? > > > > > > > > It would be good to describe how the command line tool > > > > kafka.admin.FeatureCommand will work. For example the flags that it > > will > > > > take and the output that it will generate to STDOUT. > > > > > > > > cheers, > > > > Colin > > > > > > > > > > > > On Tue, Mar 24, 2020, at 17:08, Kowshik Prakasam wrote: > > > > > Hi all, > > > > > > > > > > I've opened KIP-584 <https://issues.apache.org/jira/browse/KIP-584> > > <https://issues.apache.org/jira/browse/KIP-584> > > > > > which > > > > > is intended to provide a versioning scheme for features. I'd like to > > use > > > > > this thread to discuss the same. I'd appreciate any feedback on this. > > > > > Here > > > > > is a link to KIP-584 <https://issues.apache.org/jira/browse/KIP-584> > > : > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-584%3A+Versioning+scheme+for+features > > > > > . > > > > > > > > > > Thank you! > > > > > > > > > > > > > > > Cheers, > > > > > Kowshik > > > > > > > > > > > > > > >
