Hi Jason,
Thanks for your comments.
1. Yes, the directory will still be based on the topic names.
LeaderAndIsrRequest is one of the few requests that will still contain the
topic name. So I think we have this covered. Sorry for confusion.
2. Part of the usage of the file is to have persistent storage of the topic
ID and use it to compare with the ID supplied in the LeaderAndIsr Request.
There is some discussion in the KIP about changes to the directory
structure, but I believe directory changes were considered to be out of
scope when the KIP was written.
3. I think this is a good point, but I again I wonder about the scope of
the KIP. Most of the changes mentioned in the KIP are for supporting topic
deletion and I believe that is why the produce request was listed under
future work.
4. This sounds like it might be a good solution, but I will need to discuss
more with KIP-500 folks to get the details right.
Thanks,
Justine
On Thu, Sep 24, 2020 at 12:30 PM Jason Gustafson <ja...@confluent.io> wrote:
> Hi Justine,
>
> Thanks for picking up this work. I have a few questions/comments:
>
> 1. It sounds like the directory structure is still going to be based on
> topic names. Do I have that right? One complication is that the
> LeaderAndIsr request does not include the topic name any longer. This means
> that a replica must wait for the UpdateMetadata request to arrive with the
> topic name mapping before it can create the directory. I wonder if it would
> be simpler to avoid assumptions on the ordering of UpdateMetadata and let
> LeaderAndIsr include the topic name as well. Feels like we are not saving
> that much by excluding it.
>
> 2. On a related note, it seems that the reason we have the partition
> metadata file is because we are not changing the directory structure. We
> need it so that we remember which directories map to which topic id. I
> think it would be useful to add some detail about why changing the
> directory layout was rejected. Basically trying to understand how much
> effort we are saving by skipping this step if we have to implement this
> additional bookkeeping. One concern I have for example is that the
> partition metadata file gets corrupt and then all bets are off as far as
> topic consistency.
>
> 3. Is it worth adding support now for topic ids in the `Produce` request
> now? Overhead is mentioned as one of the motivations and the best APIs to
> get savings from are `Produce` and `Fetch`.
>
> 4. When it comes to bootstrapping the metadata topic with respect to
> KIP-500, one suggestion would be to reserve a sentinel topic ID which can
> be used initially by new brokers. When the first controller is elected, it
> can assign a topicId to the metadata topic. This is a bit similar to how we
> were planning to generate the clusterId.
>
> Thanks,
> Jason
>
>
> On Thu, Sep 24, 2020 at 11:10 AM Jun Rao <j...@confluent.io> wrote:
>
> > Hi, Justine,
> >
> > Thanks for the updated KIP. A few more comments below.
> >
> > 30. {"name": "id", "type": "string", "doc": "version id"}}: The doc
> should
> > say UUID.
> >
> > 31. LeaderAndIsrResponse v5 and StopReplicaResponse v4 : It seems there
> is
> > no need to add topic_id at partitions level.
> >
> > 32. Regarding partition metadata file. Perhaps the key can be a single
> > word, sth like the following.
> > version: 0
> > topic_id: 46bdb63f-9e8d-4a38-bf7b-ee4eb2a794e4
> >
> > 33. Another tricky thing that I realized is how to support the metadata
> > topic introduced in KIP-595. It's a bit tricky to assign a UUID to the
> > metadata topic since we have a chicken-and-egg problem. The controller
> > needs to persist the UUID in the metadata topic in order to assign one
> > successfully, but the metadata topic is needed to elect a controller
> > (KIP-631). So, this probably needs a bit more thought.
> >
> > Jun
> >
> > On Thu, Sep 24, 2020 at 3:04 AM Ismael Juma <ism...@juma.me.uk> wrote:
> >
> > > Also, can we provide more details on how the Partition Metadata file
> will
> > > be used?
> > >
> > > Ismael
> > >
> > > On Thu, Sep 24, 2020 at 3:01 AM Ismael Juma <ism...@juma.me.uk> wrote:
> > >
> > > > Hi Justine,
> > > >
> > > > I think we need to update the "Rejected Alternatives" section to take
> > > into
> > > > account that the proposal now removes the topic name from the fetch
> > > > request. Also, if we are removing it from the Fetch request, does it
> > make
> > > > sense not to remove it from similar requests like ListOffsetRequest?
> > > >
> > > > Ismael
> > > >
> > > > On Thu, Sep 24, 2020 at 2:46 AM David Jacot <dja...@confluent.io>
> > wrote:
> > > >
> > > >> Hi Justine,
> > > >>
> > > >> Thanks for the KIP. I finally had time to read it :). It is a great
> > > >> improvement.
> > > >>
> > > >> I have a few comments/questions:
> > > >>
> > > >> 1. It seems that the schema of the StopReplicaRequest is slightly
> > > >> outdated.
> > > >> We
> > > >> did some changes as part of KIP-570. V3 is already organized by
> > topics.
> > > >>
> > > >> 2. I just want to make sure that I understand the reconciliation
> > > >> logic correctly. When
> > > >> an "INCREMENTAL" LeaderAndIsr Request is received, the broker will
> > also
> > > >> reconcile
> > > >> when the local uuid does not match the uuid in the request, right?
> In
> > > this
> > > >> case, the
> > > >> local log is staged for deletion.
> > > >>
> > > >> 3. In the documentation of the `delete.stale.topic.delay.ms`
> config,
> > it
> > > >> says "When a
> > > >> FULL LeaderAndIsrRequest is received..." but I suppose it applies to
> > > both
> > > >> types.
> > > >>
> > > >> Best,
> > > >> David
> > > >>
> > > >> On Thu, Sep 24, 2020 at 1:40 AM Justine Olshan <
> jols...@confluent.io>
> > > >> wrote:
> > > >>
> > > >> > Hi Jun,
> > > >> >
> > > >> > Thanks for the comments. I apologize for some of the typos and
> > > >> confusion.
> > > >> > I’ve updated the KIP to fix some of the issues you mentioned.
> > > >> >
> > > >> > 20.2 I’ve changed the type to String.
> > > >> > 20.1/3 I’ve updated the TopicZNode to fix formatting and reflect
> the
> > > >> latest
> > > >> > version before this change.
> > > >> >
> > > >> > 21. You are correct and I’ve removed this line. I’ve also added a
> > line
> > > >> > mentioning an IBP bump is necessary for migration
> > > >> >
> > > >> > 22. I think the wording was unclear but your summary is what was
> > > >> intended
> > > >> > by this line. I’ve updated to make this point more clear. “Remove
> > > >> deleted
> > > >> > topics from replicas by sending StopReplicaRequest V3 before the
> IBP
> > > >> bump
> > > >> > using the old logic, and using V4 and the new logic with topic IDs
> > > after
> > > >> > the IBP bump.”
> > > >> >
> > > >> > 23. I’ve removed the unspecified type from the KIP and mention
> that
> > > IBP
> > > >> > will be used to handle this request. “IBP will be used to
> determine
> > > >> whether
> > > >> > this new form of the request will be used. For older requests, we
> > will
> > > >> > ignore this field and default to previous behavior.”
> > > >> >
> > > >> > 24. I’ve fixed this typo.
> > > >> >
> > > >> > 25. I've added a topics at a higher level for LeaderAndIsrResponse
> > v5,
> > > >> > StopReplicaResponse v4. I've also changed StopReplicaRequest v4 to
> > > have
> > > >> > topics at a higher level.
> > > >> >
> > > >> > 26. I’ve updated forgotten_topics_data--added the topic ID and
> > removed
> > > >> the
> > > >> > topic name
> > > >> >
> > > >> > 27. I’ve decided on plain text, and I’ve added an example.
> > > >> >
> > > >> > 28. I’ve added this idea to future work.
> > > >> >
> > > >> > Thanks again for taking a look,
> > > >> >
> > > >> > Justine
> > > >> >
> > > >> > On Wed, Sep 23, 2020 at 10:28 AM Jun Rao <j...@confluent.io>
> wrote:
> > > >> >
> > > >> > > Hi, Justine,
> > > >> > >
> > > >> > > Thanks for the response. Made another pass. A few more comments
> > > below.
> > > >> > >
> > > >> > > 20. znode schema:
> > > >> > > 20.1 It seems that {"name": "version", "type": "int", "id":
> > "UUID",
> > > >> > "doc":
> > > >> > > "version id"} should be {"name": "version", "type": "int"},
> > {"name":
> > > >> > "id",
> > > >> > > "type": "UUID", "doc": "version id"}.
> > > >> > > 20.2 The znode format is JSON which doesn't have UUID type. So
> the
> > > >> type
> > > >> > > probably should be string?
> > > >> > > 20.3 Also, the existing format used seems outdated. It should
> have
> > > the
> > > >> > > following format.
> > > >> > > Json.encodeAsBytes(Map(
> > > >> > > "version" -> 2,
> > > >> > > "partitions" -> replicaAssignmentJson.asJava,
> > > >> > > "adding_replicas" -> addingReplicasAssignmentJson.asJava,
> > > >> > > "removing_replicas" ->
> removingReplicasAssignmentJson.asJava
> > > >> > > ).asJava)
> > > >> > > }
> > > >> > >
> > > >> > > 21. Migration: The KIP says "The migration process can take
> place
> > > >> without
> > > >> > > an inter-broker protocol bump, as the format stored in
> > > >> > > /brokers/topics/[topic] will be compatible with older broker
> > > >> versions."
> > > >> > > However, since we are bumping up the version of inter-broker
> > > >> requests, it
> > > >> > > seems that we need to use IBP for migration.
> > > >> > >
> > > >> > > 22. The KIP says "Remove deleted topics from replicas by sending
> > > >> > > StopReplicaRequest V3 for any topics which do not contain a
> topic
> > > ID,
> > > >> and
> > > >> > > V4 for any topics which do contain a topic ID.". However, if we
> > use
> > > >> IBP,
> > > >> > it
> > > >> > > seems that the controller will either send StopReplicaRequest V3
> > > >> > > or StopReplicaRequest V4, but never mixed V3 and V4 for
> different
> > > >> topics.
> > > >> > > Basically, before the IBP bump, V3 will be used. After the IBP
> > bump,
> > > >> > > topicId will be created and V4 will be used.
> > > >> > >
> > > >> > > 23. Given that we depend on IBP, do we still need "0
> UNSPECIFIED"
> > > >> > > in LeaderAndIsr?
> > > >> > >
> > > >> > > 24. LeaderAndIsrResponse v5 : It still has the topic field.
> > > >> > >
> > > >> > > 25. LeaderAndIsrResponse v5, StopReplicaResponse v4: Could we
> use
> > > this
> > > >> > > opportunity to organize the response in 2 levels, first by
> topic,
> > > >> then by
> > > >> > > partition, as most other requests/responses?
> > > >> > >
> > > >> > > 26. FetchRequest v13 : Should forgotten_topics_data use topicId
> > too?
> > > >> > >
> > > >> > > 27. "This file can either be plain text (key/value pairs) or
> > JSON."
> > > >> Have
> > > >> > we
> > > >> > > decided which one to use? Also, it would be helpful to provide
> an
> > > >> > example.
> > > >> > >
> > > >> > > 28. Future improvement: Another future improvement opportunity
> is
> > to
> > > >> use
> > > >> > > topicId in GroupMetadataManager.offsetCommitKey in the
> > offset_commit
> > > >> > topic.
> > > >> > > This may save some space.
> > > >> > >
> > > >> > > Thanks,
> > > >> > >
> > > >> > > Jun
> > > >> > >
> > > >> > > On Wed, Sep 23, 2020 at 8:50 AM Justine Olshan <
> > > jols...@confluent.io>
> > > >> > > wrote:
> > > >> > >
> > > >> > > > Hi Tom,
> > > >> > > >
> > > >> > > > Thanks for the comment. I think this is a really good idea and
> > it
> > > >> has
> > > >> > > been
> > > >> > > > added to the KIP under the newly added tooling section.
> > > >> > > >
> > > >> > > > Thanks again,
> > > >> > > > Justine
> > > >> > > >
> > > >> > > > On Wed, Sep 23, 2020 at 3:17 AM Tom Bentley <
> > tbent...@redhat.com>
> > > >> > wrote:
> > > >> > > >
> > > >> > > > > Hi Justine,
> > > >> > > > >
> > > >> > > > > I know you started the vote thread, but on re-reading the
> KIP
> > I
> > > >> > noticed
> > > >> > > > > that although the topic id is included in the
> MetadataResponse
> > > >> it's
> > > >> > not
> > > >> > > > > surfaced in the output from `kafka-topics.sh --describe`.
> > Maybe
> > > >> that
> > > >> > > was
> > > >> > > > > intentional because ids are intentionally not really
> something
> > > the
> > > >> > user
> > > >> > > > > should care deeply about, but it would also make life harder
> > for
> > > >> > anyone
> > > >> > > > > debugging Kafka and this would likely get worse the more
> topic
> > > ids
> > > >> > got
> > > >> > > > > rolled out across the protocols, clients etc. It seems
> likely
> > > that
> > > >> > > > > `kafka-topics.sh` will eventually need the ability to show
> the
> > > id
> > > >> of
> > > >> > a
> > > >> > > > > topic and perhaps find a topic name given an id. Is there
> any
> > > >> reason
> > > >> > > not
> > > >> > > > to
> > > >> > > > > implement that in this KIP?
> > > >> > > > >
> > > >> > > > > Many thanks,
> > > >> > > > >
> > > >> > > > > Tom
> > > >> > > > >
> > > >> > > > > On Mon, Sep 21, 2020 at 9:54 PM Justine Olshan <
> > > >> jols...@confluent.io
> > > >> > >
> > > >> > > > > wrote:
> > > >> > > > >
> > > >> > > > > > Hi all,
> > > >> > > > > >
> > > >> > > > > > After thinking about it, I've decided to remove the topic
> > name
> > > >> from
> > > >> > > the
> > > >> > > > > > Fetch Request and Response after all. Since there are so
> > many
> > > of
> > > >> > > these
> > > >> > > > > > requests per second, it is worth removing the extra
> > > information.
> > > >> > I've
> > > >> > > > > > updated the KIP to reflect this change.
> > > >> > > > > >
> > > >> > > > > > Please let me know if there is anything else we should
> > discuss
> > > >> > before
> > > >> > > > > > voting.
> > > >> > > > > >
> > > >> > > > > > Thank you,
> > > >> > > > > > Justine
> > > >> > > > > >
> > > >> > > > > > On Fri, Sep 18, 2020 at 9:46 AM Justine Olshan <
> > > >> > jols...@confluent.io
> > > >> > > >
> > > >> > > > > > wrote:
> > > >> > > > > >
> > > >> > > > > > > Hi Jun,
> > > >> > > > > > >
> > > >> > > > > > > I see what you are saying. For now we can remove the
> extra
> > > >> > > > information.
> > > >> > > > > > > I'll leave the option to add more fields to the file in
> > the
> > > >> > future.
> > > >> > > > The
> > > >> > > > > > KIP
> > > >> > > > > > > has been updated to reflect this change.
> > > >> > > > > > >
> > > >> > > > > > > Thanks,
> > > >> > > > > > > Justine
> > > >> > > > > > >
> > > >> > > > > > > On Fri, Sep 18, 2020 at 8:46 AM Jun Rao <
> j...@confluent.io
> > >
> > > >> > wrote:
> > > >> > > > > > >
> > > >> > > > > > >> Hi, Justine,
> > > >> > > > > > >>
> > > >> > > > > > >> Thanks for the reply.
> > > >> > > > > > >>
> > > >> > > > > > >> 13. If the log directory is the source of truth, it
> means
> > > >> that
> > > >> > the
> > > >> > > > > > >> redundant info in the metadata file will be ignored.
> Then
> > > the
> > > >> > > > question
> > > >> > > > > > is
> > > >> > > > > > >> why do we need to put the redundant info in the
> metadata
> > > file
> > > >> > now?
> > > >> > > > > > >>
> > > >> > > > > > >> Thanks,
> > > >> > > > > > >>
> > > >> > > > > > >> Jun
> > > >> > > > > > >>
> > > >> > > > > > >> On Thu, Sep 17, 2020 at 5:07 PM Justine Olshan <
> > > >> > > > jols...@confluent.io>
> > > >> > > > > > >> wrote:
> > > >> > > > > > >>
> > > >> > > > > > >> > Hi Jun,
> > > >> > > > > > >> > Thanks for the quick response!
> > > >> > > > > > >> >
> > > >> > > > > > >> > 12. I've decided to bump up the versions on the
> > requests
> > > >> and
> > > >> > > > updated
> > > >> > > > > > the
> > > >> > > > > > >> > KIP. I think it's good we thoroughly discussed the
> > > options
> > > >> > here,
> > > >> > > > so
> > > >> > > > > we
> > > >> > > > > > >> know
> > > >> > > > > > >> > we made a good choice. :)
> > > >> > > > > > >> >
> > > >> > > > > > >> > 13. This is an interesting situation. I think if this
> > > does
> > > >> > occur
> > > >> > > > we
> > > >> > > > > > >> should
> > > >> > > > > > >> > give a warning. I agree that it's hard to know the
> > source
> > > >> of
> > > >> > > truth
> > > >> > > > > for
> > > >> > > > > > >> sure
> > > >> > > > > > >> > since the directory or the file could be manually
> > > >> modified. I
> > > >> > > > guess
> > > >> > > > > > the
> > > >> > > > > > >> > directory could be used as the source of truth. To be
> > > >> honest,
> > > >> > > I'm
> > > >> > > > > not
> > > >> > > > > > >> > really sure what happens in kafka when the log
> > directory
> > > is
> > > >> > > > renamed
> > > >> > > > > > >> > manually in such a way. I'm also wondering if the
> > > >> situation is
> > > >> > > > > > >> recoverable
> > > >> > > > > > >> > in this scenario.
> > > >> > > > > > >> >
> > > >> > > > > > >> > Thanks,
> > > >> > > > > > >> > Justine
> > > >> > > > > > >> >
> > > >> > > > > > >> > On Thu, Sep 17, 2020 at 4:28 PM Jun Rao <
> > > j...@confluent.io>
> > > >> > > wrote:
> > > >> > > > > > >> >
> > > >> > > > > > >> > > Hi, Justine,
> > > >> > > > > > >> > >
> > > >> > > > > > >> > > Thanks for the reply.
> > > >> > > > > > >> > >
> > > >> > > > > > >> > > 12. I don't have a strong preference either.
> However,
> > > if
> > > >> we
> > > >> > > need
> > > >> > > > > IBP
> > > >> > > > > > >> > > anyway, maybe it's easier to just bump up the
> version
> > > for
> > > >> > all
> > > >> > > > > inter
> > > >> > > > > > >> > broker
> > > >> > > > > > >> > > requests and add the topic id field as a regular
> > > field. A
> > > >> > > > regular
> > > >> > > > > > >> field
> > > >> > > > > > >> > is
> > > >> > > > > > >> > > a bit more concise in wire transfer than a flexible
> > > >> field.
> > > >> > > > > > >> > >
> > > >> > > > > > >> > > 13. The confusion that I was referring to is
> between
> > > the
> > > >> > topic
> > > >> > > > > name
> > > >> > > > > > >> and
> > > >> > > > > > >> > > partition number between the log dir and the
> metadata
> > > >> file.
> > > >> > > For
> > > >> > > > > > >> example,
> > > >> > > > > > >> > if
> > > >> > > > > > >> > > the log dir is topicA-1 and the metadata file in it
> > has
> > > >> > topicB
> > > >> > > > and
> > > >> > > > > > >> > > partition 0 (say due to a bug or manual
> > modification),
> > > >> which
> > > >> > > one
> > > >> > > > > do
> > > >> > > > > > we
> > > >> > > > > > >> > use
> > > >> > > > > > >> > > as the source of truth?
> > > >> > > > > > >> > >
> > > >> > > > > > >> > > Jun
> > > >> > > > > > >> > >
> > > >> > > > > > >> > > On Thu, Sep 17, 2020 at 3:43 PM Justine Olshan <
> > > >> > > > > > jols...@confluent.io>
> > > >> > > > > > >> > > wrote:
> > > >> > > > > > >> > >
> > > >> > > > > > >> > > > Hi Jun,
> > > >> > > > > > >> > > > Thanks for the comments.
> > > >> > > > > > >> > > >
> > > >> > > > > > >> > > > 12. I bumped the LeaderAndIsrRequest because I
> > > removed
> > > >> the
> > > >> > > > topic
> > > >> > > > > > >> name
> > > >> > > > > > >> > > field
> > > >> > > > > > >> > > > in the response. It may be possible to avoid
> > bumping
> > > >> the
> > > >> > > > version
> > > >> > > > > > >> > without
> > > >> > > > > > >> > > > that change, but I may be missing something.
> > > >> > > > > > >> > > > I believe StopReplica is actually on version 3
> now,
> > > but
> > > >> > > > because
> > > >> > > > > > >> > version 2
> > > >> > > > > > >> > > > is flexible, I kept that listed as version 2 on
> the
> > > KIP
> > > >> > > page.
> > > >> > > > > > >> However,
> > > >> > > > > > >> > > you
> > > >> > > > > > >> > > > may be right in that we may need to bump the
> > version
> > > on
> > > >> > > > > > StopReplica
> > > >> > > > > > >> to
> > > >> > > > > > >> > > deal
> > > >> > > > > > >> > > > with deletion differently as mentioned above. I
> > don't
> > > >> know
> > > >> > > if
> > > >> > > > I
> > > >> > > > > > >> have a
> > > >> > > > > > >> > > big
> > > >> > > > > > >> > > > preference over used tagged fields or not.
> > > >> > > > > > >> > > >
> > > >> > > > > > >> > > > 13. I was thinking that in the case where the
> file
> > > and
> > > >> the
> > > >> > > > > request
> > > >> > > > > > >> > topic
> > > >> > > > > > >> > > > ids don't match, it means that the broker's
> > topic/the
> > > >> one
> > > >> > in
> > > >> > > > the
> > > >> > > > > > >> file
> > > >> > > > > > >> > has
> > > >> > > > > > >> > > > been deleted. In that case, we would need to
> delete
> > > the
> > > >> > old
> > > >> > > > > topic
> > > >> > > > > > >> and
> > > >> > > > > > >> > > start
> > > >> > > > > > >> > > > receiving the new version. If the topic name were
> > to
> > > >> > change,
> > > >> > > > but
> > > >> > > > > > the
> > > >> > > > > > >> > ids
> > > >> > > > > > >> > > > still matched, the file would also need to
> update.
> > > Am I
> > > >> > > > missing
> > > >> > > > > a
> > > >> > > > > > >> case
> > > >> > > > > > >> > > > where the file would be correct and not the
> > request?
> > > >> > > > > > >> > > >
> > > >> > > > > > >> > > > Thanks,
> > > >> > > > > > >> > > > Justine
> > > >> > > > > > >> > > >
> > > >> > > > > > >> > > > On Thu, Sep 17, 2020 at 3:18 PM Jun Rao <
> > > >> j...@confluent.io
> > > >> > >
> > > >> > > > > wrote:
> > > >> > > > > > >> > > >
> > > >> > > > > > >> > > > > Hi, Justine,
> > > >> > > > > > >> > > > >
> > > >> > > > > > >> > > > > Thanks for the reply. A couple of more comments
> > > >> below.
> > > >> > > > > > >> > > > >
> > > >> > > > > > >> > > > > 12. ListOffset and OffsetForLeader currently
> > don't
> > > >> > support
> > > >> > > > > > >> flexible
> > > >> > > > > > >> > > > fields.
> > > >> > > > > > >> > > > > So, we have to bump up the version number and
> use
> > > >> IBP at
> > > >> > > > least
> > > >> > > > > > for
> > > >> > > > > > >> > > these
> > > >> > > > > > >> > > > > two requests. Note that it seems 2.7.0 will
> > require
> > > >> IBP
> > > >> > > > anyway
> > > >> > > > > > >> > because
> > > >> > > > > > >> > > of
> > > >> > > > > > >> > > > > changes in KAFKA-10435. Also, it seems that the
> > > >> version
> > > >> > > for
> > > >> > > > > > >> > > > > LeaderAndIsrRequest and StopReplica are bumped
> > even
> > > >> > though
> > > >> > > > we
> > > >> > > > > > only
> > > >> > > > > > >> > > added
> > > >> > > > > > >> > > > a
> > > >> > > > > > >> > > > > tagged field. But since IBP is needed anyway,
> we
> > > may
> > > >> > want
> > > >> > > to
> > > >> > > > > > >> revisit
> > > >> > > > > > >> > > the
> > > >> > > > > > >> > > > > overall tagged field choice.
> > > >> > > > > > >> > > > >
> > > >> > > > > > >> > > > > 13. The only downside is the potential
> confusion
> > on
> > > >> > which
> > > >> > > > one
> > > >> > > > > is
> > > >> > > > > > >> the
> > > >> > > > > > >> > > > source
> > > >> > > > > > >> > > > > of truth if they don't match. Another option is
> > to
> > > >> > include
> > > >> > > > > those
> > > >> > > > > > >> > fields
> > > >> > > > > > >> > > > in
> > > >> > > > > > >> > > > > the metadata file when we actually change the
> > > >> directory
> > > >> > > > > > structure.
> > > >> > > > > > >> > > > >
> > > >> > > > > > >> > > > > Thanks,
> > > >> > > > > > >> > > > >
> > > >> > > > > > >> > > > > Jun
> > > >> > > > > > >> > > > >
> > > >> > > > > > >> > > > > On Thu, Sep 17, 2020 at 2:01 PM Justine Olshan
> <
> > > >> > > > > > >> jols...@confluent.io
> > > >> > > > > > >> > >
> > > >> > > > > > >> > > > > wrote:
> > > >> > > > > > >> > > > >
> > > >> > > > > > >> > > > > > Hello all,
> > > >> > > > > > >> > > > > >
> > > >> > > > > > >> > > > > > I've thought some more about removing the
> topic
> > > >> name
> > > >> > > field
> > > >> > > > > > from
> > > >> > > > > > >> > some
> > > >> > > > > > >> > > of
> > > >> > > > > > >> > > > > the
> > > >> > > > > > >> > > > > > requests. On closer inspection of the
> > > >> > > requests/responses,
> > > >> > > > it
> > > >> > > > > > >> seems
> > > >> > > > > > >> > > that
> > > >> > > > > > >> > > > > the
> > > >> > > > > > >> > > > > > internal changes would be much larger than I
> > > >> expected.
> > > >> > > > Some
> > > >> > > > > > >> > protocols
> > > >> > > > > > >> > > > > > involve clients, so they would require
> changes
> > > too.
> > > >> > I'm
> > > >> > > > > > thinking
> > > >> > > > > > >> > that
> > > >> > > > > > >> > > > for
> > > >> > > > > > >> > > > > > now, removing the topic name from these
> > requests
> > > >> and
> > > >> > > > > responses
> > > >> > > > > > >> are
> > > >> > > > > > >> > > out
> > > >> > > > > > >> > > > of
> > > >> > > > > > >> > > > > > scope.
> > > >> > > > > > >> > > > > >
> > > >> > > > > > >> > > > > > I have decided to just keep the change
> > > >> > > > LeaderAndIsrResponse
> > > >> > > > > to
> > > >> > > > > > >> > remove
> > > >> > > > > > >> > > > the
> > > >> > > > > > >> > > > > > topic name, and have updated the KIP to
> reflect
> > > >> this
> > > >> > > > > change. I
> > > >> > > > > > >> have
> > > >> > > > > > >> > > > also
> > > >> > > > > > >> > > > > > mentioned the other requests and responses in
> > > >> future
> > > >> > > work.
> > > >> > > > > > >> > > > > >
> > > >> > > > > > >> > > > > > I'm hoping to start the voting process soon,
> so
> > > >> let me
> > > >> > > > know
> > > >> > > > > if
> > > >> > > > > > >> > there
> > > >> > > > > > >> > > is
> > > >> > > > > > >> > > > > > anything else we should discuss.
> > > >> > > > > > >> > > > > >
> > > >> > > > > > >> > > > > > Thank you,
> > > >> > > > > > >> > > > > > Justine
> > > >> > > > > > >> > > > > >
> > > >> > > > > > >> > > > > > On Tue, Sep 15, 2020 at 3:57 PM Justine
> Olshan
> > <
> > > >> > > > > > >> > jols...@confluent.io
> > > >> > > > > > >> > > >
> > > >> > > > > > >> > > > > > wrote:
> > > >> > > > > > >> > > > > >
> > > >> > > > > > >> > > > > > > Hello again,
> > > >> > > > > > >> > > > > > > To follow up on some of the other comments:
> > > >> > > > > > >> > > > > > >
> > > >> > > > > > >> > > > > > > 10/11) We can remove the topic name from
> > these
> > > >> > > > > > >> > requests/responses,
> > > >> > > > > > >> > > > and
> > > >> > > > > > >> > > > > > > that means we will just have to make a few
> > > >> internal
> > > >> > > > > changes
> > > >> > > > > > to
> > > >> > > > > > >> > make
> > > >> > > > > > >> > > > > > > partitions accessible by topic id and
> > > partition.
> > > >> I
> > > >> > can
> > > >> > > > > > update
> > > >> > > > > > >> the
> > > >> > > > > > >> > > KIP
> > > >> > > > > > >> > > > > to
> > > >> > > > > > >> > > > > > > remove them unless anyone thinks they
> should
> > > >> stay.
> > > >> > > > > > >> > > > > > >
> > > >> > > > > > >> > > > > > > 12) Addressed in the previous email. I've
> > > updated
> > > >> > the
> > > >> > > > KIP
> > > >> > > > > to
> > > >> > > > > > >> > > include
> > > >> > > > > > >> > > > > > > tagged fields for the requests and
> responses.
> > > >> (More
> > > >> > on
> > > >> > > > > that
> > > >> > > > > > >> > below)
> > > >> > > > > > >> > > > > > >
> > > >> > > > > > >> > > > > > > 13) I think part of the idea for including
> > this
> > > >> > > > > information
> > > >> > > > > > >> is to
> > > >> > > > > > >> > > > > prepare
> > > >> > > > > > >> > > > > > > for future changes. Perhaps the directory
> > > >> structure
> > > >> > > > might
> > > >> > > > > > >> change
> > > >> > > > > > >> > > from
> > > >> > > > > > >> > > > > > > topicName_partitionNumber to something like
> > > >> > > > > > >> > > topicID_partitionNumber.
> > > >> > > > > > >> > > > > Then
> > > >> > > > > > >> > > > > > > it would be useful to have the topic name
> in
> > > the
> > > >> > file
> > > >> > > > > since
> > > >> > > > > > it
> > > >> > > > > > >> > > would
> > > >> > > > > > >> > > > > not
> > > >> > > > > > >> > > > > > be
> > > >> > > > > > >> > > > > > > in the directory structure. Supporting
> topic
> > > >> renames
> > > >> > > > might
> > > >> > > > > > be
> > > >> > > > > > >> > > easier
> > > >> > > > > > >> > > > if
> > > >> > > > > > >> > > > > > the
> > > >> > > > > > >> > > > > > > other fields are included. Would there be
> any
> > > >> > > downsides
> > > >> > > > to
> > > >> > > > > > >> > > including
> > > >> > > > > > >> > > > > this
> > > >> > > > > > >> > > > > > > information?
> > > >> > > > > > >> > > > > > >
> > > >> > > > > > >> > > > > > > 14) Yes, we would need to copy the
> partition
> > > >> > metadata
> > > >> > > > > file
> > > >> > > > > > in
> > > >> > > > > > >> > this
> > > >> > > > > > >> > > > > > > process. I've updated the KIP to include
> > this.
> > > >> > > > > > >> > > > > > >
> > > >> > > > > > >> > > > > > > 15) I believe Lucas meant v1 and v2 here.
> He
> > > was
> > > >> > > > referring
> > > >> > > > > > to
> > > >> > > > > > >> how
> > > >> > > > > > >> > > the
> > > >> > > > > > >> > > > > > > requests would fall under different IBP and
> > > meant
> > > >> > that
> > > >> > > > > older
> > > >> > > > > > >> > > brokers
> > > >> > > > > > >> > > > > > would
> > > >> > > > > > >> > > > > > > have to use the older version of the
> request
> > > and
> > > >> the
> > > >> > > > > > existing
> > > >> > > > > > >> > topic
> > > >> > > > > > >> > > > > > > deletion process. At first, it seemed like
> > > tagged
> > > >> > > fields
> > > >> > > > > > would
> > > >> > > > > > >> > > > resolve
> > > >> > > > > > >> > > > > > > the IBP issue. However, we may need IBP for
> > > this
> > > >> > > request
> > > >> > > > > > after
> > > >> > > > > > >> > all
> > > >> > > > > > >> > > > > since
> > > >> > > > > > >> > > > > > > the controller handles the topic deletion
> > > >> > differently
> > > >> > > > > > >> depending
> > > >> > > > > > >> > on
> > > >> > > > > > >> > > > the
> > > >> > > > > > >> > > > > > IBP
> > > >> > > > > > >> > > > > > > version. In an older version, we can't just
> > > send
> > > >> a
> > > >> > > > > > StopReplica
> > > >> > > > > > >> > > delete
> > > >> > > > > > >> > > > > the
> > > >> > > > > > >> > > > > > > topic immediately like we'd want to for
> this
> > > KIP.
> > > >> > > > > > >> > > > > > >
> > > >> > > > > > >> > > > > > > This makes me wonder if we want tagged
> fields
> > > on
> > > >> all
> > > >> > > the
> > > >> > > > > > >> requests
> > > >> > > > > > >> > > > after
> > > >> > > > > > >> > > > > > > all. Let me know your thoughts!
> > > >> > > > > > >> > > > > > >
> > > >> > > > > > >> > > > > > > Justine
> > > >> > > > > > >> > > > > > >
> > > >> > > > > > >> > > > > > > On Tue, Sep 15, 2020 at 1:03 PM Justine
> > Olshan
> > > <
> > > >> > > > > > >> > > jols...@confluent.io
> > > >> > > > > > >> > > > >
> > > >> > > > > > >> > > > > > > wrote:
> > > >> > > > > > >> > > > > > >
> > > >> > > > > > >> > > > > > >> Hi all,
> > > >> > > > > > >> > > > > > >> Jun brought up a good point in his last
> > email
> > > >> about
> > > >> > > > > tagged
> > > >> > > > > > >> > fields,
> > > >> > > > > > >> > > > and
> > > >> > > > > > >> > > > > > >> I've updated the KIP to reflect that the
> > > >> changes to
> > > >> > > > > > requests
> > > >> > > > > > >> and
> > > >> > > > > > >> > > > > > responses
> > > >> > > > > > >> > > > > > >> will be in the form of tagged fields to
> > avoid
> > > >> > > changing
> > > >> > > > > IBP.
> > > >> > > > > > >> > > > > > >>
> > > >> > > > > > >> > > > > > >> Jun: I plan on sending a followup email to
> > > >> address
> > > >> > > some
> > > >> > > > > of
> > > >> > > > > > >> the
> > > >> > > > > > >> > > other
> > > >> > > > > > >> > > > > > >> points.
> > > >> > > > > > >> > > > > > >>
> > > >> > > > > > >> > > > > > >> Thanks,
> > > >> > > > > > >> > > > > > >> Justine
> > > >> > > > > > >> > > > > > >>
> > > >> > > > > > >> > > > > > >> On Mon, Sep 14, 2020 at 4:25 PM Jun Rao <
> > > >> > > > > j...@confluent.io>
> > > >> > > > > > >> > wrote:
> > > >> > > > > > >> > > > > > >>
> > > >> > > > > > >> > > > > > >>> Hi, Justine,
> > > >> > > > > > >> > > > > > >>>
> > > >> > > > > > >> > > > > > >>> Thanks for the updated KIP. A few
> comments
> > > >> below.
> > > >> > > > > > >> > > > > > >>>
> > > >> > > > > > >> > > > > > >>> 10. LeaderAndIsr Response: Do we need the
> > > topic
> > > >> > > name?
> > > >> > > > > > >> > > > > > >>>
> > > >> > > > > > >> > > > > > >>> 11. For the changed request/response,
> other
> > > >> than
> > > >> > > > > > >> LeaderAndIsr,
> > > >> > > > > > >> > > > > > >>> UpdateMetadata, Metadata, do we need to
> > > include
> > > >> > the
> > > >> > > > > topic
> > > >> > > > > > >> name?
> > > >> > > > > > >> > > > > > >>>
> > > >> > > > > > >> > > > > > >>> 12. It seems that upgrades don't require
> > IBP.
> > > >> Does
> > > >> > > > that
> > > >> > > > > > mean
> > > >> > > > > > >> > the
> > > >> > > > > > >> > > > new
> > > >> > > > > > >> > > > > > >>> fields
> > > >> > > > > > >> > > > > > >>> in all the request/response are added as
> > > tagged
> > > >> > > fields
> > > >> > > > > > >> without
> > > >> > > > > > >> > > > > bumping
> > > >> > > > > > >> > > > > > up
> > > >> > > > > > >> > > > > > >>> the request version? It would be useful
> to
> > > make
> > > >> > that
> > > >> > > > > > clear.
> > > >> > > > > > >> > > > > > >>>
> > > >> > > > > > >> > > > > > >>> 13. Partition Metadata file: Do we need
> to
> > > >> include
> > > >> > > the
> > > >> > > > > > topic
> > > >> > > > > > >> > name
> > > >> > > > > > >> > > > and
> > > >> > > > > > >> > > > > > the
> > > >> > > > > > >> > > > > > >>> partition id since they are implied in
> the
> > > >> > directory
> > > >> > > > > name?
> > > >> > > > > > >> > > > > > >>>
> > > >> > > > > > >> > > > > > >>> 14. In the JBOD mode, we support moving a
> > > >> > > partition's
> > > >> > > > > data
> > > >> > > > > > >> from
> > > >> > > > > > >> > > one
> > > >> > > > > > >> > > > > > disk
> > > >> > > > > > >> > > > > > >>> to
> > > >> > > > > > >> > > > > > >>> another. Will the new partition metadata
> > file
> > > >> be
> > > >> > > > copied
> > > >> > > > > > >> during
> > > >> > > > > > >> > > that
> > > >> > > > > > >> > > > > > >>> process?
> > > >> > > > > > >> > > > > > >>>
> > > >> > > > > > >> > > > > > >>> 15. The KIP says "Remove deleted topics
> > from
> > > >> > > replicas
> > > >> > > > by
> > > >> > > > > > >> > sending
> > > >> > > > > > >> > > > > > >>> StopReplicaRequest V2 for any topics
> which
> > do
> > > >> not
> > > >> > > > > contain
> > > >> > > > > > a
> > > >> > > > > > >> > topic
> > > >> > > > > > >> > > > ID,
> > > >> > > > > > >> > > > > > and
> > > >> > > > > > >> > > > > > >>> V3 for any topics which do contain a
> topic
> > > >> ID.".
> > > >> > > > > However,
> > > >> > > > > > it
> > > >> > > > > > >> > > seems
> > > >> > > > > > >> > > > > the
> > > >> > > > > > >> > > > > > >>> updated controller will create all
> missing
> > > >> topic
> > > >> > IDs
> > > >> > > > > first
> > > >> > > > > > >> > before
> > > >> > > > > > >> > > > > doing
> > > >> > > > > > >> > > > > > >>> other actions. So, is StopReplicaRequest
> V2
> > > >> > needed?
> > > >> > > > > > >> > > > > > >>>
> > > >> > > > > > >> > > > > > >>> Jun
> > > >> > > > > > >> > > > > > >>>
> > > >> > > > > > >> > > > > > >>> On Fri, Sep 11, 2020 at 10:31 AM John
> > > Roesler <
> > > >> > > > > > >> > > vvcep...@apache.org
> > > >> > > > > > >> > > > >
> > > >> > > > > > >> > > > > > >>> wrote:
> > > >> > > > > > >> > > > > > >>>
> > > >> > > > > > >> > > > > > >>> > Thanks, Justine!
> > > >> > > > > > >> > > > > > >>> >
> > > >> > > > > > >> > > > > > >>> > Your response seems compelling to me.
> > > >> > > > > > >> > > > > > >>> >
> > > >> > > > > > >> > > > > > >>> > -John
> > > >> > > > > > >> > > > > > >>> >
> > > >> > > > > > >> > > > > > >>> > On Fri, 2020-09-11 at 10:17 -0700,
> > Justine
> > > >> > Olshan
> > > >> > > > > wrote:
> > > >> > > > > > >> > > > > > >>> > > Hello all,
> > > >> > > > > > >> > > > > > >>> > > Thanks for continuing the
> discussion! I
> > > >> have a
> > > >> > > few
> > > >> > > > > > >> > responses
> > > >> > > > > > >> > > to
> > > >> > > > > > >> > > > > > your
> > > >> > > > > > >> > > > > > >>> > points.
> > > >> > > > > > >> > > > > > >>> > >
> > > >> > > > > > >> > > > > > >>> > > Tom: You are correct in that this KIP
> > has
> > > >> not
> > > >> > > > > > mentioned
> > > >> > > > > > >> the
> > > >> > > > > > >> > > > > > >>> > > DeleteTopicsRequest. I think that
> this
> > > >> would
> > > >> > be
> > > >> > > > out
> > > >> > > > > of
> > > >> > > > > > >> > scope
> > > >> > > > > > >> > > > for
> > > >> > > > > > >> > > > > > >>> now, but
> > > >> > > > > > >> > > > > > >>> > > may be something worth adding in the
> > > >> future.
> > > >> > > > > > >> > > > > > >>> > >
> > > >> > > > > > >> > > > > > >>> > > John: We did consider sequence ids,
> but
> > > >> there
> > > >> > > are
> > > >> > > > a
> > > >> > > > > > few
> > > >> > > > > > >> > > reasons
> > > >> > > > > > >> > > > > to
> > > >> > > > > > >> > > > > > >>> favor
> > > >> > > > > > >> > > > > > >>> > > UUIDs. There are several cases where
> > > topics
> > > >> > from
> > > >> > > > > > >> different
> > > >> > > > > > >> > > > > clusters
> > > >> > > > > > >> > > > > > >>> may
> > > >> > > > > > >> > > > > > >>> > > interact now and in the future. For
> > > >> example,
> > > >> > > > Mirror
> > > >> > > > > > >> Maker 2
> > > >> > > > > > >> > > may
> > > >> > > > > > >> > > > > > >>> benefit
> > > >> > > > > > >> > > > > > >>> > > from being able to detect when a
> > cluster
> > > >> being
> > > >> > > > > > mirrored
> > > >> > > > > > >> is
> > > >> > > > > > >> > > > > deleted
> > > >> > > > > > >> > > > > > >>> and
> > > >> > > > > > >> > > > > > >>> > > recreated and globally unique
> > identifiers
> > > >> > would
> > > >> > > > make
> > > >> > > > > > >> > > resolving
> > > >> > > > > > >> > > > > > issues
> > > >> > > > > > >> > > > > > >>> > > easier than sequence IDs which may
> > > collide
> > > >> > > between
> > > >> > > > > > >> > clusters.
> > > >> > > > > > >> > > > > > KIP-405
> > > >> > > > > > >> > > > > > >>> > > (tiered storage) will also benefit
> from
> > > >> > globally
> > > >> > > > > > unique
> > > >> > > > > > >> IDs
> > > >> > > > > > >> > > as
> > > >> > > > > > >> > > > > > shared
> > > >> > > > > > >> > > > > > >>> > > buckets may be used between clusters.
> > > >> > > > > > >> > > > > > >>> > >
> > > >> > > > > > >> > > > > > >>> > > Globally unique IDs would also make
> > > >> > > functionality
> > > >> > > > > like
> > > >> > > > > > >> > moving
> > > >> > > > > > >> > > > > > topics
> > > >> > > > > > >> > > > > > >>> > > between disparate clusters easier in
> > the
> > > >> > future,
> > > >> > > > > > >> simplify
> > > >> > > > > > >> > any
> > > >> > > > > > >> > > > > > future
> > > >> > > > > > >> > > > > > >>> > > implementations of backups and
> > restores,
> > > >> and
> > > >> > > more.
> > > >> > > > > In
> > > >> > > > > > >> > > general,
> > > >> > > > > > >> > > > > > >>> unique IDs
> > > >> > > > > > >> > > > > > >>> > > would ensure that the source cluster
> > > >> topics do
> > > >> > > not
> > > >> > > > > > >> conflict
> > > >> > > > > > >> > > > with
> > > >> > > > > > >> > > > > > the
> > > >> > > > > > >> > > > > > >>> > > destination cluster topics.
> > > >> > > > > > >> > > > > > >>> > >
> > > >> > > > > > >> > > > > > >>> > > If we were to use sequence ids, we
> > would
> > > >> need
> > > >> > > > > > >> sufficiently
> > > >> > > > > > >> > > > large
> > > >> > > > > > >> > > > > > >>> cluster
> > > >> > > > > > >> > > > > > >>> > > ids to be stored with the topic
> > > >> identifiers or
> > > >> > > we
> > > >> > > > > run
> > > >> > > > > > >> the
> > > >> > > > > > >> > > risk
> > > >> > > > > > >> > > > of
> > > >> > > > > > >> > > > > > >>> > > collisions. This will give up any
> > > >> advantage in
> > > >> > > > > > >> compactness
> > > >> > > > > > >> > > that
> > > >> > > > > > >> > > > > > >>> sequence
> > > >> > > > > > >> > > > > > >>> > > numbers may bring. Given these
> > > advantages I
> > > >> > > think
> > > >> > > > it
> > > >> > > > > > >> makes
> > > >> > > > > > >> > > > sense
> > > >> > > > > > >> > > > > to
> > > >> > > > > > >> > > > > > >>> use
> > > >> > > > > > >> > > > > > >>> > > UUIDs.
> > > >> > > > > > >> > > > > > >>> > >
> > > >> > > > > > >> > > > > > >>> > > Gokul: This is an interesting idea,
> but
> > > >> this
> > > >> > is
> > > >> > > a
> > > >> > > > > > >> breaking
> > > >> > > > > > >> > > > > change.
> > > >> > > > > > >> > > > > > >>> Out of
> > > >> > > > > > >> > > > > > >>> > > scope for now, but maybe worth
> > discussing
> > > >> in
> > > >> > the
> > > >> > > > > > future.
> > > >> > > > > > >> > > > > > >>> > >
> > > >> > > > > > >> > > > > > >>> > > Hope this explains some of the
> > decisions,
> > > >> > > > > > >> > > > > > >>> > >
> > > >> > > > > > >> > > > > > >>> > > Justine
> > > >> > > > > > >> > > > > > >>> > >
> > > >> > > > > > >> > > > > > >>> > >
> > > >> > > > > > >> > > > > > >>> > >
> > > >> > > > > > >> > > > > > >>> > > On Fri, Sep 11, 2020 at 8:27 AM Gokul
> > > >> Ramanan
> > > >> > > > > > >> Subramanian <
> > > >> > > > > > >> > > > > > >>> > > gokul24...@gmail.com> wrote:
> > > >> > > > > > >> > > > > > >>> > >
> > > >> > > > > > >> > > > > > >>> > > > Hi.
> > > >> > > > > > >> > > > > > >>> > > >
> > > >> > > > > > >> > > > > > >>> > > > Thanks for the KIP.
> > > >> > > > > > >> > > > > > >>> > > >
> > > >> > > > > > >> > > > > > >>> > > > Have you thought about whether it
> > makes
> > > >> > sense
> > > >> > > to
> > > >> > > > > > >> support
> > > >> > > > > > >> > > > > > >>> authorizing a
> > > >> > > > > > >> > > > > > >>> > > > principal for a topic ID rather
> than
> > a
> > > >> topic
> > > >> > > > name
> > > >> > > > > to
> > > >> > > > > > >> > > achieve
> > > >> > > > > > >> > > > > > >>> tighter
> > > >> > > > > > >> > > > > > >>> > > > security?
> > > >> > > > > > >> > > > > > >>> > > >
> > > >> > > > > > >> > > > > > >>> > > > Or is the topic ID fundamentally an
> > > >> internal
> > > >> > > > > detail
> > > >> > > > > > >> > similar
> > > >> > > > > > >> > > > to
> > > >> > > > > > >> > > > > > >>> epochs
> > > >> > > > > > >> > > > > > >>> > used
> > > >> > > > > > >> > > > > > >>> > > > in a bunch of other places in
> Kafka?
> > > >> > > > > > >> > > > > > >>> > > >
> > > >> > > > > > >> > > > > > >>> > > > Thanks.
> > > >> > > > > > >> > > > > > >>> > > >
> > > >> > > > > > >> > > > > > >>> > > > On Fri, Sep 11, 2020 at 4:06 PM
> John
> > > >> > Roesler <
> > > >> > > > > > >> > > > > > vvcep...@apache.org>
> > > >> > > > > > >> > > > > > >>> > wrote:
> > > >> > > > > > >> > > > > > >>> > > >
> > > >> > > > > > >> > > > > > >>> > > > > Hello Justine,
> > > >> > > > > > >> > > > > > >>> > > > >
> > > >> > > > > > >> > > > > > >>> > > > > Thanks for the KIP!
> > > >> > > > > > >> > > > > > >>> > > > >
> > > >> > > > > > >> > > > > > >>> > > > > I happen to have been confronted
> > > >> recently
> > > >> > > with
> > > >> > > > > the
> > > >> > > > > > >> need
> > > >> > > > > > >> > > to
> > > >> > > > > > >> > > > > keep
> > > >> > > > > > >> > > > > > >>> > track of
> > > >> > > > > > >> > > > > > >>> > > > a
> > > >> > > > > > >> > > > > > >>> > > > > large number of topics as
> compactly
> > > as
> > > >> > > > > possible. I
> > > >> > > > > > >> was
> > > >> > > > > > >> > > > going
> > > >> > > > > > >> > > > > to
> > > >> > > > > > >> > > > > > >>> come
> > > >> > > > > > >> > > > > > >>> > up
> > > >> > > > > > >> > > > > > >>> > > > > with some way to dictionary
> encode
> > > the
> > > >> > topic
> > > >> > > > > names
> > > >> > > > > > >> as
> > > >> > > > > > >> > > > > integers,
> > > >> > > > > > >> > > > > > >>> but
> > > >> > > > > > >> > > > > > >>> > this
> > > >> > > > > > >> > > > > > >>> > > > > seems much better!
> > > >> > > > > > >> > > > > > >>> > > > >
> > > >> > > > > > >> > > > > > >>> > > > > Apologies if this has been raised
> > > >> before,
> > > >> > > but
> > > >> > > > > I’m
> > > >> > > > > > >> > > wondering
> > > >> > > > > > >> > > > > > >>> about the
> > > >> > > > > > >> > > > > > >>> > > > > choice of UUID vs sequence number
> > for
> > > >> the
> > > >> > > ids.
> > > >> > > > > > >> > Typically,
> > > >> > > > > > >> > > > > I’ve
> > > >> > > > > > >> > > > > > >>> seen
> > > >> > > > > > >> > > > > > >>> > UUIDs
> > > >> > > > > > >> > > > > > >>> > > > > in two situations:
> > > >> > > > > > >> > > > > > >>> > > > > 1. When processes need to
> generate
> > > >> > > > non-colliding
> > > >> > > > > > >> > > > identifiers
> > > >> > > > > > >> > > > > > >>> without
> > > >> > > > > > >> > > > > > >>> > > > > coordination.
> > > >> > > > > > >> > > > > > >>> > > > > 2. When the identifier needs to
> be
> > > >> > > > “universally
> > > >> > > > > > >> > unique”;
> > > >> > > > > > >> > > > > I.e.,
> > > >> > > > > > >> > > > > > >>> the
> > > >> > > > > > >> > > > > > >>> > > > > identifier needs to distinguish
> the
> > > >> entity
> > > >> > > > from
> > > >> > > > > > all
> > > >> > > > > > >> > other
> > > >> > > > > > >> > > > > > >>> entities
> > > >> > > > > > >> > > > > > >>> > that
> > > >> > > > > > >> > > > > > >>> > > > > could ever exist. This is useful
> in
> > > >> cases
> > > >> > > > where
> > > >> > > > > > >> > entities
> > > >> > > > > > >> > > > from
> > > >> > > > > > >> > > > > > all
> > > >> > > > > > >> > > > > > >>> > kinds
> > > >> > > > > > >> > > > > > >>> > > > of
> > > >> > > > > > >> > > > > > >>> > > > > systems get mixed together, such
> as
> > > >> when
> > > >> > > > dumping
> > > >> > > > > > >> logs
> > > >> > > > > > >> > > from
> > > >> > > > > > >> > > > > all
> > > >> > > > > > >> > > > > > >>> > processes
> > > >> > > > > > >> > > > > > >>> > > > in
> > > >> > > > > > >> > > > > > >>> > > > > a company into a common system.
> > > >> > > > > > >> > > > > > >>> > > > >
> > > >> > > > > > >> > > > > > >>> > > > > Maybe I’m being short-sighted,
> but
> > it
> > > >> > > doesn’t
> > > >> > > > > seem
> > > >> > > > > > >> like
> > > >> > > > > > >> > > > > either
> > > >> > > > > > >> > > > > > >>> really
> > > >> > > > > > >> > > > > > >>> > > > > applies here. It seems like the
> > > brokers
> > > >> > > could
> > > >> > > > > and
> > > >> > > > > > >> would
> > > >> > > > > > >> > > > > achieve
> > > >> > > > > > >> > > > > > >>> > consensus
> > > >> > > > > > >> > > > > > >>> > > > > when creating a topic anyway,
> which
> > > is
> > > >> all
> > > >> > > > > that’s
> > > >> > > > > > >> > > required
> > > >> > > > > > >> > > > to
> > > >> > > > > > >> > > > > > >>> > generate
> > > >> > > > > > >> > > > > > >>> > > > > non-colliding sequence ids. For
> the
> > > >> > second,
> > > >> > > as
> > > >> > > > > you
> > > >> > > > > > >> > > mention,
> > > >> > > > > > >> > > > > we
> > > >> > > > > > >> > > > > > >>> could
> > > >> > > > > > >> > > > > > >>> > > > assign
> > > >> > > > > > >> > > > > > >>> > > > > a UUID to the cluster as a whole,
> > > which
> > > >> > > would
> > > >> > > > > > render
> > > >> > > > > > >> > any
> > > >> > > > > > >> > > > > > resource
> > > >> > > > > > >> > > > > > >>> > scoped
> > > >> > > > > > >> > > > > > >>> > > > to
> > > >> > > > > > >> > > > > > >>> > > > > the broker universally unique as
> > > well.
> > > >> > > > > > >> > > > > > >>> > > > >
> > > >> > > > > > >> > > > > > >>> > > > > The reason I mention this is
> that,
> > > >> > although
> > > >> > > a
> > > >> > > > > UUID
> > > >> > > > > > >> is
> > > >> > > > > > >> > way
> > > >> > > > > > >> > > > > more
> > > >> > > > > > >> > > > > > >>> > compact
> > > >> > > > > > >> > > > > > >>> > > > > than topic names, it’s still 16
> > > bytes.
> > > >> In
> > > >> > > > > > contrast,
> > > >> > > > > > >> a
> > > >> > > > > > >> > > > 4-byte
> > > >> > > > > > >> > > > > > >>> integer
> > > >> > > > > > >> > > > > > >>> > > > > sequence id would give us 4
> billion
> > > >> unique
> > > >> > > > > topics
> > > >> > > > > > >> per
> > > >> > > > > > >> > > > > cluster,
> > > >> > > > > > >> > > > > > >>> which
> > > >> > > > > > >> > > > > > >>> > > > seems
> > > >> > > > > > >> > > > > > >>> > > > > like enough ;)
> > > >> > > > > > >> > > > > > >>> > > > >
> > > >> > > > > > >> > > > > > >>> > > > > Considering the number of
> different
> > > >> times
> > > >> > > > these
> > > >> > > > > > >> topic
> > > >> > > > > > >> > > > > > >>> identifiers are
> > > >> > > > > > >> > > > > > >>> > > > sent
> > > >> > > > > > >> > > > > > >>> > > > > over the wire or stored in
> memory,
> > it
> > > >> > seems
> > > >> > > > like
> > > >> > > > > > it
> > > >> > > > > > >> > might
> > > >> > > > > > >> > > > be
> > > >> > > > > > >> > > > > > >>> worth
> > > >> > > > > > >> > > > > > >>> > the
> > > >> > > > > > >> > > > > > >>> > > > > additional 4x space savings.
> > > >> > > > > > >> > > > > > >>> > > > >
> > > >> > > > > > >> > > > > > >>> > > > > What do you think about this?
> > > >> > > > > > >> > > > > > >>> > > > >
> > > >> > > > > > >> > > > > > >>> > > > > Thanks,
> > > >> > > > > > >> > > > > > >>> > > > > John
> > > >> > > > > > >> > > > > > >>> > > > >
> > > >> > > > > > >> > > > > > >>> > > > > On Fri, Sep 11, 2020, at 03:20,
> Tom
> > > >> > Bentley
> > > >> > > > > wrote:
> > > >> > > > > > >> > > > > > >>> > > > > > Hi Justine,
> > > >> > > > > > >> > > > > > >>> > > > > >
> > > >> > > > > > >> > > > > > >>> > > > > > This looks like a very welcome
> > > >> > > improvement.
> > > >> > > > > > >> Thanks!
> > > >> > > > > > >> > > > > > >>> > > > > >
> > > >> > > > > > >> > > > > > >>> > > > > > Maybe I missed it, but the KIP
> > > >> doesn't
> > > >> > > seem
> > > >> > > > to
> > > >> > > > > > >> > mention
> > > >> > > > > > >> > > > > > changing
> > > >> > > > > > >> > > > > > >>> > > > > > DeleteTopicsRequest to identify
> > the
> > > >> > topic
> > > >> > > > > using
> > > >> > > > > > an
> > > >> > > > > > >> > id.
> > > >> > > > > > >> > > > > Maybe
> > > >> > > > > > >> > > > > > >>> > that's out
> > > >> > > > > > >> > > > > > >>> > > > > of
> > > >> > > > > > >> > > > > > >>> > > > > > scope, but DeleteTopicsRequest
> is
> > > not
> > > >> > > listed
> > > >> > > > > > among
> > > >> > > > > > >> > the
> > > >> > > > > > >> > > > > Future
> > > >> > > > > > >> > > > > > >>> Work
> > > >> > > > > > >> > > > > > >>> > APIs
> > > >> > > > > > >> > > > > > >>> > > > > > either.
> > > >> > > > > > >> > > > > > >>> > > > > >
> > > >> > > > > > >> > > > > > >>> > > > > > Kind regards,
> > > >> > > > > > >> > > > > > >>> > > > > >
> > > >> > > > > > >> > > > > > >>> > > > > > Tom
> > > >> > > > > > >> > > > > > >>> > > > > >
> > > >> > > > > > >> > > > > > >>> > > > > > On Thu, Sep 10, 2020 at 3:59 PM
> > > >> Satish
> > > >> > > > > Duggana <
> > > >> > > > > > >> > > > > > >>> > > > satish.dugg...@gmail.com
> > > >> > > > > > >> > > > > > >>> > > > > > wrote:
> > > >> > > > > > >> > > > > > >>> > > > > >
> > > >> > > > > > >> > > > > > >>> > > > > > > Thanks Lucas/Justine for the
> > nice
> > > >> KIP.
> > > >> > > > > > >> > > > > > >>> > > > > > >
> > > >> > > > > > >> > > > > > >>> > > > > > > It has several benefits which
> > > also
> > > >> > > include
> > > >> > > > > > >> > > simplifying
> > > >> > > > > > >> > > > > the
> > > >> > > > > > >> > > > > > >>> topic
> > > >> > > > > > >> > > > > > >>> > > > > > > deletion process by
> controller
> > > and
> > > >> > logs
> > > >> > > > > > cleanup
> > > >> > > > > > >> by
> > > >> > > > > > >> > > > > brokers
> > > >> > > > > > >> > > > > > in
> > > >> > > > > > >> > > > > > >>> > corner
> > > >> > > > > > >> > > > > > >>> > > > > > > cases.
> > > >> > > > > > >> > > > > > >>> > > > > > >
> > > >> > > > > > >> > > > > > >>> > > > > > > Best,
> > > >> > > > > > >> > > > > > >>> > > > > > > Satish.
> > > >> > > > > > >> > > > > > >>> > > > > > >
> > > >> > > > > > >> > > > > > >>> > > > > > > On Wed, Sep 9, 2020 at 10:07
> PM
> > > >> > Justine
> > > >> > > > > > Olshan <
> > > >> > > > > > >> > > > > > >>> > jols...@confluent.io
> > > >> > > > > > >> > > > > > >>> > > > > > > wrote:
> > > >> > > > > > >> > > > > > >>> > > > > > > > Hello all, it's been
> almost a
> > > >> year!
> > > >> > > I've
> > > >> > > > > > made
> > > >> > > > > > >> > some
> > > >> > > > > > >> > > > > > changes
> > > >> > > > > > >> > > > > > >>> to
> > > >> > > > > > >> > > > > > >>> > this
> > > >> > > > > > >> > > > > > >>> > > > > KIP
> > > >> > > > > > >> > > > > > >>> > > > > > > and hope to continue the
> > > >> discussion.
> > > >> > > > > > >> > > > > > >>> > > > > > > > One of the main changes
> I've
> > > >> added
> > > >> > is
> > > >> > > > now
> > > >> > > > > > the
> > > >> > > > > > >> > > > metadata
> > > >> > > > > > >> > > > > > >>> response
> > > >> > > > > > >> > > > > > >>> > > > will
> > > >> > > > > > >> > > > > > >>> > > > > > > include the topic ID (as
> Colin
> > > >> > > suggested).
> > > >> > > > > > >> Clients
> > > >> > > > > > >> > > can
> > > >> > > > > > >> > > > > > >>> obtain the
> > > >> > > > > > >> > > > > > >>> > > > > topicID
> > > >> > > > > > >> > > > > > >>> > > > > > > of a given topic through a
> > > >> > > > TopicDescription.
> > > >> > > > > > The
> > > >> > > > > > >> > > > topicId
> > > >> > > > > > >> > > > > > will
> > > >> > > > > > >> > > > > > >>> > also be
> > > >> > > > > > >> > > > > > >>> > > > > > > included with the
> > UpdateMetadata
> > > >> > > request.
> > > >> > > > > > >> > > > > > >>> > > > > > > > Let me know what you all
> > think.
> > > >> > > > > > >> > > > > > >>> > > > > > > > Thank you,
> > > >> > > > > > >> > > > > > >>> > > > > > > > Justine
> > > >> > > > > > >> > > > > > >>> > > > > > > >
> > > >> > > > > > >> > > > > > >>> > > > > > > > On 2019/09/13 16:38:26,
> > "Colin
> > > >> > > McCabe" <
> > > >> > > > > > >> > > > > > cmcc...@apache.org
> > > >> > > > > > >> > > > > > >>> >
> > > >> > > > > > >> > > > > > >>> > wrote:
> > > >> > > > > > >> > > > > > >>> > > > > > > > > Hi Lucas,
> > > >> > > > > > >> > > > > > >>> > > > > > > > >
> > > >> > > > > > >> > > > > > >>> > > > > > > > > Thanks for tackling this.
> > > >> Topic
> > > >> > IDs
> > > >> > > > > are a
> > > >> > > > > > >> > great
> > > >> > > > > > >> > > > > idea,
> > > >> > > > > > >> > > > > > >>> and
> > > >> > > > > > >> > > > > > >>> > this
> > > >> > > > > > >> > > > > > >>> > > > is
> > > >> > > > > > >> > > > > > >>> > > > > a
> > > >> > > > > > >> > > > > > >>> > > > > > > really good writeup.
> > > >> > > > > > >> > > > > > >>> > > > > > > > > For
> > /brokers/topics/[topic],
> > > >> the
> > > >> > > > schema
> > > >> > > > > > >> version
> > > >> > > > > > >> > > > > should
> > > >> > > > > > >> > > > > > be
> > > >> > > > > > >> > > > > > >>> > bumped
> > > >> > > > > > >> > > > > > >>> > > > to
> > > >> > > > > > >> > > > > > >>> > > > > > > version 3, rather than 2.
> > > KIP-455
> > > >> > > bumped
> > > >> > > > > the
> > > >> > > > > > >> > version
> > > >> > > > > > >> > > > of
> > > >> > > > > > >> > > > > > this
> > > >> > > > > > >> > > > > > >>> > znode
> > > >> > > > > > >> > > > > > >>> > > > to
> > > >> > > > > > >> > > > > > >>> > > > > 2
> > > >> > > > > > >> > > > > > >>> > > > > > > already :)
> > > >> > > > > > >> > > > > > >>> > > > > > > > > Given that we're going to
> > be
> > > >> > seeing
> > > >> > > > > these
> > > >> > > > > > >> > things
> > > >> > > > > > >> > > as
> > > >> > > > > > >> > > > > > >>> strings
> > > >> > > > > > >> > > > > > >>> > as
> > > >> > > > > > >> > > > > > >>> > > > lot
> > > >> > > > > > >> > > > > > >>> > > > > (in
> > > >> > > > > > >> > > > > > >>> > > > > > > logs, in ZooKeeper, on the
> > > >> > command-line,
> > > >> > > > > > etc.),
> > > >> > > > > > >> > does
> > > >> > > > > > >> > > it
> > > >> > > > > > >> > > > > > make
> > > >> > > > > > >> > > > > > >>> > sense to
> > > >> > > > > > >> > > > > > >>> > > > > use
> > > >> > > > > > >> > > > > > >>> > > > > > > base64 when converting them
> to
> > > >> > strings?
> > > >> > > > > > >> > > > > > >>> > > > > > > > > Here is an example of the
> > hex
> > > >> > > > > > >> representation:
> > > >> > > > > > >> > > > > > >>> > > > > > > > >
> > > >> > 6fcb514b-b878-4c9d-95b7-8dc3a7ce6fd8
> > > >> > > > > > >> > > > > > >>> > > > > > > > >
> > > >> > > > > > >> > > > > > >>> > > > > > > > > And here is an example in
> > > >> base64.
> > > >> > > > > > >> > > > > > >>> > > > > > > > > b8tRS7h4TJ2Vt43Dp85v2A
> > > >> > > > > > >> > > > > > >>> > > > > > > > >
> > > >> > > > > > >> > > > > > >>> > > > > > > > > The base64 version saves
> 15
> > > >> > letters
> > > >> > > > (to
> > > >> > > > > be
> > > >> > > > > > >> > fair,
> > > >> > > > > > >> > > 4
> > > >> > > > > > >> > > > of
> > > >> > > > > > >> > > > > > >>> those
> > > >> > > > > > >> > > > > > >>> > were
> > > >> > > > > > >> > > > > > >>> > > > > > > dashes that we could have
> > elided
> > > in
> > > >> > the
> > > >> > > > hex
> > > >> > > > > > >> > > > > > representation.)
> > > >> > > > > > >> > > > > > >>> > > > > > > > > Another thing to consider
> > is
> > > >> that
> > > >> > we
> > > >> > > > > > should
> > > >> > > > > > >> > > specify
> > > >> > > > > > >> > > > > > that
> > > >> > > > > > >> > > > > > >>> the
> > > >> > > > > > >> > > > > > >>> > > > > > > all-zeroes UUID is not a
> valid
> > > >> topic
> > > >> > > UUID.
> > > >> > > > > > We
> > > >> > > > > > >> > can't
> > > >> > > > > > >> > > > use
> > > >> > > > > > >> > > > > > >>> null
> > > >> > > > > > >> > > > > > >>> > for
> > > >> > > > > > >> > > > > > >>> > > > this
> > > >> > > > > > >> > > > > > >>> > > > > > > because we can't pass a null
> > UUID
> > > >> over
> > > >> > > the
> > > >> > > > > RPC
> > > >> > > > > > >> > > protocol
> > > >> > > > > > >> > > > > > >>> (there
> > > >> > > > > > >> > > > > > >>> > is no
> > > >> > > > > > >> > > > > > >>> > > > > > > special pattern for null, nor
> > do
> > > we
> > > >> > want
> > > >> > > > to
> > > >> > > > > > >> waste
> > > >> > > > > > >> > > space
> > > >> > > > > > >> > > > > > >>> reserving
> > > >> > > > > > >> > > > > > >>> > > > such
> > > >> > > > > > >> > > > > > >>> > > > > a
> > > >> > > > > > >> > > > > > >>> > > > > > > pattern.)
> > > >> > > > > > >> > > > > > >>> > > > > > > > > Maybe I missed it, but
> did
> > > you
> > > >> > > > describe
> > > >> > > > > > >> > > "migration
> > > >> > > > > > >> > > > > > of...
> > > >> > > > > > >> > > > > > >>> > existing
> > > >> > > > > > >> > > > > > >>> > > > > > > topic[s] without topic IDs"
> in
> > > >> detail
> > > >> > in
> > > >> > > > any
> > > >> > > > > > >> > section?
> > > >> > > > > > >> > > > It
> > > >> > > > > > >> > > > > > >>> seems
> > > >> > > > > > >> > > > > > >>> > like
> > > >> > > > > > >> > > > > > >>> > > > > when
> > > >> > > > > > >> > > > > > >>> > > > > > > the new controller becomes
> > > active,
> > > >> it
> > > >> > > > should
> > > >> > > > > > >> just
> > > >> > > > > > >> > > > > generate
> > > >> > > > > > >> > > > > > >>> random
> > > >> > > > > > >> > > > > > >>> > > > > UUIDs for
> > > >> > > > > > >> > > > > > >>> > > > > > > these, and write the random
> > UUIDs
> > > >> back
> > > >> > > to
> > > >> > > > > > >> > ZooKeeper.
> > > >> > > > > > >> > > > It
> > > >> > > > > > >> > > > > > >>> would be
> > > >> > > > > > >> > > > > > >>> > > > good
> > > >> > > > > > >> > > > > > >>> > > > > to
> > > >> > > > > > >> > > > > > >>> > > > > > > spell that out. We should
> make
> > > it
> > > >> > clear
> > > >> > > > > that
> > > >> > > > > > >> this
> > > >> > > > > > >> > > > > happens
> > > >> > > > > > >> > > > > > >>> > regardless
> > > >> > > > > > >> > > > > > >>> > > > > of
> > > >> > > > > > >> > > > > > >>> > > > > > > the inter-broker protocol
> > version
> > > >> > (it's
> > > >> > > a
> > > >> > > > > > >> > compatible
> > > >> > > > > > >> > > > > > change).
> > > >> > > > > > >> > > > > > >>> > > > > > > > > "LeaderAndIsrRequests
> > > >> including an
> > > >> > > > > > >> > > > is_every_partition
> > > >> > > > > > >> > > > > > >>> flag"
> > > >> > > > > > >> > > > > > >>> > > > seems a
> > > >> > > > > > >> > > > > > >>> > > > > > > bit wordy. Can we just call
> > > these
> > > >> > "full
> > > >> > > > > > >> > > > > > >>> LeaderAndIsrRequests"?
> > > >> > > > > > >> > > > > > >>> > Then
> > > >> > > > > > >> > > > > > >>> > > > > the
> > > >> > > > > > >> > > > > > >>> > > > > > > RPC field could be named
> > "full".
> > > >> > Also,
> > > >> > > it
> > > >> > > > > > would
> > > >> > > > > > >> > > > probably
> > > >> > > > > > >> > > > > > be
> > > >> > > > > > >> > > > > > >>> > better
> > > >> > > > > > >> > > > > > >>> > > > > for the
> > > >> > > > > > >> > > > > > >>> > > > > > > RPC field to be an enum of {
> > > >> > > UNSPECIFIED,
> > > >> > > > > > >> > > INCREMENTAL,
> > > >> > > > > > >> > > > > FULL
> > > >> > > > > > >> > > > > > >>> }, so
> > > >> > > > > > >> > > > > > >>> > > > that
> > > >> > > > > > >> > > > > > >>> > > > > we
> > > >> > > > > > >> > > > > > >>> > > > > > > can cleanly handle old
> versions
> > > (by
> > > >> > > > treating
> > > >> > > > > > >> them
> > > >> > > > > > >> > as
> > > >> > > > > > >> > > > > > >>> UNSPECIFIED)
> > > >> > > > > > >> > > > > > >>> > > > > > > > > In the
> LeaderAndIsrRequest
> > > >> > section,
> > > >> > > > you
> > > >> > > > > > >> write
> > > >> > > > > > >> > "A
> > > >> > > > > > >> > > > > final
> > > >> > > > > > >> > > > > > >>> > deletion
> > > >> > > > > > >> > > > > > >>> > > > > event
> > > >> > > > > > >> > > > > > >>> > > > > > > will be secheduled for X ms
> > after
> > > >> the
> > > >> > > > > > >> > > > LeaderAndIsrRequest
> > > >> > > > > > >> > > > > > was
> > > >> > > > > > >> > > > > > >>> > first
> > > >> > > > > > >> > > > > > >>> > > > > > > received..." I guess the X
> > was a
> > > >> > > > > placeholder
> > > >> > > > > > >> that
> > > >> > > > > > >> > > you
> > > >> > > > > > >> > > > > > >>> intended
> > > >> > > > > > >> > > > > > >>> > to
> > > >> > > > > > >> > > > > > >>> > > > > replace
> > > >> > > > > > >> > > > > > >>> > > > > > > before posting? :) In any
> > case,
> > > >> this
> > > >> > > > seems
> > > >> > > > > > like
> > > >> > > > > > >> > the
> > > >> > > > > > >> > > > kind
> > > >> > > > > > >> > > > > > of
> > > >> > > > > > >> > > > > > >>> > thing
> > > >> > > > > > >> > > > > > >>> > > > we'd
> > > >> > > > > > >> > > > > > >>> > > > > > > want a configuration for.
> > Let's
> > > >> > > describe
> > > >> > > > > that
> > > >> > > > > > >> > > > > > configuration
> > > >> > > > > > >> > > > > > >>> key
> > > >> > > > > > >> > > > > > >>> > > > > somewhere
> > > >> > > > > > >> > > > > > >>> > > > > > > in this KIP, including what
> its
> > > >> > default
> > > >> > > > > value
> > > >> > > > > > >> is.
> > > >> > > > > > >> > > > > > >>> > > > > > > > > We should probably also
> > log a
> > > >> > bunch
> > > >> > > of
> > > >> > > > > > >> messages
> > > >> > > > > > >> > > at
> > > >> > > > > > >> > > > > WARN
> > > >> > > > > > >> > > > > > >>> level
> > > >> > > > > > >> > > > > > >>> > > > when
> > > >> > > > > > >> > > > > > >>> > > > > > > something is scheduled for
> > > >> deletion,
> > > >> > as
> > > >> > > > > well.
> > > >> > > > > > >> > (Maybe
> > > >> > > > > > >> > > > > this
> > > >> > > > > > >> > > > > > >>> was
> > > >> > > > > > >> > > > > > >>> > > > > assumed, but
> > > >> > > > > > >> > > > > > >>> > > > > > > it would be good to mention
> > it).
> > > >> > > > > > >> > > > > > >>> > > > > > > > > I feel like there are a
> few
> > > >> > sections
> > > >> > > > > that
> > > >> > > > > > >> > should
> > > >> > > > > > >> > > be
> > > >> > > > > > >> > > > > > >>> moved to
> > > >> > > > > > >> > > > > > >>> > > > > "rejected
> > > >> > > > > > >> > > > > > >>> > > > > > > alternatives." For example,
> in
> > > the
> > > >> > > > > > DeleteTopics
> > > >> > > > > > >> > > > section,
> > > >> > > > > > >> > > > > > >>> since
> > > >> > > > > > >> > > > > > >>> > we're
> > > >> > > > > > >> > > > > > >>> > > > > not
> > > >> > > > > > >> > > > > > >>> > > > > > > going to do option 1 or 2,
> > these
> > > >> > should
> > > >> > > be
> > > >> > > > > > moved
> > > >> > > > > > >> > into
> > > >> > > > > > >> > > > > > >>> "rejected
> > > >> > > > > > >> > > > > > >>> > > > > > > alternatives," rather than
> > > >> appearing
> > > >> > > > > inline.
> > > >> > > > > > >> > > Another
> > > >> > > > > > >> > > > > case
> > > >> > > > > > >> > > > > > >>> is
> > > >> > > > > > >> > > > > > >>> > the
> > > >> > > > > > >> > > > > > >>> > > > > "Should
> > > >> > > > > > >> > > > > > >>> > > > > > > we remove topic name from the
> > > >> protocol
> > > >> > > > where
> > > >> > > > > > >> > > possible"
> > > >> > > > > > >> > > > > > >>> section.
> > > >> > > > > > >> > > > > > >>> > This
> > > >> > > > > > >> > > > > > >>> > > > > is
> > > >> > > > > > >> > > > > > >>> > > > > > > clearly discussing a design
> > > >> > alternative
> > > >> > > > that
> > > >> > > > > > >> we're
> > > >> > > > > > >> > > not
> > > >> > > > > > >> > > > > > >>> proposing
> > > >> > > > > > >> > > > > > >>> > to
> > > >> > > > > > >> > > > > > >>> > > > > > > implement: removing the topic
> > > name
> > > >> > from
> > > >> > > > > those
> > > >> > > > > > >> > > > protocols.
> > > >> > > > > > >> > > > > > >>> > > > > > > > > Is it really necessary to
> > > have
> > > >> a
> > > >> > new
> > > >> > > > > > >> > > > > > >>> > /admin/delete_topics_by_id
> > > >> > > > > > >> > > > > > >>> > > > > path
> > > >> > > > > > >> > > > > > >>> > > > > > > in ZooKeeper? It seems like
> we
> > > >> don't
> > > >> > > > really
> > > >> > > > > > >> need
> > > >> > > > > > >> > > this.
> > > >> > > > > > >> > > > > > >>> Whenever
> > > >> > > > > > >> > > > > > >>> > > > > there is
> > > >> > > > > > >> > > > > > >>> > > > > > > a new controller, we'll send
> > out
> > > >> full
> > > >> > > > > > >> > > > > LeaderAndIsrRequests
> > > >> > > > > > >> > > > > > >>> which
> > > >> > > > > > >> > > > > > >>> > will
> > > >> > > > > > >> > > > > > >>> > > > > > > trigger the stale topics to
> be
> > > >> cleaned
> > > >> > > up.
> > > >> > > > > > The
> > > >> > > > > > >> > > active
> > > >> > > > > > >> > > > > > >>> > controller
> > > >> > > > > > >> > > > > > >>> > > > will
> > > >> > > > > > >> > > > > > >>> > > > > > > also send the full
> > > >> LeaderAndIsrRequest
> > > >> > > to
> > > >> > > > > > >> brokers
> > > >> > > > > > >> > > that
> > > >> > > > > > >> > > > > are
> > > >> > > > > > >> > > > > > >>> just
> > > >> > > > > > >> > > > > > >>> > > > > starting
> > > >> > > > > > >> > > > > > >>> > > > > > > up. So we don't really
> need
> > > this
> > > >> > kind
> > > >> > > > of
> > > >> > > > > > >> > two-phase
> > > >> > > > > > >> > > > > > commit
> > > >> > > > > > >> > > > > > >>> > (send
> > > >> > > > > > >> > > > > > >>> > > > out
> > > >> > > > > > >> > > > > > >>> > > > > > > StopReplicasRequest, get ACKs
> > > from
> > > >> all
> > > >> > > > > nodes,
> > > >> > > > > > >> > commit
> > > >> > > > > > >> > > by
> > > >> > > > > > >> > > > > > >>> removing
> > > >> > > > > > >> > > > > > >>> > > > > > > /admin/delete_topics node)
> any
> > > >> more.
> > > >> > > > > > >> > > > > > >>> > > > > > > > > You mention that
> > FetchRequest
> > > >> will
> > > >> > > now
> > > >> > > > > > >> include
> > > >> > > > > > >> > > UUID
> > > >> > > > > > >> > > > > to
> > > >> > > > > > >> > > > > > >>> avoid
> > > >> > > > > > >> > > > > > >>> > > > issues
> > > >> > > > > > >> > > > > > >>> > > > > > > where requests are made to
> > stale
> > > >> > > > partitions.
> > > >> > > > > > >> > > However,
> > > >> > > > > > >> > > > > > >>> adding a
> > > >> > > > > > >> > > > > > >>> > UUID
> > > >> > > > > > >> > > > > > >>> > > > to
> > > >> > > > > > >> > > > > > >>> > > > > > > MetadataRequest is listed as
> > > future
> > > >> > > work,
> > > >> > > > > out
> > > >> > > > > > of
> > > >> > > > > > >> > > scope
> > > >> > > > > > >> > > > > for
> > > >> > > > > > >> > > > > > >>> this
> > > >> > > > > > >> > > > > > >>> > KIP.
> > > >> > > > > > >> > > > > > >>> > > > > How
> > > >> > > > > > >> > > > > > >>> > > > > > > will the client learn what
> the
> > > >> topic
> > > >> > > UUID
> > > >> > > > > is,
> > > >> > > > > > if
> > > >> > > > > > >> > the
> > > >> > > > > > >> > > > > > metadata
> > > >> > > > > > >> > > > > > >>> > > > response
> > > >> > > > > > >> > > > > > >>> > > > > > > doesn't include that
> > information?
> > > >> It
> > > >> > > > seems
> > > >> > > > > > like
> > > >> > > > > > >> > > adding
> > > >> > > > > > >> > > > > the
> > > >> > > > > > >> > > > > > >>> UUID
> > > >> > > > > > >> > > > > > >>> > to
> > > >> > > > > > >> > > > > > >>> > > > > > > MetadataResponse would be an
> > > >> > improvement
> > > >> > > > > here
> > > >> > > > > > >> that
> > > >> > > > > > >> > > > might
> > > >> > > > > > >> > > > > > not
> > > >> > > > > > >> > > > > > >>> be
> > > >> > > > > > >> > > > > > >>> > too
> > > >> > > > > > >> > > > > > >>> > > > > hard to
> > > >> > > > > > >> > > > > > >>> > > > > > > make.
> > > >> > > > > > >> > > > > > >>> > > > > > > > > best,
> > > >> > > > > > >> > > > > > >>> > > > > > > > > Colin
> > > >> > > > > > >> > > > > > >>> > > > > > > > >
> > > >> > > > > > >> > > > > > >>> > > > > > > > >
> > > >> > > > > > >> > > > > > >>> > > > > > > > > On Mon, Sep 9, 2019, at
> > > 17:48,
> > > >> > > Ryanne
> > > >> > > > > > Dolan
> > > >> > > > > > >> > > wrote:
> > > >> > > > > > >> > > > > > >>> > > > > > > > > > Lucas, this would be
> > great.
> > > >> I've
> > > >> > > run
> > > >> > > > > > into
> > > >> > > > > > >> > > issues
> > > >> > > > > > >> > > > > with
> > > >> > > > > > >> > > > > > >>> > topics
> > > >> > > > > > >> > > > > > >>> > > > > being
> > > >> > > > > > >> > > > > > >>> > > > > > > > > > resurrected
> accidentally,
> > > >> since
> > > >> > a
> > > >> > > > > client
> > > >> > > > > > >> > cannot
> > > >> > > > > > >> > > > > > easily
> > > >> > > > > > >> > > > > > >>> > > > > distinguish
> > > >> > > > > > >> > > > > > >>> > > > > > > between
> > > >> > > > > > >> > > > > > >>> > > > > > > > > > a deleted topic and a
> new
> > > >> topic
> > > >> > > with
> > > >> > > > > the
> > > >> > > > > > >> same
> > > >> > > > > > >> > > > name.
> > > >> > > > > > >> > > > > > I'd
> > > >> > > > > > >> > > > > > >>> > need
> > > >> > > > > > >> > > > > > >>> > > > the
> > > >> > > > > > >> > > > > > >>> > > > > ID
> > > >> > > > > > >> > > > > > >>> > > > > > > > > > accessible from the
> > client
> > > to
> > > >> > > solve
> > > >> > > > > that
> > > >> > > > > > >> > issue,
> > > >> > > > > > >> > > > but
> > > >> > > > > > >> > > > > > >>> this
> > > >> > > > > > >> > > > > > >>> > is a
> > > >> > > > > > >> > > > > > >>> > > > > good
> > > >> > > > > > >> > > > > > >>> > > > > > > first
> > > >> > > > > > >> > > > > > >>> > > > > > > > > > step.
> > > >> > > > > > >> > > > > > >>> > > > > > > > > >
> > > >> > > > > > >> > > > > > >>> > > > > > > > > > Ryanne
> > > >> > > > > > >> > > > > > >>> > > > > > > > > >
> > > >> > > > > > >> > > > > > >>> > > > > > > > > > On Wed, Sep 4, 2019 at
> > 1:41
> > > >> PM
> > > >> > > Lucas
> > > >> > > > > > >> > > Bradstreet <
> > > >> > > > > > >> > > > > > >>> > > > > lu...@confluent.io>
> > > >> > > > > > >> > > > > > >>> > > > > > > wrote:
> > > >> > > > > > >> > > > > > >>> > > > > > > > > > > Hi all,
> > > >> > > > > > >> > > > > > >>> > > > > > > > > > >
> > > >> > > > > > >> > > > > > >>> > > > > > > > > > > I would like to kick
> > off
> > > >> > > > discussion
> > > >> > > > > of
> > > >> > > > > > >> > > KIP-516,
> > > >> > > > > > >> > > > > an
> > > >> > > > > > >> > > > > > >>> > > > > implementation
> > > >> > > > > > >> > > > > > >>> > > > > > > of topic
> > > >> > > > > > >> > > > > > >>> > > > > > > > > > > IDs for Kafka. Topic
> > IDs
> > > >> aim
> > > >> > to
> > > >> > > > > solve
> > > >> > > > > > >> topic
> > > >> > > > > > >> > > > > > >>> uniqueness
> > > >> > > > > > >> > > > > > >>> > > > > problems in
> > > >> > > > > > >> > > > > > >>> > > > > > > Kafka,
> > > >> > > > > > >> > > > > > >>> > > > > > > > > > > where referring to a
> > > topic
> > > >> by
> > > >> > > name
> > > >> > > > > > >> alone is
> > > >> > > > > > >> > > > > > >>> insufficient.
> > > >> > > > > > >> > > > > > >>> > > > Such
> > > >> > > > > > >> > > > > > >>> > > > > > > cases
> > > >> > > > > > >> > > > > > >>> > > > > > > > > > > include when a topic
> > has
> > > >> been
> > > >> > > > > deleted
> > > >> > > > > > >> and
> > > >> > > > > > >> > > > > recreated
> > > >> > > > > > >> > > > > > >>> with
> > > >> > > > > > >> > > > > > >>> > the
> > > >> > > > > > >> > > > > > >>> > > > > same
> > > >> > > > > > >> > > > > > >>> > > > > > > name.
> > > >> > > > > > >> > > > > > >>> > > > > > > > > > > Unique identifiers
> will
> > > >> help
> > > >> > > > > simplify
> > > >> > > > > > >> and
> > > >> > > > > > >> > > > improve
> > > >> > > > > > >> > > > > > >>> Kafka's
> > > >> > > > > > >> > > > > > >>> > > > topic
> > > >> > > > > > >> > > > > > >>> > > > > > > deletion
> > > >> > > > > > >> > > > > > >>> > > > > > > > > > > process, as well as
> > > prevent
> > > >> > > cases
> > > >> > > > > > where
> > > >> > > > > > >> > > brokers
> > > >> > > > > > >> > > > > may
> > > >> > > > > > >> > > > > > >>> > > > incorrectly
> > > >> > > > > > >> > > > > > >>> > > > > > > interact
> > > >> > > > > > >> > > > > > >>> > > > > > > > > > > with stale versions
> of
> > > >> topics.
> > > >> > > > > > >> > > > > > >>> > > > > > > > > > >
> > > >> > > > > > >> > > > > > >>> > > > > > > > > > >
> > > >> > > > > > >> > > > > > >>> > > > > > > > > > >
> > > >> > > > > > >> > > > > > >>> > > >
> > > >> > > > > > >> > > > > > >>> >
> > > >> > > > > > >> > > > > > >>>
> > > >> > > > > > >> > > > > >
> > > >> > > > > > >> > > > >
> > > >> > > > > > >> > > >
> > > >> > > > > > >> > >
> > > >> > > > > > >> >
> > > >> > > > > > >>
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-516%3A+Topic+Identifiers
> > > >> > > > > > >> > > > > > >>> > > > > > > > > > > Looking forward to
> your
> > > >> > > thoughts.
> > > >> > > > > > >> > > > > > >>> > > > > > > > > > >
> > > >> > > > > > >> > > > > > >>> > > > > > > > > > > Lucas
> > > >> > > > > > >> > > > > > >>> > > > > > > > > > >
> > > >> > > > > > >> > > > > > >>> >
> > > >> > > > > > >> > > > > > >>> >
> > > >> > > > > > >> > > > > > >>>
> > > >> > > > > > >> > > > > > >>
> > > >> > > > > > >> > > > > >
> > > >> > > > > > >> > > > >
> > > >> > > > > > >> > > >
> > > >> > > > > > >> > >
> > > >> > > > > > >> >
> > > >> > > > > > >>
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > > >
> > >
> >
>
