> In general, these problems hint to me that we need better definitions > for "system" and "special" topics that break the rules of auto created > topics in well defined ways.
This point reminds me of this discussion from a year and a half ago: https://lists.apache.org/thread/qgbpzr6o3k5rqbs2jvpkdh8hr9jpnw39 For PIP 124, I challenged a feature based on the question of whether the DLQ is a special topic or not. Ultimately, we determined it is a special topic (see Matteo's response later in that thread). I think it is interesting because it feels like we continue to find exceptions. Just this week, we found yet another and lamented the lack of design for these topics: https://github.com/apache/pulsar/pull/20514#issuecomment-1579937937. I mention those points because I am concerned that creating a DLQ specific piece of boolean metadata in the topic properties does not move us toward a unified and extensible abstraction. Unless we want to continue to infer everything from topic names, it seems like we almost need a "topic type" enum to capture the many types of special topics we have. Type has the same issues I mentioned in my last email, but it could help create a more generic solution. > ### Properties of Topic > ```properties > "wasAutoCreated": boolean > ``` I don't have the answer yet, but I think we should spend more time working on a general solution for topic classification before adding per topic metadata. I'd be interested to know what others think on the subject. Thanks, Michael On Thu, Jun 1, 2023 at 2:08 AM Yubiao Feng <yubiao.f...@streamnative.io.invalid> wrote: > > Hi Michael, Enrico > > > The identifier is the name. The topics have `-DLQ` and `-RETRY` > > as suffixes, which makes them "special" (which reminds me of this > > thread [0]). We could choose to always make these non partitioned > > or to make them configurable in some other way. > > I changed the design like this: > > ### Properties of Topic > ```properties > "wasAutoCreated": boolean > ``` > > - If the topic name ends with "-RETRY" or "-DLQ", only non-partitioned > topics will be automatically created > - If the property `wasAutoCreated` of the topic is true and the topic name > ends with "-RETRY" or "-DLQ", retry topics and dead letter queues > will no longer be created for this topic. > > How are you feeling? > > Thanks > Yubiao Feng > > On Thu, Jun 1, 2023 at 12:48 PM Michael Marshall <mmarsh...@apache.org> > wrote: > > > We've had several bugs related to this topic of auto created > > partitioned topics recently. These three PRs come to mind: > > > > * https://github.com/apache/pulsar/pull/20370 > > * https://github.com/apache/pulsar/pull/20392 > > * https://github.com/apache/pulsar/pull/20397 > > > > Those PRs rely on server side inference based on the topic name to > > know that a topic should not be a partitioned topic. > > > > In general, these problems hint to me that we need better definitions > > for "system" and "special" topics that break the rules of auto created > > topics in well defined ways. > > > > I agree with Enrico that "purposeOfAutoCreatedTopic" will create overhead. > > > > > I agree with you, but now the problem is that we still need > > > an identifier to say that it is a DLQ. Do you have some > > > suggestions? > > > > The identifier is the name. The topics have `-DLQ` and `-RETRY` as > > suffixes, which makes them "special" (which reminds me of this thread > > [0]). We could choose to always make these non partitioned or to make > > them configurable in some other way. > > > > Or, we could consider giving producers and consumers the option to > > configure a `partitionCountHint`. This could be used by the broker > > during auto creation to create a partitioned topic with x partitions > > when greater than 0 and to create a non-partitioned topic when 0. This > > moves the logic out of inference on topic names and into the realm of > > client configuration. However, it could create hard to debug > > scenarios. > > > > Thanks, > > Michael > > > > [0] https://lists.apache.org/thread/yrkf88jjpjzhmk6hy15ynnk3l6n96l9w > > > > On Wed, May 31, 2023 at 7:42 AM Yubiao Feng > > <yubiao.f...@streamnative.io.invalid> wrote: > > > > > > Hi @Enrico > > > > > > > I think that it is better to add flags like: > > > > - allowAutoTopiCreation: default "true", if "false" the broker > > > > won't create the topic in any case > > > > autoTopicCreationMode: undefined/partitioned/non-partitioned > > > > > > I agree with you, but now the problem is that we still need > > > an identifier to say that it is a DLQ. Do you have some > > > suggestions? > > > > > > Thanks > > > > > > Yubiao Feng > > > > > > > > > On Wed, May 31, 2023 at 8:15 PM Enrico Olivelli <eolive...@gmail.com> > > wrote: > > > > > > > Yubiao, > > > > I understand your problem: DLQ and retry topic don't play well with > > > > automatic topic creation. > > > > > > > > We are introducing wire protocol changes and also starting to add > > > > topic properties for every new topic (they come with a cost on > > > > metadata). > > > > > > > > I am not sure that adding the purposeOfAutoCreatedTopic property adds > > > > much value and I suggest to drop it. > > > > Logging on the broker that you created the topic is enough. I don't > > > > think that any application or even the broker would use that piece of > > > > information. > > > > > > > > > > > > Regarding topicPurpose I think that having this "list" of > > > > possibilities will become hard to maintain un the future. > > > > > > > > I think that it is better to add flags like: > > > > - allowAutoTopiCreation: default "true", if "false" the broker won't > > > > create the topic in any case > > > > - autoTopicCreationMode: undefined/partitioned/non-partitioned > > > > > > > > One problem with partitioned topic and DLQ is that you probably want > > > > the same number of partitions on the side topic as the original topic, > > > > so probably you have to carry that parameter (number of partitions). > > > > > > > > IIRC the Subscribe command is sent directly per each partition, so I > > > > don't know how these things may work well together. > > > > > > > > Enrico > > > > > > > > > > > > Il giorno sab 8 apr 2023 alle ore 05:55 Xiangying Meng > > > > <xiangy...@apache.org> ha scritto: > > > > > > > > > > Hi Yubiao, > > > > > > > > > > Appreciate your effort in initiating this PIP. I believe these > > changes > > > > will > > > > > address the existing issues and make DLQ and Retry Topic handling > > more > > > > > efficient and straightforward. > > > > > > > > > > The goals you outlined are clear and, upon implementation, will > > improve > > > > the > > > > > overall functionality of Pulsar. The proposed API changes also seem > > > > > suitable for achieving the desired outcomes. > > > > > > > > > > Looking forward to the progress on this PIP. > > > > > > > > > > Best regards, > > > > > Xiangying > > > > > > > > > > On Fri, Apr 7, 2023 at 1:56 AM Yubiao Feng > > > > > <yubiao.f...@streamnative.io.invalid> wrote: > > > > > > > > > > > Hi community > > > > > > > > > > > > I started a PIP about "Just auto-create no-partitioned DLQ And > > Prevent > > > > > > auto-create a DLQ for a DLQ". > > > > > > > > > > > > PIP link: https://github.com/apache/pulsar/issues/20033 > > > > > > > > > > > > ### Motivation > > > > > > > > > > > > #### Just auto-create no-partitioned DLQ/Retry Topic > > > > > > If enabled the config `allowAutoTopicCreation,` Pulsar will > > > > auto-create a > > > > > > topic when the client loads it; After setting config > > > > > > `allowAutoTopicCreationType=partitioned, defaultNumPartitions=2`, > > > > Pulsar > > > > > > will auto-create a partitioned topic(which have two partitions) > > when > > > > the > > > > > > client loads it. > > > > > > > > > > > > After the above, if using the feature [Retry Topic]( > > > > > > > > > > > > > > > > > > https://pulsar.apache.org/docs/2.11.x/concepts-messaging/#retry-letter-topic > > > > > > ) > > > > > > and [DLQ]( > > > > > > > > > > > > https://pulsar.apache.org/docs/2.11.x/concepts-messaging/#dead-letter-topic > > > > > > ) > > > > > > enable topic auto-creation, we will get a partitioned DLQ and a > > > > partitioned > > > > > > Retry Topic like this: > > > > > > - `{primary_topic_name}-{sub_name}-DLQ` > > > > > > -`{primary_topic_name}-{sub_name}-DLQ-partition-0` > > > > > > -`{primary_topic_name}-{sub_name}-DLQ-partition-1` > > > > > > - `{primary_topic_name}-{sub_name}-RETRY` > > > > > > -`{primary_topic_name}-{sub_name}-RETRY-partition-0` > > > > > > -`{primary_topic_name}-{sub_name}-RETRY-partition-1` > > > > > > > > > > > > ---- > > > > > > > > > > > > I feel that almost all users will not use the multi-partitioned > > DLQ or > > > > > > multi-partitioned Retry topic because there is a bug that causes > > the > > > > above > > > > > > behavior to be incorrect, but we have yet to receive any issues > > about > > > > it. > > > > > > This bug causes the above behavior to look like this: When the > > > > partitioned > > > > > > DLQ is auto-created for the topic `tp1-partition-0`, Pulsar will > > > > create a > > > > > > partitioned topic meta which has two partitioned but only create a > > > > topic > > > > > > named `{primary_topic_name}-{sub_name}-DLQ,` there is no topic > > named > > > > > > `{primary_topic_name}-{sub_name}-DLQ-partition-x.` Please look at > > this > > > > > > [PR]( https://github.com/apache/pulsar/pull/19841) for a detailed > > bug > > > > > > description. > > > > > > > > > > > > So I want to change the behavior to Just auto-create no-partitioned > > > > > > DLQ/Retry Topic. > > > > > > > > > > > > ---- > > > > > > > > > > > > #### Prevent auto-create the DLQ for a DLQ > > > > > > Please look at this [Discussion]( > > > > > > https://lists.apache.org/thread/q1m23ckyy10wvtzy65v8bwqwnh7r0gc8) > > for > > > > the > > > > > > detail. > > > > > > > > > > > > ---- > > > > > > > > > > > > ### Goal > > > > > > > > > > > > - Just auto-create no-partitioned DLQ/Retry Topic(with the other > > words: > > > > > > prevent auto-create partitioned DLQ) > > > > > > - DLQ/Retry topic should not create for a DLQ/Retry Topic > > > > > > - roles: > > > > > > - DLQ will not auto-create for a DLQ > > > > > > - Retry Topic will not auto-create for a Retry Topic > > > > > > - DLQ will not auto-create for a Retry Topic > > > > > > - Retry Topic will not auto-create for a DLQ > > > > > > - client changes: Clients will not create a DLQ for a DLQ > > > > > > - broker changes: rejected the request which wants to > > auto-create a > > > > DLQ > > > > > > for a DLQ > > > > > > > > > > > > ---- > > > > > > > > > > > > ### API Changes > > > > > > > > > > > > #### CommandSubscribe.java > > > > > > ```java > > > > > > /** > > > > > > * This is an enumeration value with tree options: "standard", > > "dead > > > > > > letter", "retry letter". > > > > > > */ > > > > > > private String topicPurpose; > > > > > > ``` > > > > > > > > > > > > #### Properties of Topic > > > > > > ```properties > > > > > > "purposeOfAutoCreatedTopic": value with tree options: "standard", > > "dead > > > > > > letter", "retry letter" > > > > > > ``` > > > > > > > > > > > > Why not use two properties: `isAutoCreated` and `topicPurpose`? > > > > > > Because there is a scenario like this: auto-create a topic, use it > > as > > > > a DLQ > > > > > > after a few days, and not use it as a DLQ after a few days, this > > Topic > > > > will > > > > > > be allowed to have DLQ/Retry Topic. We only mark the topics > > created for > > > > > > DLQ/Retry purposes. > > > > > > > > > > > > > > > > > > Thanks > > > > > > Yubiao Feng > > > > > > > > > > > >
