Hi Jiuming > Is it better to create the same number of retry/dlq topic > partitions as the original topic?
That is what I want to change, in other words, https://github.com/apache/pulsar/pull/22705 did this change. But the topic name is not the same as you mentioned, the behavior is like the below after the PR: - create topic `t1` with `16` partitions - you will get `t1-partition-0`, `t1-partition-1`...`tp-partition-15`. - after you enable DLQ, you will get the following `16` DLQs: `t1-partition-0-{subscription}-DLQ`, `t1-partition-1-{subscription}-DLQ`...`t1-partition-15-{subscription}-DLQ` We can not name the DLQ as `t1-DLQ` if the primary topic is partitioned because we can not define the behavior after users expending the partitions of `t1` to `20`. After the PR, the DLQ related to the partitions expanded will be created automatically, which is expected. Thanks Yubiao Feng On Wed, May 15, 2024 at 1:21 AM Tao Jiuming <dao...@apache.org> wrote: > Hi, Yubiao > > Is it better to create the same number of retry/dlq topic partitions as > the original topic? > > I mean, if the original topic has 16 partitions, we also create 16(or some > other numbers, can be configurable) partitions for the retry/dlq topic. > > If the topic/partition is a retry/dlq topic, it's name matches the > pattern: > `{topic}-partition-{n}-{subscription}-DLQ`, right? > > Can we just parse the name and then get the origin topic name, and create > retry/dlq topic with the same partitions? > > Say, if client wants to create > `persistent://public/default/some_topic-partition-1-test_sub-DLQ`, > we parse the topic name, then we get the origin topic name > `persistent://public/default/some_topic` and the subscription name > `test_sub`. > After this, we look up the number of partitions of the origin topic, for > instance, 16 partitions, in the last, create a partitioned topic > `admin.topics.createPartitionedTopic("persistent://public/default/some_topic-test_sub-DLQ", > 16)`. > > And we can also increase the retry/dlq topic partitions if the origin > topic partitions increased. > > WDYT, is it feasible? > > Thanks, > Tao Jiuming > > On 2024/05/13 18:56:01 Yubiao Feng wrote: > > Hi community > > > > I want to change the behavior of DLQ/Retry Topic auto creation. > > > > ### Motivation > > After you set `defaultNumPartitions` to `16`, you will get `16` > partitions > > per topic, and `16*16` DLQ partitions per subscription, you will get a > huge > > number of DLQs if you have more than one subscription under one topic. > For > > example: > > - create topic `t1` with `16` partitions > > - you will get `t1-partition-0`, `t1-partition-1`...`tp-partition-15`. > > - after you enable DLQ, you will get the following `16*16` DLQs per > > subscription > > - `t1-partition-0-{subscription}-DLQ-partition-0` > > - `t1-partition-0-{subscription}-DLQ-partition-1` > > - ... > > - `t1-partition-15-{subscription}-DLQ-partition-15` > > - `t1-partition-1-{subscription}-DLQ-partition-0` > > - `t1-partition-1-{subscription}-DLQ-partition-1` > > - ... > > - ... > > - ... > > - `t1-partition-15-{subscription}-partition-15` > > > > ### Goal > > - Do not create partitioned DLQs/Retry topics automatically. > > - Users can also create partitioned DLQ manually if they need it. > > > > I have drafted a PR, please take a look > > - https://github.com/apache/pulsar/pull/22705 > > > > Thanks > > Yubiao Feng > > >
