> Let's proceed with reverting #24118 from branch-3.0, branch-4.3, and > branch-4.0. We'll keep this feature only in Pulsar@4.1.
+1. I'll handle reverting PR 24118 changes in the maintenance branches. There's also a follow-up PR 24154 which I'll also consider reverting before PR 24118. Since we will target the "topic consistency check" feature for Pulsar 4.1, it would be useful to document the details in a PIP so that everyone is well aware of the change. PIPs are a good way to document the changes even if a PIP is made after the changes have already been merged to master. The voting on a PIP isn't the most relevant part of PIPs. In this case, when the PIP is written afterwards, LLM/AI can automate most of the writing of the PIP document when given the full context and PIP document template. Providing the context can be done by adding the diffs of the implementation PRs and the PDF printed PR descriptions, similar to how PR descriptions can be improved with LLMs as documented in https://gist.github.com/lhotari/81f533af4b9ad515e02d96e543c4408b. In addition, mailing list discussions could be added to the context and instead of the PR template, the PIP template https://github.com/apache/pulsar/blob/master/pip/TEMPLATE.md and PIP readme https://github.com/apache/pulsar/blob/master/pip/README.md would be used. -Lari On 2025/04/23 09:59:32 Zixuan Liu wrote: > If a partitioned topic lacks metadata, that indicates an unusual and > inconsistent state. > > For the demo, I understand your intention to use a partitioned one after > the partitioned topic is deleted, right now it should be the > non-partitioned topic. I'm open to adding a switch to verify topic > consistency, but I don’t think that’s the right long-term direction. As the > downstream, the priority should be aligning with upstream changes rather > than trying to prevent them. > > Let's proceed with reverting #24118 from branch-3.0, branch-4.3, and > branch-4.0. We'll keep this feature only in Pulsar@4.1. > > > Thanks, > Zixuan > > Yunze Xu <x...@apache.org> 于2025年4月23日周三 15:50写道： > > > Hi Zixuan, > > > > The use case is that the client side does not care about how many > > partitions the topic has. Creating a partitioned topic for this use > > case just acts as a convenient approach to create and delete all these > > partitions, as well as querying the total count and the whole > > properties. > > > > However, the partition metadata is not required for this use case. > > Actually we don't have to respect the number of partitions. For > > example, given a topic with 50 partitions, creating 50 producers and > > 50 consumers on that partitioned topic might bring too much pressure > > on an application without much CPU. We can configure the number of > > partitions manually, here is an example that explains the use case > > > > ```java > > // Before the restart (numPartitions is configured statically) > > final var numPartitions = 50; > > admin.topics().createPartitionedTopic(topic, 50); > > final var producers = new ArrayList<Producer<byte[]>>(); > > for (int i = 0; i < numPartitions; i++) { > > producers.add(pulsarClient.newProducer().topic(topic + > > TopicName.PARTITIONED_TOPIC_SUFFIX + i).create()); > > } > > // Now there are 50 producers > > for (var producer : producers) { > > producer.close(); > > } > > producers.clear(); > > // After the restart (we changed the numPartitions config from 50 to 4) > > admin.topics().deletePartitionedTopic(topic, true); > > final var newNumPartitions = 4; > > for (int i = 0; i < newNumPartitions; i++) { > > producers.add(pulsarClient.newProducer().topic(topic + > > TopicName.PARTITIONED_TOPIC_SUFFIX + i).create()); > > } > > // Now there are only 4 producers > > ``` > > > > In the code above, deleting the partitioned topic can clear old data > > and avoid new partitions being deleted by deleting the partitioned > > topic. > > > > Thanks, > > yunze > > > > On Wed, Apr 23, 2025 at 11:47 AM Zixuan Liu <zix...@apache.org> wrote: > > > > > > I noticed that the following KoP tests need to be updated: > > > OffsetResetTest.java, GroupCoordinatorTest.java, and > > > TransactionStateManagerTest.java. The breakage is limited to test > > scenarios > > > only. > > > > It has been valid and allowed to create producers or readers > > > directly on partitions if the partition count is known in advance > > > > > > PR #24118 does not break this feature. If the partition count (i.e., > > > partitioned metadata) is known, the broker still allows producers and > > > consumers to create partitions as expected. > > > > There is a trick that this partitioned topic can be deleted by force > > > and then producers will reconnect the partitions to continue > > > producing. However, with this change, the producers will fail to > > > reconnect due to `NotAllowedException`, Sometimes, users just want to > > > discard huge legacy data. Regarding the loss of partition metadata, it > > > can be recreated by creating the same partitioned topic again. I admit > > > it's a tricky solution. > > > > > > In cases where a partitioned topic is forcibly deleted while the producer > > > or consumer is online, the client will retry and may end up creating a > > > non-partitioned topic with `-partition-`. Since the partitioned metadata > > is > > > deleted, the client will no longer recognize it as a partitioned topic. > > > Upon restart, producers and consumers won’t be able to publish or consume > > > messages as expected from a partitioned topic. > > > PR #24118 can prevent this scenario. It enforces that users must > > explicitly > > > create the partitioned metadata. > > > > > > I also discuss this issue on slack: > > > https://apache-pulsar.slack.com/archives/C5ZSVEN4E/p1742803935952389 > > > > > > Once we've concluded this discussion, I'll review your PR. Thanks again > > for > > > the fix! > > > Thanks, > > > Zixuan > > > > > > Yunze Xu <x...@apache.org> 于2025年4月23日周三 00:34写道： > > > > > > > Hi all, > > > > > > > > I prepared a quick fix: https://github.com/apache/pulsar/pull/24195 so > > > > that we can fix the regression quickly in 3.0, 3.3 and 4.0 releases. I > > > > devoted my time addressing this breaking change because it's my > > > > responsibility that I missed the previous open discussion in the ML > > > > list. But I didn't have more time to write a PIP and go through the > > > > PIP process for that. > > > > > > > > Thanks, > > > > Yunze > > > > > > > > On Tue, Apr 22, 2025 at 11:24 PM Yunze Xu <x...@apache.org> wrote: > > > > > > > > > > Hi Zixuan, > > > > > > > > > > If you're maintaining your own private KoP fork and upgrading the > > > > > Pulsar dependency to include this PR, all tests in > > > > > `GroupCoordinatorTest` fail without changes. I didn't have a chance > > to > > > > > look into all the failed tests for now. But at least, there are 37 > > > > > tests failing due to the same reason. I've verified that some can be > > > > > fixed by creating a partitioned topic manually. > > > > > > > > > > Even out of the scope of KoP, Pulsar's partitioned topic can be used > > > > > as a few partitions and a metadata that represents the partition > > > > > count. It has been valid and allowed to create producers or readers > > > > > directly on partitions if the partition count is known in advance > > > > > (e.g. via the admin API or UI). > > > > > - producer0 on `topic-partition-0` > > > > > - producer1 on `topic-partition-1` > > > > > - ... > > > > > > > > > > There is a trick that this partitioned topic can be deleted by force > > > > > and then producers will reconnect the partitions to continue > > > > > producing. However, with this change, the producers will fail to > > > > > reconnect due to `NotAllowedException`, Sometimes, users just want to > > > > > discard huge legacy data. Regarding the loss of partition metadata, > > it > > > > > can be recreated by creating the same partitioned topic again. I > > admit > > > > > it's a tricky solution. > > > > > > > > > > Thanks, > > > > > Yunze > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Apr 22, 2025 at 10:33 PM Zixuan Liu <zix...@apache.org> > > wrote: > > > > > > > > > > > > I'm aware that #24118 breaks some test cases, but I believe those > > test > > > > > > cases were incorrect. I also flagged this PR as introducing > > breaking > > > > > > changes in the mailing list, please see > > > > > > https://lists.apache.org/thread/g1jq0vkq0wt28mdnbg63bpw8fpx9ml42 > > > > > > > > > > > > I'd like to better understand the scope of the breakage. Is it > > limited > > > > to > > > > > > internal KoP usage—such as test cases where developers use > > > > > > partitioned-topic format without metadata? Or does it also affect > > > > users who > > > > > > might be doing this in practice? In our organization, we also > > maintain > > > > KoP > > > > > > and have updated our test cases accordingly. > > > > > > > > > > > > Personally, I believe that the partitioned topic should have > > associated > > > > > > metadata. If someone uses the partitioned format without metadata, > > it > > > > feels > > > > > > more like a workaround or an unsupported edge case. > > > > > > > > > > > > I'm open to discussion. If this behavior is something users > > actually > > > > depend > > > > > > on, then we should talk about how to handle it more gracefully. > > > > > > > > > > > > Thanks, > > > > > > Zixuan > > > > > > >
