Hi Girish and Zike,
@Girish,
Regarding your question about the scope of the regex-based deletion
feature, the initial proposal is to delete the specified topics or
namespaces only in the local cluster where the command is run. If you
need to delete them in remote clusters as well, you can simply run the
command again in the respective clusters.
@Zike,
You raised a valid point about the regex-based deletion feature not
covering the case Yubiao mentioned. To address this, we can introduce
another flag (e.g., `--from-file`) that allows users to read a list of
namespaces and topics from a file for deletion. This would cater to
situations where an arbitrary list of topics/namespaces needs to be
deleted, providing a more comprehensive solution.
Let me know if you have any further questions or suggestions.
Best regards,
Xiangying Meng
On Tue, Apr 25, 2023 at 5:23 PM Zike Yang <z...@apache.org> wrote:
>
> Hi, Xiangying
>
> > 1. I understand that one of your concerns is whether the proposed
> > regex-based deletion feature would provide significant advantages over
> > using a simple one-liner script to call the delete topic command for
> > each topic. As Yubiao pointed out, using scripts to delete topics one
> > by one can lead to increased network overhead and slow performance,
> > particularly when dealing with a large number of topics. Implementing
> > regex support for delete operations would provide a more efficient and
> > convenient way to manage resources in Pulsar.
>
> From my understanding, introducing regex-based deletion feature
> doesn't solve the case yubiao mentioned above. They are two different
> cases.
> Regex-based deletion can only delete topics of the same format, not an
> arbitrary list of topics.
>
>
> BR,
> Zike Yang
> On Tue, Apr 25, 2023 at 11:22 AM Xiangying Meng <xiangy...@apache.org> wrote:
> >
> > Hi Girish,
> >
> > Thank you for raising concerns about the proposed feature. I would
> > like to address the points you mentioned in your email.
> >
> > 1. I understand that one of your concerns is whether the proposed
> > regex-based deletion feature would provide significant advantages over
> > using a simple one-liner script to call the delete topic command for
> > each topic. As Yubiao pointed out, using scripts to delete topics one
> > by one can lead to increased network overhead and slow performance,
> > particularly when dealing with a large number of topics. Implementing
> > regex support for delete operations would provide a more efficient and
> > convenient way to manage resources in Pulsar.
> >
> > 2. In addition to the benefits for testing purposes, we have
> > communicated with business users of Pulsar and found that the proposed
> > regex-based deletion feature can be helpful in production environments
> > as well. For instance, it can be used to efficiently clean up
> > subscriptions associated with deprecated services, ensuring better
> > resource management and reducing clutter in the system.
> >
> > 3. As I suggested earlier, we can introduce a new option flag (e.g.,
> > `--regex` or `--pattern`) to the existing `pulsar-admin topics delete`
> > command to prevent breaking changes for users who have already used
> > the command in their scripts. This would ensure backward compatibility
> > while providing the new functionality for those who want to use regex
> > for deletion.
> >
> > I hope this clears up any confusion and addresses your concerns.
> > Please let me know if you have any further questions or suggestions.
> >
> > Best regards,
> > Xiangying Meng
> >
> > On Mon, Apr 24, 2023 at 6:23 PM Girish Sharma <scrapmachi...@gmail.com>
> > wrote:
> > >
> > > Hello Yubiao,
> > > As per my understanding, this feature suggestion is intended to delete the
> > > topics from all replicated clusters under the namespace. Thus, the example
> > > you are providing may not be a good fit for this?
> > >
> > > Xiangying, please clarify if my understanding is incorrect.
> > >
> > > On Mon, Apr 24, 2023 at 3:24 PM Yubiao Feng
> > > <yubiao.f...@streamnative.io.invalid> wrote:
> > >
> > > > Hi Girish Sharma
> > > >
> > > > > What additional advantage would one get by using that approach
> > > > > rather than simply using a one liner script to just call delete
> > > > > topic for each of those topics if the list of topics is known.
> > > >
> > > > If users enabled `Geo-Replication` on a namespace in mistake(expected
> > > > only to enable one topic),
> > > > it is possible to create many topics on the remote cluster in one
> > > > second.
> > > >
> > > > Not long ago, 10,000 topics were created per second because of this
> > > > mistake. It took us a long time to
> > > > remove these topics. We delete these topics in this way:
> > > > ```
> > > > cat topics_name_file | awk '{system("bin/pulsar-admin topics delete
> > > > "$0)}'
> > > > )
> > > > ```
> > > > It deletes topics one by one.
> > > >
> > > > We conclude later that stress test tools such as `Jmeter` or `ab`
> > > > should be
> > > > used to delete so many topics.
> > > >
> > > > If Pulsar could provide these APIs, it would be better.
> > > >
> > > > Thanks
> > > > Yubiao Feng
> > > >
> > > >
> > > >
> > > >
> > > > On Wed, Apr 19, 2023 at 3:29 PM Girish Sharma <scrapmachi...@gmail.com>
> > > > wrote:
> > > >
> > > > > Hello Yubiao,
> > > > >
> > > > > What additional advantage would one get by using that approach rather
> > > > than
> > > > > simply using a one liner script to just call delete topic for each of
> > > > those
> > > > > topics if the list of topics is known.
> > > > >
> > > > > Regards
> > > > >
> > > > > On Wed, Apr 19, 2023 at 12:54 PM Yubiao Feng
> > > > > <yubiao.f...@streamnative.io.invalid> wrote:
> > > > >
> > > > > > In addition to these two, It is recommended to add a method to batch
> > > > > delete
> > > > > > topics, such as this:
> > > > > >
> > > > > > ```
> > > > > > pulsar-admin topics delete-all-topics <topic_1>, <topic_2>
> > > > > >
> > > > > > or
> > > > > >
> > > > > > pulsar-admin topics delete-all-topic <a file contains a topic_name
> > > > lists>
> > > > > > ```
> > > > > >
> > > > > > Thanks
> > > > > > Yubiao Feng
> > > > > >
> > > > > > On Sat, Apr 15, 2023 at 5:37 PM Xiangying Meng
> > > > > > <xiangy...@apache.org>
> > > > > > wrote:
> > > > > >
> > > > > > > Dear Apache Pulsar Community,
> > > > > > >
> > > > > > > I hope this email finds you well.I am writing to suggest a
> > > > > > > potential
> > > > > > > improvement to the Pulsar-admin tool,
> > > > > > > which I believe could simplify the process of cleaning up tenants
> > > > and
> > > > > > > namespaces in Apache Pulsar.
> > > > > > >
> > > > > > > Currently, cleaning up all the namespaces and topics within a
> > > > > > > tenant
> > > > or
> > > > > > > cleaning up all the topics within a namespace requires several
> > > > > > > manual
> > > > > > > steps,
> > > > > > > such as listing the namespaces, listing the topics, and then
> > > > > > > deleting
> > > > > > each
> > > > > > > topic individually.
> > > > > > > This process can be time-consuming and error-prone for users.
> > > > > > >
> > > > > > > To address this issue, I propose the addition of a "clear"
> > > > > > > parameter
> > > > to
> > > > > > the
> > > > > > > Pulsar-admin tool,
> > > > > > > which would automate the cleanup process for tenants and
> > > > > > > namespaces.
> > > > > > Here's
> > > > > > > a conceptual implementation:
> > > > > > >
> > > > > > > 1. To clean up all namespaces and topics within a tenant:
> > > > > > > ``` bash
> > > > > > > pulsar-admin tenants clear <tenant-name>
> > > > > > > ```
> > > > > > > 2. To clean up all topics within a namespace:
> > > > > > > ```bash
> > > > > > > pulsar-admin namespaces clear <tenant-name>/<namespace-name>
> > > > > > > ```
> > > > > > >
> > > > > > > By implementing these new parameters, users would be able to
> > > > > > > perform
> > > > > > > cleanup operations more efficiently and with fewer manual steps.
> > > > > > > I believe this improvement would greatly enhance the user
> > > > > > > experience
> > > > > when
> > > > > > > working with Apache Pulsar.
> > > > > > >
> > > > > > > I'd like to discuss the feasibility of this suggestion and gather
> > > > > > feedback
> > > > > > > from the community.
> > > > > > > If everyone agrees, I can work on implementing this feature and
> > > > submit
> > > > > a
> > > > > > > pull request for review.
> > > > > > >
> > > > > > > Looking forward to hearing your thoughts on this.
> > > > > > >
> > > > > > > Best regards,
> > > > > > > Xiangying
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Girish Sharma
> > > > >
> > > >
> > >
> > >
> > > --
> > > Girish Sharma
