This is an automated email from the ASF dual-hosted git repository.
yubiao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/pulsar.git
The following commit(s) were added to refs/heads/master by this push:
new 64bacfd483f [improve][pip]PIP-422 Support global topic-level policy:
replicated clusters and new API to delete topic-level policies (#24368)
64bacfd483f is described below
commit 64bacfd483fa061d5c1604ac72146d80cfdade29
Author: fengyubiao <[email protected]>
AuthorDate: Wed Jun 18 00:34:20 2025 +0800
[improve][pip]PIP-422 Support global topic-level policy: replicated
clusters and new API to delete topic-level policies (#24368)
Co-authored-by: Penghui Li <[email protected]>
---
pip/pip-422.md | 106 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 106 insertions(+)
diff --git a/pip/pip-422.md b/pip/pip-422.md
new file mode 100644
index 00000000000..81efa28b0d3
--- /dev/null
+++ b/pip/pip-422.md
@@ -0,0 +1,106 @@
+# PIP-422: Support global topic-level policy: replicated clusters and new API
to delete topic-level policies
+
+# Background knowledge
+
+Users have two choices when configuring the metadata store when using the
feature Geo-Replication: shared or not.
+All clusters share data that are stored in the global config metadata store if
users choose to use a shared metadata store.
+By the way, the share config metadata store is often built across multiple
zones/regions to ensure fault tolerance.
+
+Since the data that is stored in the shared metadata store contains the
following pulsar resources: tenants, namespaces, partitioned topics,
+the Admin API can not be used to delete these pulsar resources for a single
cluster.
+Pulsar has a feature that is used to delete namespace-level resources for a
specific cluster when using a shared metadata store, it works as follows:
+- The original namespace-level policy that named `replicated clusters`
contains two clusters `c1` and `c2`.
+- The topics under the cluster `c1` will be deleted automatically when users
remove `c1` from the namespace-level policy.
+
+The feature above is very useful in the following scenarios:
+- Transfer a namespace from an old cluster to a new cluster: at last, the
namespace should be deleted from the old cluster.
+- Delete topics: since Pulsar does not allow deleting topics when enabled
Geo-Replication, users should disable replication first.
+
+# Motivation
+
+When using a shared metadata store and enabling Geo-Replication, there is no
mechanism for the following scenarios:
+- Transfer a partitioned topic from an old cluster to a new cluster.
+- Delete a partitioned topic from a specified cluster.
+
+# Goals & Designs
+
+Add two new APIs
+- `pulsar-admin topicPolicies --set-replication-cluster -c <clusters> --g
<topic>`
+ - Pulsar only supports setting a local topic-level `replicated clusters`,
and does not support global policies so far.
+- `pulsar-admin topicPolicies delete <topic>`
+ - To delete topic level policies, including global and local policies, it
can be called even if the topic has been deleted.
+
+Old Apis
+- Prohibits users from using local policies to delete the local cluster.
+
+
+The new APIs will be used for the following scenarios:
+
+A. Delete a partitioned topic under all clusters, if enabled Geo-Replication
with a shared config metadata store.
+- Disable global topic level `replicated clusters`, in other words, set
`replicated clusters` with a single list.
+ - The clusters that are not contained in the `replicated clusters` will
delete the sub-topics automatically, such as the namespace-level implementation.
+ - The schemas and local topic policies of the topic will be deleted after
the latest sub-topic is deleted.
+- Delete the partitioned topic since we have disabled Geo-Replication.
+- Manually delete global topic-level policies by the new API `pulsar-admin
topicPolicies delete <topic>` for other clusters that are not contained in
`replicated clusters`.
+- Note: If there is any active producers or consumers connected to the cluster
which been removed from the replicated cluster list, they will be rejected from
reconnecting because the connected cluster is not listed in `replicated
clusters`.
+
+B. Delete a partitioned topic under a specified cluster.
+- Support setting up a global topic-level `replicated clusters` policy, which
does not contain the cluster that you want to remove.
+ - Broker will delete the subtopics automatically.
+ - The schemas and local topic policies of the topic will be deleted after
the latest sub-topic is deleted.
+- Note:
+ - The global topic-level policy will not be removed, to avoid the
namespace-level `replicated cluster` from taking effect.
+ - Once you want to delete the topic under all clusters, please use Solution
A to remove the global topic-level policy left.
+
+**Question**: Why not add a mechanism that brokers clear global topic-level
policies so that the related topic is deleted regularly, which avoids calling
`pulsar-admin topicPolicies delete <topic>`?
+
+**Answer**: When users quickly delete and create topics, it will result in the
previous policies not being deleted by the scheduled cleaning mechanism,
thereby causing confusion.
+
+### Public API
+
+**new Pulsar Admin API**
+- `pulsar-admin topicPolicies --set-replication-cluster <clusters> --global
<topic>`
+- `pulsar-admin topicPolicies delete <topic>`
+- The API `pulsar-admin topics --set-replication-cluster <clusters> <topic>`
never supports to delete the local cluster anymore.
+
+### Binary protocol
+Nothing.
+
+### Configuration
+Nothing.
+
+### CLI
+Nothing.
+
+### Metrics
+Nothing.
+
+# Monitoring
+Nothing.
+
+# Security Considerations
+Nothing.
+
+# Backward & Forward Compatibility
+Nothing.
+
+## Upgrade
+Nothing.
+
+## Downgrade / Rollback
+Nothing.
+
+## Pulsar Geo-Replication Upgrade & Downgrade/Rollback Considerations
+Nothing.
+
+# Alternatives
+Passed solutions:
+1. Disable namespace-level Geo-Replication then delete the partitioned topic:
this will trigger a deletion for other topics, which is not expected.
+2. Disable topic-level Geo-Replication under each cluster, and delete the
partitioned topic by Admin API: it will delete the shared partitioned topic
metadata, which was also deleted from another cluster.
+3. Disable topic-level Geo-Replication under each cluster, and delete the
sub-topics manually by Admin API: this is the best solution without the current
PIP, the defect is that the schema data and topic-level policies are still
there.
+# General Notes
+
+# Links
+
+* Mailing List discussion thread:
https://lists.apache.org/thread/5rh536dqm2o35tzop5jl7b5x7r291pv2
+* Mailing List voting thread:
https://lists.apache.org/thread/p0wmnfr3jkmfzh69jfh7jv10184g1vlm
