Hi Pulsar Community, This is the voting thread for PIP-124. It will stay open for at least 48 hours. Pasted below for quoting convenience.
Here is the issue for this PIP: https://github.com/apache/pulsar/issues/13408 ---- ## Motivation If we enable the DLQ when consuming messages. For some messages that can't be processed successfully, the messages will be moved to the DLQ, but if we do not specify the data retention for the namespace or create a subscription for the DLQ to retain the data, the data of the DLQ will be removed automatically. Therefore, we need to create the initial subscription before sending messages to the DLQ. ## Goal Users can set the initial subscription name in the DeadLetterPolicy. The consumer will create the initial subscription before sending messages to the DLQ. At this point, subsequent messages produced to the DLQ are not automatically deleted unexpectedly. If `allowAutoSubscriptionCreation` in `broker.conf` is false, the initial subscription won't be created automatically. Otherwise, it will confuse the user. Users can explicitly create that subscription to handle this case. This PIP needs to be compatible with the previous behavior. The initial subscription name in the DeadLetterPolicy is optional. The default behavior is not to create the initial subscription which is consistent with the original behavior. ## API Changes * Add `initSubscriptionName` to the `DeadLetterPolicy` ```java /** * Name of the initial subscription name of the dead letter topic. * If this field is not set, the initial subscription will not be created. */ private String initSubscriptionName; ``` * Add a new field to the `CommandProducer`. The broker will use this field to create the initial subscription. ```proto optional string initial_subscription_name ``` * Add a new config to the Producer Configuration. This allows the consumer to specify the initial subscription when creating the deadLetterProducer. ```java /** * Use this config to automatically create an initial subscription when creating the topic. * If this field is not set, the initial subscription will not be created. * * @param initialSubscriptionName * Name of the initial subscription of the topic. * @return the producer builder instance */ ProducerBuilder<T> initialSubscriptionName(String initialSubscriptionName); ``` ## Implementation When the deadLetterProducer is initialized, the consumer will set the initial subscription of the deadLetterProducer according to the DeadLetterPolicy. When the broker creates a producer(after receiving the CommandProducer), if it finds that the topic does not exist, it will not only create the topic(if the topic automatically creation is enabled) but also create the initial subscription(if the initial subscription name is set). When creating an initial subscription, the user needs to have the `canConsume` permission and the `allowAutoSubscriptionCreation` needs to be enabled. ## Reject Alternatives ### Create the initial subscription using the consumer Before the deadLetterProducer is initialized, the consumer first tries to create a deadLetterConsumer using the initial subscription name in the DeadLetterPolicy. When this subscription already exists, the ConsumerBusy exception will occur. In this case, we can ignore that exception and create the deadLetterProducer. This is the original solution for this PIP. But this introduces too much workload for the client. In the subsequent discussion, we try to add a new command `CommandCreateSubscription` to avoid creating and closing the deadLetterConsumer, but again, this introduces workload as well as greater complexity. ### Use the retention policy to retent the data of the DLQ Before creating the deadLetterProducer, an admin can create a retention policy for the topic or namespace. Then, consumers of the topic have the duration of the retention policy to discover the topic and create a subscription before messages are lost. But users are not easy to set a different data retention policy or create a new subscription for a lazy created DLQ topic. ### Add a new policy to retain data when the topic has no subscriptions Adding a new policy for pulsar topics: a namespace or a topic level policy that makes it possible to retain messages indefinitely when a topic has no subscriptions. For the auto-created topic, the subscription can only be determined at the time of creation. It may or may not create. If users are able to determine which consumers are, and these consumers need to receive any message sent by the producer, they should create the topic and subscription manually or use the consumer to trigger the topic auto-creation, not the producer. ### Add the initial subscription name field to the metadata of the CommandProducer Use producer metadata to carry the init subscription name so that we don't need to introduce a new field in CommandProducer, and the new field looks a little confusing. While this does not change protobuf, it still implicitly changes the protocol. This solution has different advantages and disadvantages than the current one. Finally, we decided to go with the current solution. In addition, there is a discussion about whether we need to create the initial subscription on the retryLetterTopic. The consumer has already subscribed to the retryLetterTopic and created the subscription before creating the retryLetterProducer, so we don't need to create the initial subscription for it. Prototype implementation PR: https://github.com/apache/pulsar/pull/13355 ---- Thanks Zike Yang
