+1
On Tue, Jan 25, 2022 at 8:28 PM mattison chao <mattisonc...@gmail.com> wrote: > +1 (non-binding) > > > On Jan 25, 2022, at 5:34 PM, Zike Yang <zky...@streamnative.io.INVALID> > wrote: > > > > Hi Pulsar Community, > > > > This is the voting thread for PIP-124. It will stay open for at least > > 48 hours. Pasted below for quoting convenience. > > > > Here is the issue for this PIP: > https://github.com/apache/pulsar/issues/13408 > > > > ---- > > > > ## Motivation > > > > If we enable the DLQ when consuming messages. For some messages that > > can't be processed successfully, the messages will be moved to the > > DLQ, but if we do not specify the data retention for the namespace or > > create a subscription for the DLQ to retain the data, the data of the > > DLQ will be removed automatically. Therefore, we need to create the > > initial subscription before sending messages to the DLQ. > > > > ## Goal > > > > Users can set the initial subscription name in the DeadLetterPolicy. > > The consumer will create the initial subscription before sending > > messages to the DLQ. At this point, subsequent messages produced to > > the DLQ are not automatically deleted unexpectedly. If > > `allowAutoSubscriptionCreation` in `broker.conf` is false, the initial > > subscription won't be created automatically. Otherwise, it will > > confuse the user. Users can explicitly create that subscription to > > handle this case. > > > > This PIP needs to be compatible with the previous behavior. The > > initial subscription name in the DeadLetterPolicy is optional. The > > default behavior is not to create the initial subscription which is > > consistent with the original behavior. > > > > ## API Changes > > > > * Add `initSubscriptionName` to the `DeadLetterPolicy` > > > > ```java > > /** > > * Name of the initial subscription name of the dead letter topic. > > * If this field is not set, the initial subscription will not be created. > > */ > > private String initSubscriptionName; > > ``` > > > > * Add a new field to the `CommandProducer`. The broker will use this > > field to create the initial subscription. > > > > ```proto > > optional string initial_subscription_name > > ``` > > > > * Add a new config to the Producer Configuration. This allows the > > consumer to specify the initial subscription when creating the > > deadLetterProducer. > > > > ```java > > /** > > * Use this config to automatically create an initial subscription > > when creating the topic. > > * If this field is not set, the initial subscription will not be created. > > * > > * @param initialSubscriptionName > > * Name of the initial subscription of the topic. > > * @return the producer builder instance > > */ > > ProducerBuilder<T> initialSubscriptionName(String > initialSubscriptionName); > > ``` > > > > > > ## Implementation > > > > When the deadLetterProducer is initialized, the consumer will set the > > initial subscription of the deadLetterProducer according to the > > DeadLetterPolicy. > > > > When the broker creates a producer(after receiving the > > CommandProducer), if it finds that the topic does not exist, it will > > not only create the topic(if the topic automatically creation is > > enabled) but also create the initial subscription(if the initial > > subscription name is set). When creating an initial subscription, the > > user needs to have the `canConsume` permission and the > > `allowAutoSubscriptionCreation` needs to be enabled. > > > > ## Reject Alternatives > > > > ### Create the initial subscription using the consumer > > > > Before the deadLetterProducer is initialized, the consumer first tries > > to create a deadLetterConsumer using the initial subscription name in > > the DeadLetterPolicy. When this subscription already exists, the > > ConsumerBusy exception will occur. In this case, we can ignore that > > exception and create the deadLetterProducer. > > > > This is the original solution for this PIP. But this introduces too > > much workload for the client. In the subsequent discussion, we try to > > add a new command `CommandCreateSubscription` to avoid creating and > > closing the deadLetterConsumer, but again, this introduces workload as > > well as greater complexity. > > > > ### Use the retention policy to retent the data of the DLQ > > > > Before creating the deadLetterProducer, an admin can create a > > retention policy for the topic or namespace. Then, consumers of the > > topic have the duration of the retention policy to discover the topic > > and create a subscription before messages are lost. > > > > But users are not easy to set a different data retention policy or > > create a new subscription for a lazy created DLQ topic. > > > > ### Add a new policy to retain data when the topic has no subscriptions > > > > Adding a new policy for pulsar topics: a namespace or a topic level > > policy that makes it possible to retain messages indefinitely when a > > topic has no subscriptions. > > > > For the auto-created topic, the subscription can only be determined at > > the time of creation. It may or may not create. If users are able to > > determine which consumers are, and these consumers need to receive any > > message sent by the producer, they should create the topic and > > subscription manually or use the consumer to trigger the topic > > auto-creation, not the producer. > > > > ### Add the initial subscription name field to the metadata of the > > CommandProducer > > > > Use producer metadata to carry the init subscription name so that we > > don't need to introduce a new field in CommandProducer, and the new > > field looks a little confusing. > > > > While this does not change protobuf, it still implicitly changes the > > protocol. This solution has different advantages and disadvantages > > than the current one. Finally, we decided to go with the current > > solution. > > > > In addition, there is a discussion about whether we need to create the > > initial subscription on the retryLetterTopic. The consumer has already > > subscribed to the retryLetterTopic and created the subscription before > > creating the retryLetterProducer, so we don't need to create the > > initial subscription for it. > > > > > > > > Prototype implementation PR: https://github.com/apache/pulsar/pull/13355 > > > > ---- > > > > Thanks > > Zike Yang > >
