+1 (binding) Thanks, Hang
Jia Zhai <zhai...@apache.org> 于2022年1月26日周三 10:17写道： > > +1 > > > > On Tue, Jan 25, 2022 at 8:28 PM mattison chao <mattisonc...@gmail.com> > wrote: > > > +1 (non-binding) > > > > > On Jan 25, 2022, at 5:34 PM, Zike Yang <zky...@streamnative.io.INVALID> > > wrote: > > > > > > Hi Pulsar Community, > > > > > > This is the voting thread for PIP-124. It will stay open for at least > > > 48 hours. Pasted below for quoting convenience. > > > > > > Here is the issue for this PIP: > > https://github.com/apache/pulsar/issues/13408 > > > > > > ---- > > > > > > ## Motivation > > > > > > If we enable the DLQ when consuming messages. For some messages that > > > can't be processed successfully, the messages will be moved to the > > > DLQ, but if we do not specify the data retention for the namespace or > > > create a subscription for the DLQ to retain the data, the data of the > > > DLQ will be removed automatically. Therefore, we need to create the > > > initial subscription before sending messages to the DLQ. > > > > > > ## Goal > > > > > > Users can set the initial subscription name in the DeadLetterPolicy. > > > The consumer will create the initial subscription before sending > > > messages to the DLQ. At this point, subsequent messages produced to > > > the DLQ are not automatically deleted unexpectedly. If > > > `allowAutoSubscriptionCreation` in `broker.conf` is false, the initial > > > subscription won't be created automatically. Otherwise, it will > > > confuse the user. Users can explicitly create that subscription to > > > handle this case. > > > > > > This PIP needs to be compatible with the previous behavior. The > > > initial subscription name in the DeadLetterPolicy is optional. The > > > default behavior is not to create the initial subscription which is > > > consistent with the original behavior. > > > > > > ## API Changes > > > > > > * Add `initSubscriptionName` to the `DeadLetterPolicy` > > > > > > ```java > > > /** > > > * Name of the initial subscription name of the dead letter topic. > > > * If this field is not set, the initial subscription will not be created. > > > */ > > > private String initSubscriptionName; > > > ``` > > > > > > * Add a new field to the `CommandProducer`. The broker will use this > > > field to create the initial subscription. > > > > > > ```proto > > > optional string initial_subscription_name > > > ``` > > > > > > * Add a new config to the Producer Configuration. This allows the > > > consumer to specify the initial subscription when creating the > > > deadLetterProducer. > > > > > > ```java > > > /** > > > * Use this config to automatically create an initial subscription > > > when creating the topic. > > > * If this field is not set, the initial subscription will not be created. > > > * > > > * @param initialSubscriptionName > > > * Name of the initial subscription of the topic. > > > * @return the producer builder instance > > > */ > > > ProducerBuilder<T> initialSubscriptionName(String > > initialSubscriptionName); > > > ``` > > > > > > > > > ## Implementation > > > > > > When the deadLetterProducer is initialized, the consumer will set the > > > initial subscription of the deadLetterProducer according to the > > > DeadLetterPolicy. > > > > > > When the broker creates a producer(after receiving the > > > CommandProducer), if it finds that the topic does not exist, it will > > > not only create the topic(if the topic automatically creation is > > > enabled) but also create the initial subscription(if the initial > > > subscription name is set). When creating an initial subscription, the > > > user needs to have the `canConsume` permission and the > > > `allowAutoSubscriptionCreation` needs to be enabled. > > > > > > ## Reject Alternatives > > > > > > ### Create the initial subscription using the consumer > > > > > > Before the deadLetterProducer is initialized, the consumer first tries > > > to create a deadLetterConsumer using the initial subscription name in > > > the DeadLetterPolicy. When this subscription already exists, the > > > ConsumerBusy exception will occur. In this case, we can ignore that > > > exception and create the deadLetterProducer. > > > > > > This is the original solution for this PIP. But this introduces too > > > much workload for the client. In the subsequent discussion, we try to > > > add a new command `CommandCreateSubscription` to avoid creating and > > > closing the deadLetterConsumer, but again, this introduces workload as > > > well as greater complexity. > > > > > > ### Use the retention policy to retent the data of the DLQ > > > > > > Before creating the deadLetterProducer, an admin can create a > > > retention policy for the topic or namespace. Then, consumers of the > > > topic have the duration of the retention policy to discover the topic > > > and create a subscription before messages are lost. > > > > > > But users are not easy to set a different data retention policy or > > > create a new subscription for a lazy created DLQ topic. > > > > > > ### Add a new policy to retain data when the topic has no subscriptions > > > > > > Adding a new policy for pulsar topics: a namespace or a topic level > > > policy that makes it possible to retain messages indefinitely when a > > > topic has no subscriptions. > > > > > > For the auto-created topic, the subscription can only be determined at > > > the time of creation. It may or may not create. If users are able to > > > determine which consumers are, and these consumers need to receive any > > > message sent by the producer, they should create the topic and > > > subscription manually or use the consumer to trigger the topic > > > auto-creation, not the producer. > > > > > > ### Add the initial subscription name field to the metadata of the > > > CommandProducer > > > > > > Use producer metadata to carry the init subscription name so that we > > > don't need to introduce a new field in CommandProducer, and the new > > > field looks a little confusing. > > > > > > While this does not change protobuf, it still implicitly changes the > > > protocol. This solution has different advantages and disadvantages > > > than the current one. Finally, we decided to go with the current > > > solution. > > > > > > In addition, there is a discussion about whether we need to create the > > > initial subscription on the retryLetterTopic. The consumer has already > > > subscribed to the retryLetterTopic and created the subscription before > > > creating the retryLetterProducer, so we don't need to create the > > > initial subscription for it. > > > > > > > > > > > > Prototype implementation PR: https://github.com/apache/pulsar/pull/13355 > > > > > > ---- > > > > > > Thanks > > > Zike Yang > > > >
