+1 (binding) Regards, Penghui
On Wed, Jan 26, 2022 at 12:17 PM Michael Marshall <mmarsh...@apache.org> wrote: > +1 (non binding) - this proposal looks great! Thank you for a good > discussion of this feature! > > Thanks, > Michael > > On Tue, Jan 25, 2022 at 8:20 PM Hang Chen <chenh...@apache.org> wrote: > > > > +1 (binding) > > > > Thanks, > > Hang > > > > Jia Zhai <zhai...@apache.org> 于2022年1月26日周三 10:17写道： > > > > > > +1 > > > > > > > > > > > > On Tue, Jan 25, 2022 at 8:28 PM mattison chao <mattisonc...@gmail.com> > > > wrote: > > > > > > > +1 (non-binding) > > > > > > > > > On Jan 25, 2022, at 5:34 PM, Zike Yang <zky...@streamnative.io > .INVALID> > > > > wrote: > > > > > > > > > > Hi Pulsar Community, > > > > > > > > > > This is the voting thread for PIP-124. It will stay open for at > least > > > > > 48 hours. Pasted below for quoting convenience. > > > > > > > > > > Here is the issue for this PIP: > > > > https://github.com/apache/pulsar/issues/13408 > > > > > > > > > > ---- > > > > > > > > > > ## Motivation > > > > > > > > > > If we enable the DLQ when consuming messages. For some messages > that > > > > > can't be processed successfully, the messages will be moved to the > > > > > DLQ, but if we do not specify the data retention for the namespace > or > > > > > create a subscription for the DLQ to retain the data, the data of > the > > > > > DLQ will be removed automatically. Therefore, we need to create the > > > > > initial subscription before sending messages to the DLQ. > > > > > > > > > > ## Goal > > > > > > > > > > Users can set the initial subscription name in the > DeadLetterPolicy. > > > > > The consumer will create the initial subscription before sending > > > > > messages to the DLQ. At this point, subsequent messages produced to > > > > > the DLQ are not automatically deleted unexpectedly. If > > > > > `allowAutoSubscriptionCreation` in `broker.conf` is false, the > initial > > > > > subscription won't be created automatically. Otherwise, it will > > > > > confuse the user. Users can explicitly create that subscription to > > > > > handle this case. > > > > > > > > > > This PIP needs to be compatible with the previous behavior. The > > > > > initial subscription name in the DeadLetterPolicy is optional. The > > > > > default behavior is not to create the initial subscription which is > > > > > consistent with the original behavior. > > > > > > > > > > ## API Changes > > > > > > > > > > * Add `initSubscriptionName` to the `DeadLetterPolicy` > > > > > > > > > > ```java > > > > > /** > > > > > * Name of the initial subscription name of the dead letter topic. > > > > > * If this field is not set, the initial subscription will not be > created. > > > > > */ > > > > > private String initSubscriptionName; > > > > > ``` > > > > > > > > > > * Add a new field to the `CommandProducer`. The broker will use > this > > > > > field to create the initial subscription. > > > > > > > > > > ```proto > > > > > optional string initial_subscription_name > > > > > ``` > > > > > > > > > > * Add a new config to the Producer Configuration. This allows the > > > > > consumer to specify the initial subscription when creating the > > > > > deadLetterProducer. > > > > > > > > > > ```java > > > > > /** > > > > > * Use this config to automatically create an initial subscription > > > > > when creating the topic. > > > > > * If this field is not set, the initial subscription will not be > created. > > > > > * > > > > > * @param initialSubscriptionName > > > > > * Name of the initial subscription of the topic. > > > > > * @return the producer builder instance > > > > > */ > > > > > ProducerBuilder<T> initialSubscriptionName(String > > > > initialSubscriptionName); > > > > > ``` > > > > > > > > > > > > > > > ## Implementation > > > > > > > > > > When the deadLetterProducer is initialized, the consumer will set > the > > > > > initial subscription of the deadLetterProducer according to the > > > > > DeadLetterPolicy. > > > > > > > > > > When the broker creates a producer(after receiving the > > > > > CommandProducer), if it finds that the topic does not exist, it > will > > > > > not only create the topic(if the topic automatically creation is > > > > > enabled) but also create the initial subscription(if the initial > > > > > subscription name is set). When creating an initial subscription, > the > > > > > user needs to have the `canConsume` permission and the > > > > > `allowAutoSubscriptionCreation` needs to be enabled. > > > > > > > > > > ## Reject Alternatives > > > > > > > > > > ### Create the initial subscription using the consumer > > > > > > > > > > Before the deadLetterProducer is initialized, the consumer first > tries > > > > > to create a deadLetterConsumer using the initial subscription name > in > > > > > the DeadLetterPolicy. When this subscription already exists, the > > > > > ConsumerBusy exception will occur. In this case, we can ignore that > > > > > exception and create the deadLetterProducer. > > > > > > > > > > This is the original solution for this PIP. But this introduces too > > > > > much workload for the client. In the subsequent discussion, we try > to > > > > > add a new command `CommandCreateSubscription` to avoid creating and > > > > > closing the deadLetterConsumer, but again, this introduces > workload as > > > > > well as greater complexity. > > > > > > > > > > ### Use the retention policy to retent the data of the DLQ > > > > > > > > > > Before creating the deadLetterProducer, an admin can create a > > > > > retention policy for the topic or namespace. Then, consumers of the > > > > > topic have the duration of the retention policy to discover the > topic > > > > > and create a subscription before messages are lost. > > > > > > > > > > But users are not easy to set a different data retention policy or > > > > > create a new subscription for a lazy created DLQ topic. > > > > > > > > > > ### Add a new policy to retain data when the topic has no > subscriptions > > > > > > > > > > Adding a new policy for pulsar topics: a namespace or a topic level > > > > > policy that makes it possible to retain messages indefinitely when > a > > > > > topic has no subscriptions. > > > > > > > > > > For the auto-created topic, the subscription can only be > determined at > > > > > the time of creation. It may or may not create. If users are able > to > > > > > determine which consumers are, and these consumers need to receive > any > > > > > message sent by the producer, they should create the topic and > > > > > subscription manually or use the consumer to trigger the topic > > > > > auto-creation, not the producer. > > > > > > > > > > ### Add the initial subscription name field to the metadata of the > > > > > CommandProducer > > > > > > > > > > Use producer metadata to carry the init subscription name so that > we > > > > > don't need to introduce a new field in CommandProducer, and the new > > > > > field looks a little confusing. > > > > > > > > > > While this does not change protobuf, it still implicitly changes > the > > > > > protocol. This solution has different advantages and disadvantages > > > > > than the current one. Finally, we decided to go with the current > > > > > solution. > > > > > > > > > > In addition, there is a discussion about whether we need to create > the > > > > > initial subscription on the retryLetterTopic. The consumer has > already > > > > > subscribed to the retryLetterTopic and created the subscription > before > > > > > creating the retryLetterProducer, so we don't need to create the > > > > > initial subscription for it. > > > > > > > > > > > > > > > > > > > > Prototype implementation PR: > https://github.com/apache/pulsar/pull/13355 > > > > > > > > > > ---- > > > > > > > > > > Thanks > > > > > Zike Yang > > > > > > > > >
