Thanks for your explanation. It now makes sense to me. So I suggest:
1. Document this use case in the PIP
2. Document the result that resetting cursor might lead to in the API
doc of this configuration
Thanks,
Yunze
On Wed, Mar 29, 2023 at 9:11 PM 丛搏 wrote:
>
> Hi, Yunze:
>
> > It's better to de
Hi, Yunze:
> It's better to describe how it could bring the benefit to transaction
> use cases, since now it's designed to be a configuration related to
> the transaction.
sorry, that I haven't explained in detail why the transaction needs it.
let's look at a simple example:
```
Transaction txn =
It's better to describe how it could bring the benefit to transaction
use cases, since now it's designed to be a configuration related to
the transaction.
I've thought about some cases:
1. A consumer received N messages, then the cursor was reset to the earliest.
2. A consumer received N messages
Hi, all :
Thanks to everyone who discussed it.
Our current care points include the following aspects:
1. The filtering efficiency of the client consumer is not as
good as doing something directly in startMessageId
2. Does not support reset cursor
Because my previous PIP description is to add co
Hi, Michael:
I thought about it carefully, and using 'startMessageId'
is indeed a good idea. But it is more complicated, we
need to ensure its absolute correctness, and take
performance into consideration. If you can come up
with a closed-loop solution based on 'startMessageId',
I support you. If
Hi, PengHui
PengHui Li 于2023年3月24日周五 10:16写道：
>
> Hi Bo,
>
> > Only support Consumer#redeliverUnacknowledgedMessages()
> >
> > If we redeliver individual messages, they will be filtered. Because we
> can't clear the record latest message in the consumer when redelivering
> individual messages. It
Hi, Yunze :
>
> > At least it will simplify the process of using cumulative ack with the
> > transaction.
>
> So the transaction use cases assume the cursor will never be reset by
> other applications? We cannot ignore the impact brought by this
> proposal that resetting the cursor could have an
> At least it will simplify the process of using cumulative ack with the
> transaction.
So the transaction use cases assume the cursor will never be reset by
other applications? We cannot ignore the impact brought by this
proposal that resetting the cursor could have an unexpected effect.
If the
Hi Bo,
> Only support Consumer#redeliverUnacknowledgedMessages()
>
> If we redeliver individual messages, they will be filtered. Because we
can't clear the record latest message in the consumer when redelivering
individual messages. It will make this config unclear, and if every
redeliver method c
Hi, Michael:
>
> Is this the underlying motivation for the PIP?
>
> From my perspective, the PIP is seeking to decrease duplicate messages
> experienced due to disconnections from the broker.
This PIP is not aimed at reducing message duplication but at
completely preventing it.
Therefore we have t
Because we already send the `startMessageId`, there is a chance where
we might not even need to update the protocol for the
CommandSubscribe. In light of that, I quickly put together a PR
showing how that field might be used to inform the broker where to
start the read position for the cursor.
htt
I am not following your objections to the protocol solution. It might
be more productive if I provided a draft PR with a sample
implementation. I'm not sure that I'll have time, but I'll try to put
something together this week.
> At least it will simplify the process of using cumulative ack with t
Hi, Michael
> In this case, the consumer does not have the source of truth for the
> readPosition. It would leave the new protocol field for `readPosition`
> empty and the broker would use its source of truth for the read
> position.
application has received all the messages by application thread.
Hi, Yunze:
It is true that hiding these details in the SDK still requires users to have a
certain level of understanding of the configuration. However, this approach
can still be helpful for users who want to use the feature but want
to avoid dealing with the nitty-gritty details of implementation
Good questions. There were some gaps in my description above.
> I think it is the[0], when the `incomingMessages` queue is empty,
> how do we get the correct `startPosition`?
In this case, the consumer does not have the source of truth for the
readPosition. It would leave the new protocol field f
I just missed the point that the reset cursor operations do not work
for the consumer. IIUC, the seek operation does not work as well. Then
I think the option is not user-friendly as the PIP says:
> It needs to be enabled with a complete understanding of this configuration.
If users want, they c
Hi, Michael:
Michael Marshall 于2023年3月21日周二 23:17写道：
>
> One more point. Instead of keeping track of the latest message seen by
> the application, the logic in my solution would actually just check
> the last message in the `incomingMessages` queue (as in the most
> recently added), and use that
One more point. Instead of keeping track of the latest message seen by
the application, the logic in my solution would actually just check
the last message in the `incomingMessages` queue (as in the most
recently added), and use that as the read position in the subscribe
command. If we made this ch
> if we add the new field in CommandSubscribe, we should ensure
> the synchronization between consumer reconnection and user
> calling receive and redeliverUnack method. it will affect the performance
> of receive. expose synchronization to hot paths it not a good idea.
I don't think this is a val
+1
Hi, Bo :
Thanks for your explanation. That makes sense to me.
Thanks,
Yubiao Feng
On Mon, Mar 20, 2023 at 10:21 PM 丛搏 wrote:
> Hi, pulsar community:
>
> I started a PIP about `Client consumer filter received messages`.
>
> PIP: https://github.com/apache/pulsar/issues/19864
>
> Thanks,
> Bo
Hi, Aloys:
Yes, it will work with `consumer.seek()`.
Sorry, I missed this, I will add this description to the PIP.
But the current seek method has some problems, the detail in
https://lists.apache.org/thread/97o9t4ltkds5pfq41l9xbbd31t41qm8w,
I am not sure, does it make sense to support seek metho
Hi, Yunze:
Yunze Xu 于2023年3月21日周二 17:19写道：
>
>
> Is there any breaking change if we just apply this filter without
> adding a configuration option?
If not add this configuration, the Pulsar Admin reset cursor will cause
the wrong behavior. It will filter the messages which have been reset.
As d
Nice proposal.
I'm interested in a point
> So when we need to reset the cursor, the client consumer should all be
closed, and then reset the cursor then restart the consumer.
Does this requirement apply to `consumer.seek`?
Because in some scenarios, we need to create consumers first and then see
First, I agree with Yubiao that we can avoid calling the `isDuplicate`
method once this option is enabled.
Then, I'm wondering in which case would users want to disable this
option? What's the disadvantage to disable the option? I think we can
just record the latest position (ledger id, entry id,
Hi, Michael
Michael Marshall 于2023年3月21日周二 13:03写道：
>
> This is a great problem to improve.
>
> What if we instead expand the CommandSubscribe [0] protocol message
> with a new field to represent the client's desired read position? This
> way, the client can tell the second broker where to start
Hi, Xiangying
Xiangying Meng 于2023年3月20日周一 23:56写道：
>
> Hi Congbo,
> I think this is a great idea.
> This is more efficient in filtering duplicate messages for a single
> consumer.
> And maybe more details about implementation should be shown in the proposal.
we add the public interface, the cod
Hi, Yubiao :
Yubiao Feng 于2023年3月20日周一 22:53写道：
>
> Hi Bo
>
> I think this is a good way to filter messages that the client has received.
>
> And I have two questions:
>
> 1. This is more powerful than the original way
> (`acknowledgmentsGroupingTracker.isDuplicate(msgId)) to filter out
> duplica
This is a great problem to improve.
What if we instead expand the CommandSubscribe [0] protocol message
with a new field to represent the client's desired read position? This
way, the client can tell the second broker where to start sending
messages, and there is no need to send the messages twice
Hi Congbo,
I think this is a great idea.
This is more efficient in filtering duplicate messages for a single
consumer.
And maybe more details about implementation should be shown in the proposal.
Best regards,
Xiangying
On Mon, Mar 20, 2023 at 10:53 PM Yubiao Feng
wrote:
> Hi Bo
>
> I think thi
Hi Bo
I think this is a good way to filter messages that the client has received.
And I have two questions:
1. This is more powerful than the original way
(`acknowledgmentsGroupingTracker.isDuplicate(msgId)) to filter out
duplicated messages.
Is it possible to turn off the original de-replay lo
Hi, pulsar community:
I started a PIP about `Client consumer filter received messages`.
PIP: https://github.com/apache/pulsar/issues/19864
Thanks,
Bo
31 matches
Mail list logo
