Hi all
Is there any other comments about this design :)
Thanks,
Xiaoyu Hou
houxiaoyu 于2023年3月8日周三 16:22写道：
> Hi Michael,
>
> > is there a reason that we couldn't also use this to improve PIP 145?
>
> The protocol described in this PIP could also be used to improve PIP-145.
> However I think th
Hi Michael,
> is there a reason that we couldn't also use this to improve PIP 145?
The protocol described in this PIP could also be used to improve PIP-145.
However I think that it' not a good reason that we use the regex sub
watcher to implement the partitioned update watcher because of the oth
Thanks for the context Xiaoyu Hou and Asaf. I appreciate the
efficiencies that we can gain by creating a specific implementation
for the partitioned topic use case. I agree that this new notification
system makes sense based on Pulsar's current features, and I have some
implementation questions.
>
Bump. Are there other concerns or suggestions about this PIP :) Ping @
Michael @Joe @Enrico
Thanks
Xiaoyu Hou
houxiaoyu 于2023年2月27日周一 14:10写道：
> Hi Joe and Michael,
>
> I think I misunderstood what you replied before. Now I understand and
> explain it again.
>
> Besides the reasons what Asaf m
Hi Joe and Michael,
I think I misunderstood what you replied before. Now I understand and
explain it again.
Besides the reasons what Asaf mentioned above, there are also some limits
for using topic list watcher. For example the `topicsPattern.pattern` must
less that `maxSubscriptionPatternLeng`
Hi Asaf,
> Prefer something more explicit: onTopicPartitionsCountChanged
> Maybe we should mark "PartitionsChangedListener" deprecated?
> 60 seconds seems like *a lot* to miss an update. Maybe 5-10 sec
It looks good to me
> Let's think about the case of endless retries
We will remove the watch
Hi Michael,
> I think we just need the client to "subscribe" to a topic notification
for
> "-partition-[0-9]+" to eliminate the polling
If pulsar users want to pub/sub a partitioned-topic, I think most of the
users would like to create a simple producer or consumer like following:
```
Producer
Michael,
I think the current suggested protocol is more optimized compared to
existing regex mechanism:
- If the broker sends notification and it's lost due network issues,
you'll only know about it due to the client doing constant polling, using
its hash to minimize response.
- I
> Just the way to implements partitioned-topic metadata
> notification mechanism is much like notifications on regex sub changes
Why do we need a separate notification implementation? The regex
subscription feature is about discovering topics (not subscriptions)
that match a regular expression. As
Hi Joe,
When we use PartitionedProducerImpl or MultiTopicsConsumerImpl, there is a
poll task to fetch the metadata of the partitioned-topic regularly for the
number of partitions updated. This PIP wants to use a
notification mechanism to replace the metadata poll task.
Just the way to implement
Why is this needed when we have notifications on regex sub changes? Aren't
the partition names a well-defined regex?
Joe
On Thu, Feb 23, 2023 at 8:52 PM houxiaoyu wrote:
> Hi Asaf,
> thanks for your reminder.
>
> ## Changing
> I have updated the following changes to make sure the notification a
Hi Asaf,
thanks for your reminder.
## Changing
I have updated the following changes to make sure the notification arrived
successfully:
1. The watch success response `CommandWatchPartitionUpdateSuccess` will
contain all the concerned topics of this watcher
2. The notification `CommandPartitionUpda
How about edge cases?
In Andra's PIP he took into account cases where updates were lost, so he
created a secondary poll. Not saying it's the best situation for your case
of course.
I'm saying that when a broker sends an update CommandPartitionUpdate, how
do you know it arrived successfully? From my
Thanks for your great suggestion Enrico.
I agreed with you. It's more reasonable to add a
`supports_partition_update_watchers` in `FeatureFlags` to detect that the
connected broker supporting this feature , and add a new broker
configuration property `enableNotificationForPartitionUpdate` with d
I support this proposal.
Coping here my comments from GH:
can't we enable this by default in case we detect that the connected
Broker supports it ?
I can't find any reason for not using this mechanism if it is available.
Maybe we can set the default to "true" and allow users to disable it
in case
Hi Pulsar community:
I opened a PIP to discuss "Notifications for partitions update"
### Motivation
Pulsar client will poll brokers at fix time for checking the partitions
update if we publish/subscribe the partitioned topics with
`autoUpdatePartitions` as true. This causes unnecessary load for
16 matches
Mail list logo
