Re: PIP-187 Add API to analyse a subscription backlog and provide a accurate value

2022-07-24 Thread Asaf Mesika
On Thu, Jul 21, 2022 at 10:36 AM Enrico Olivelli wrote: > Il giorno gio 21 lug 2022 alle ore 06:25 PengHui Li > ha scritto: > > > > > What if the topic owner creates an internal subscription, consumes the > > messages, and updates a count per filter. > > > > I agree with this approach. If we nee

Re: PIP-187 Add API to analyse a subscription backlog and provide a accurate value

2022-07-22 Thread Enrico Olivelli
Il giorno ven 22 lug 2022 alle ore 11:16 PengHui Li ha scritto: > > > As you can't know when you will need this value, you are going to > waste resources in order to calculate > something that you won't use. > > > This API to analyze a subscription is not meant to be used often, but > only to debu

Re: PIP-187 Add API to analyse a subscription backlog and provide a accurate value

2022-07-22 Thread PengHui Li
> As you can't know when you will need this value, you are going to waste resources in order to calculate something that you won't use. > This API to analyze a subscription is not meant to be used often, but only to debug (manually) problems. Oh, I got your point. One concern is if we are using

Re: PIP-187 Add API to analyse a subscription backlog and provide a accurate value

2022-07-21 Thread Enrico Olivelli
Il giorno gio 21 lug 2022 alle ore 06:25 PengHui Li ha scritto: > > > What if the topic owner creates an internal subscription, consumes the > messages, and updates a count per filter. > > I agree with this approach. If we need to scan all the backlogs to > calculate the > accurate backlogs for ea

Re: PIP-187 Add API to analyse a subscription backlog and provide a accurate value

2022-07-20 Thread PengHui Li
> What if the topic owner creates an internal subscription, consumes the messages, and updates a count per filter. I agree with this approach. If we need to scan all the backlogs to calculate the accurate backlogs for each operation, it's so expensive and difficult to apply to the production envir

Re: PIP-187 Add API to analyse a subscription backlog and provide a accurate value

2022-07-20 Thread Asaf Mesika
On Wed, Jul 20, 2022 at 5:46 PM Enrico Olivelli wrote: > Asaf, > > Il giorno mer 20 lug 2022 alle ore 15:40 Asaf Mesika > ha scritto: > > > > I'm not sure I understand the context exactly: > > > > You say today we can only know the number of entries, hence we'll have a > > wrong number of backlo

Re: PIP-187 Add API to analyse a subscription backlog and provide a accurate value

2022-07-20 Thread Enrico Olivelli
Asaf, Il giorno mer 20 lug 2022 alle ore 15:40 Asaf Mesika ha scritto: > > I'm not sure I understand the context exactly: > > You say today we can only know the number of entries, hence we'll have a > wrong number of backlog for subscription since: > 1. One entry contains multiple messages (batch

Re: PIP-187 Add API to analyse a subscription backlog and provide a accurate value

2022-07-20 Thread Asaf Mesika
I'm not sure I understand the context exactly: You say today we can only know the number of entries, hence we'll have a wrong number of backlog for subscription since: 1. One entry contains multiple messages (batch message) 2. Subscription may contain a filter, which requires you to read the entir

Re: PIP-187 Add API to analyse a subscription backlog and provide a accurate value

2022-07-19 Thread Enrico Olivelli
Thank you all Let me start a official VOTE thread Enrico Il Mar 19 Lug 2022, 15:10 mattison chao ha scritto: > +1, > This feature is very useful for users. > > Best, > Mattison > > On Mon, 18 Jul 2022 at 18:11, Nicolò Boschi wrote: > > > +1, good work, I think it's very valuable for users for

Re: PIP-187 Add API to analyse a subscription backlog and provide a accurate value

2022-07-19 Thread mattison chao
+1, This feature is very useful for users. Best, Mattison On Mon, 18 Jul 2022 at 18:11, Nicolò Boschi wrote: > +1, good work, I think it's very valuable for users for monitoring > purposes. > > Nicolò Boschi > > > Il giorno lun 18 lug 2022 alle ore 11:47 Lothruin Mirwen < > lothruin.mir...@gmai

Re: PIP-187 Add API to analyse a subscription backlog and provide a accurate value

2022-07-18 Thread Nicolò Boschi
+1, good work, I think it's very valuable for users for monitoring purposes. Nicolò Boschi Il giorno lun 18 lug 2022 alle ore 11:47 Lothruin Mirwen < lothruin.mir...@gmail.com> ha scritto: > This should be really helpful to monitor real backlog in production > environments. > Currently we are s

Re: PIP-187 Add API to analyse a subscription backlog and provide a accurate value

2022-07-18 Thread Lothruin Mirwen
This should be really helpful to monitor real backlog in production environments. Currently we are struck at "entries" count but doesn't describe well how much real message lag we have. +1 excited to see it work on our production environments Diego Salvi Il giorno lun 18 lug 2022 alle ore 11:18

Re: PIP-187 Add API to analyse a subscription backlog and provide a accurate value

2022-07-18 Thread Enrico Olivelli
Any comments ? FYI I have updated the PIP after addressing some feedback on the PR: - no more need to create a dummy Dispatcher - now we are reading entries in batches I would like to see this in 2.11 please Enrico Il giorno gio 14 lug 2022 alle ore 09:34 Enrico Olivelli ha scritto: > > Hello,

PIP-187 Add API to analyse a subscription backlog and provide a accurate value

2022-07-14 Thread Enrico Olivelli
Hello, this is a PIP to implement a tool to analyse the subscription backlog Link: https://github.com/apache/pulsar/issues/16597 Prototype: https://github.com/apache/pulsar/pull/16545 Below you can find the proposal (I will amend the GH issue while we discuss, as usual) Enrico Motivation Curre