Re: [Discussion] Allowing configure if function consumer should skip to latest

2023-03-13 Thread PengHui Li
Got it. Thanks for the explanation. LGTM Penghui On Thu, Mar 9, 2023 at 3:06 AM Neng Lu wrote: > Hi everyone, > > This discussion has been one week old. If there's no objection or > concerns, I'll move forward and close the discussion with the conclusion > that we are good with the proposed ch

Re: [Discussion] Allowing configure if function consumer should skip to latest

2023-03-08 Thread Neng Lu
Hi everyone, This discussion has been one week old. If there's no objection or concerns, I'll move forward and close the discussion with the conclusion that we are good with the proposed change. This will result in the PR merge (although it was already merged, so the merged change won't be rev

Re: [Discussion] Allowing configure if function consumer should skip to latest

2023-03-06 Thread Neng Lu
Hi Penghui, Thanks for your question. One case is failure recovery for a windowing function. A windowing function will ack message until its window is emitted. If the window function fails due to issues such as OOM and restarts, it has a massive backlog to catch up. And the function will never

Re: [Discussion] Allowing configure if function consumer should skip to latest

2023-03-02 Thread PengHui Li
Hi Neng, Thanks for raising up the discussion > In certain failure cases, the function needs to skip all the content between the last successfully acked message and the latest message in the topic in order to skip the huge backlog and quick recovery. Do you have some real cases that can help us

[Discussion] Allowing configure if function consumer should skip to latest

2023-02-28 Thread Neng Lu
Hello Community, In this [PR](https://github.com/apache/pulsar/pull/17214), we changed the function protobuf by adding one more field `bool skipToLatest = 14;`. The change itself is minimum and self-contained for function internal usage. Given the new community guideline that protobuf change sho