Re: [DISCUSS] [Proposal] PIP 84: Cluster-Wide and Function-Specific Producer Defaults

All the tests in my personal Github CI passed except for ones in the Flaky group, so I created an Apache Pulsar PR: https://github.com/apache/pulsar/pull/10154 (Again, this is for Part 1 of the changes.) I think that code is ready for review. Devin G. Bost On Tue, Apr 6, 2021 at 12:37 AM Devin B

Re: [DISCUSS] [Proposal] PIP 84: Cluster-Wide and Function-Specific Producer Defaults

I think I was able to resolve that issue. Fortunately, only FunctionResultRouter had the singleton behavior, so resolving that was straightforward. The current PR for part 1 is here (in my personal CI): https://github.com/devinbost/pulsar/pull/5/files Once the tests look good, I'll submit a PR to A

Re: [DISCUSS] [Proposal] PIP 84: Cluster-Wide and Function-Specific Producer Defaults

It appears that the reason the router classes are singletons is to ensure the clock behavior is consistent across producers. I wonder if we might be able to avoid creating a breaking change by refactoring out the clock into a singleton that could be shared across the routers, which we could then ma

Re: [DISCUSS] [Proposal] PIP 84: Cluster-Wide and Function-Specific Producer Defaults

Jerry, Thanks for the information and for the links! Thanks also for the suggestion to split the PR into two parts. That will make it easier to get some of this work completed in time for the 2.8.0 release. While working on the first part, I discovered something that needs community attention. I

Re: [DISCUSS] [Proposal] PIP 84: Cluster-Wide and Function-Specific Producer Defaults

> > The interceptor would be loaded at function registration. The interceptor classes should be loaded at broker/worker startup time. In order for an > interceptor to be picked up, a jar would need to be compiled with the class > that implements the interface, and the jar would need to be deplo

Re: [DISCUSS] [Proposal] PIP 84: Cluster-Wide and Function-Specific Producer Defaults

Here's my understanding of what I'd need to do to implement the interceptor approach. The interceptor would be loaded at function registration. In order for an interceptor to be picked up, a jar would need to be compiled with the class that implements the interface, and the jar would need to be dep

Re: [DISCUSS] [Proposal] PIP 84: Cluster-Wide and Function-Specific Producer Defaults

If we use the interceptor approach, does that mean a cluster-admin would need to write custom code in order to override the function producer settings? I worry about adding additional burden to cluster administration. -- Devin G. Bost On Sat, Apr 3, 2021, 8:26 AM Devin Bost wrote: > Hi Rui, > >

Re: [DISCUSS] [Proposal] PIP 84: Cluster-Wide and Function-Specific Producer Defaults

Hi Rui, Thanks for the feedback. My understanding about the interceptor is that it's not something that would be touched by users but would only be modified by cluster admins. So, I think we'd still need to include the parameters the parameters on the ProducerConfig. I already have the other appro

Re: [DISCUSS] [Proposal] PIP 84: Cluster-Wide and Function-Specific Producer Defaults

Hi Devin, Great proposal for giving function such useful feature, and thanks for your detailed writing. After going through the history of this discussion, I would like to have +1 with Jerry’s suggestion. With the interceptor, the admins and users may have the maximum flexibility to customize t

Re: [DISCUSS] [Proposal] PIP 84: Cluster-Wide and Function-Specific Producer Defaults

What would be some of the additional benefits of using the interceptor approach? -- Devin G. Bost On Thu, Apr 1, 2021, 6:03 PM Jerry Peng wrote: > Devin, > > Interceptors are located on the server side (broker/worker). Cluster > admins would create the plugins based on the interfaces I descr

Re: [DISCUSS] [Proposal] PIP 84: Cluster-Wide and Function-Specific Producer Defaults

Devin, Interceptors are located on the server side (broker/worker). Cluster admins would create the plugins based on the interfaces I described and install them on the server side. Regular function developers will not implement or interact directly with the interceptor. > Would it be better t

Re: [DISCUSS] [Proposal] PIP 84: Cluster-Wide and Function-Specific Producer Defaults

Would it be better to just ignore WorkerConfig defaults during function update? We could still update the function producer behavior through the producerConfig passed as a REST parameter, but we'd avoid the edge case I mentioned. Devin G. Bost On Thu, Apr 1, 2021 at 5:42 PM Devin Bost wrote: >

Re: [DISCUSS] [Proposal] PIP 84: Cluster-Wide and Function-Specific Producer Defaults

I just remembered an edge case that motivated me to create that additional flow. In my PR, if any producerConfig values are null at registration or update, we get those values from WorkerConfig instead. However, when a user runs an update (on a function) for the first time after upgrading to the ve

Re: [DISCUSS] [Proposal] PIP 84: Cluster-Wide and Function-Specific Producer Defaults

Hi Jerry, I guess we could just ignore the cluster function defaults when functions are updated. Maybe I was overthinking the update process. Regarding the interceptor idea, you're saying that we'd modify the producerConfig properties or class definition at runtime? I love the power of using an

Re: [DISCUSS] [Proposal] PIP 84: Cluster-Wide and Function-Specific Producer Defaults

Hi Devin, I understand the usefulness of giving cluster admins the ability to specify some defaults for functions at a cluster level. I think the right time to apply those defaults is during function registration time. I am not sure I understand the usefulness of what you are proposing we do dur

[DISCUSS] [Proposal] PIP 84: Cluster-Wide and Function-Specific Producer Defaults

*Cluster-Wide and Function-Specific Producer Defaults* * - Status: Proposal- Author: Devin Bost (with guidance from Jerry Peng)- Pull Request: https://github.com/apache/pulsar/pull/9987 - Mailing List discussion: - Release: 2.8.0* *Motivation* Puls