> pass a function that accepts a ProducerBuilder as an argument for configuration This sounds more reasonable
And I agree that this should be handled in a separate PIP. Sincerely Pengcheng Jiang Lari Hotari <lhot...@apache.org> 于2025年1月3日周五 03:45写道： > It seems that your suggestion could be one viable solution. However, my > main concern is that users would need to learn yet another API for > configuring a producer. An alternative would be to provide a way to pass a > function that accepts a ProducerBuilder as an argument for configuration. > The underlying framework could handle caching behind the scenes to prevent > initializing a new producer for each call. One challenge would be > determining how to calculate the cache key for the producer. > > The Context.newOutputMessage support for custom producer configuration > could be handled separately so that we don't unnecessarily delay the PIP > for supporting batching configs for Pulsar Functions & Sources. > > -Lari > > On 2025/01/01 06:30:15 Pengcheng Jiang wrote: > > Thank for the advice, I will add the example. > > > > Currently, Context.newOutputMessage uses the same producer configuration > as > > the function/source's default output, which is converted from the > > FunctionDetails.ProducerSpec, so they will use same batching configs . To > > enable different topics to use distinct batching configurations, I think > > it's better to introduce a new newOutputMessage method in Context that > > accepts a producerConfig argument. This would allow users to override all > > producer settings, including batching configurations, HDYT? > > > > Sincerely > > Pengcheng Jiang > > > > Lari Hotari <lhot...@apache.org> 于2025年1月1日周三 03:11写道： > > > > > This is a very useful addition. Thanks for driving this. > > > > > > I'd like to request an improvement to the PIP description. The current > > > version of the PIP description doesn't provide an example of how this > would > > > be configured with Pulsar Functions and Pulsar Sources by end users. > I'd > > > assume that end users mainly use `pulsar-admin functions create` and > > > `pulsar-admin sources create`. > > > Please add examples of how this would affect the end user facing > > > configuration by either using the pulsar-admin command line or by > passing a > > > yaml config file with `--function-config-file` or > `--source-config-file` > > > command line parameter. > > > Currently the Pulsar Functions documentation is not very great in > > > explaining how different configuration options are used by end users. > The > > > current PIP document explains the internal interface changes, but the > > > actual end user facing API is the pulsar-admin command line for > creating > > > Pulsar Functions and Pulsar IO Sources. > > > > > > I have a question about handling use cases where a Pulsar Function > > > implemented in Java produces messages to multiple destination topics > using > > > Context.newOutputMessage. Would this also use the configured batching > > > configuration? I wonder if there are needs to have different batching > > > configurations for different destination topics? > > > > > > -Lari > > > > > > On 2024/12/30 08:43:15 Pengcheng Jiang wrote: > > > > Hello community, > > > > > > > > I opened the PIP-401 <https://github.com/apache/pulsar/pull/23793> > to > > > make > > > > users able to customize the batching settings for Pulsar Functions & > > > > Sources, leave any comments you want, many thanks. > > > > > > > > Refers: > > > > > > > > - https://github.com/apache/pulsar/pull/23793 > > > > > > > > Sincerely, > > > > Pengcheng Jiang > > > > > > > > > >
