It seems that your suggestion could be one viable solution. However, my main concern is that users would need to learn yet another API for configuring a producer. An alternative would be to provide a way to pass a function that accepts a ProducerBuilder as an argument for configuration. The underlying framework could handle caching behind the scenes to prevent initializing a new producer for each call. One challenge would be determining how to calculate the cache key for the producer.
The Context.newOutputMessage support for custom producer configuration could be handled separately so that we don't unnecessarily delay the PIP for supporting batching configs for Pulsar Functions & Sources. -Lari On 2025/01/01 06:30:15 Pengcheng Jiang wrote: > Thank for the advice, I will add the example. > > Currently, Context.newOutputMessage uses the same producer configuration as > the function/source's default output, which is converted from the > FunctionDetails.ProducerSpec, so they will use same batching configs . To > enable different topics to use distinct batching configurations, I think > it's better to introduce a new newOutputMessage method in Context that > accepts a producerConfig argument. This would allow users to override all > producer settings, including batching configurations, HDYT? > > Sincerely > Pengcheng Jiang > > Lari Hotari <lhot...@apache.org> 于2025年1月1日周三 03:11写道： > > > This is a very useful addition. Thanks for driving this. > > > > I'd like to request an improvement to the PIP description. The current > > version of the PIP description doesn't provide an example of how this would > > be configured with Pulsar Functions and Pulsar Sources by end users. I'd > > assume that end users mainly use `pulsar-admin functions create` and > > `pulsar-admin sources create`. > > Please add examples of how this would affect the end user facing > > configuration by either using the pulsar-admin command line or by passing a > > yaml config file with `--function-config-file` or `--source-config-file` > > command line parameter. > > Currently the Pulsar Functions documentation is not very great in > > explaining how different configuration options are used by end users. The > > current PIP document explains the internal interface changes, but the > > actual end user facing API is the pulsar-admin command line for creating > > Pulsar Functions and Pulsar IO Sources. > > > > I have a question about handling use cases where a Pulsar Function > > implemented in Java produces messages to multiple destination topics using > > Context.newOutputMessage. Would this also use the configured batching > > configuration? I wonder if there are needs to have different batching > > configurations for different destination topics? > > > > -Lari > > > > On 2024/12/30 08:43:15 Pengcheng Jiang wrote: > > > Hello community, > > > > > > I opened the PIP-401 <https://github.com/apache/pulsar/pull/23793> to > > make > > > users able to customize the batching settings for Pulsar Functions & > > > Sources, leave any comments you want, many thanks. > > > > > > Refers: > > > > > > - https://github.com/apache/pulsar/pull/23793 > > > > > > Sincerely, > > > Pengcheng Jiang > > > > > >
