Lari, would it be possible to explain in more detail the paint points you're describing?
You say processing messages individually is slow; hence, processing them in batches is better. I guess it's especially useful if you need to group a batch based on a key. What I don't understand is how the framework today limits you from using something like a reactive client which does the batching inside. On Tue, Jun 20, 2023 at 10:33 AM Lari Hotari <lhot...@apache.org> wrote: > Dear Pulsar Community Members, > > I would like to initiate a discussion on making the Pulsar Functions > runtime "pluggable". In doing so, we can ensure that the addition of new > runtime types becomes more straightforward. > > This use case will allow us to add support for Pulsar Functions based on > various platforms such as: > > * Pulsar Client Reactive > * Node.js / JavaScript > * WebAssembly (WASM) > * Spring Pulsar & Reactive Spring > > One of the weak points in the current Pulsar Functions runtime is the > default handling of messages individually. Individual message processing > can be slow and inefficient in cases where the main function of the > Pulsar Function (or Sink) is to do backend API calls. > > Although pipelining (processing multiple in-flight messages) is possible > in current Pulsar Functions and Sinks, it often leads to complex and > error-prone solutions, especially when there's a need to combine > key-based ordered processing with retry and backoff implementations. > > The Reactive Pulsar Client provides an inbuilt solution for implementing > pipelining. With its ReactiveMessagePipelineBuilder, we can configure > concurrency levels with key-ordered processing support. This capability > could potentially eliminate the need to use key-shared subscriptions to > scale Pulsar processing. If a reactive Pulsar Function were primarily to > serve as a router for API calls, we could adjust the concurrency level > to hundreds or even thousands, provided the backend could handle the > load. > > With a pluggable Pulsar Functions runtime, we could introduce new > runtime types without the need for implementing each type in the > upstream project. This strategy could likely lead to new opportunities > for innovative ideas and contributions in this field. > > I am interested to know your thoughts on making the Pulsar Functions > runtime pluggable so that we can add new runtime types. > > Best Regards, > > -Lari >
