Lari, Il giorno mar 20 giu 2023 alle ore 09:33 Lari Hotari <lhot...@apache.org> ha scritto: > > Dear Pulsar Community Members, > > I would like to initiate a discussion on making the Pulsar Functions > runtime "pluggable". In doing so, we can ensure that the addition of new > runtime types becomes more straightforward. > > This use case will allow us to add support for Pulsar Functions based on > various platforms such as: > > * Pulsar Client Reactive > * Node.js / JavaScript > * WebAssembly (WASM) > * Spring Pulsar & Reactive Spring > > One of the weak points in the current Pulsar Functions runtime is the > default handling of messages individually. Individual message processing > can be slow and inefficient in cases where the main function of the > Pulsar Function (or Sink) is to do backend API calls. > > Although pipelining (processing multiple in-flight messages) is possible > in current Pulsar Functions and Sinks, it often leads to complex and > error-prone solutions, especially when there's a need to combine > key-based ordered processing with retry and backoff implementations. > > The Reactive Pulsar Client provides an inbuilt solution for implementing > pipelining. With its ReactiveMessagePipelineBuilder, we can configure > concurrency levels with key-ordered processing support. This capability > could potentially eliminate the need to use key-shared subscriptions to > scale Pulsar processing. If a reactive Pulsar Function were primarily to > serve as a router for API calls, we could adjust the concurrency level > to hundreds or even thousands, provided the backend could handle the > load. > > With a pluggable Pulsar Functions runtime, we could introduce new > runtime types without the need for implementing each type in the > upstream project. This strategy could likely lead to new opportunities > for innovative ideas and contributions in this field. > > I am interested to know your thoughts on making the Pulsar Functions > runtime pluggable so that we can add new runtime types.
I see that RuntimeFactory [1] is already customizable. What can we do more ? Are you talking about providing alternative implementations for JavaInstanceRunnable [2] ? Thanks or bringing up this topic Enrico [1] https://github.com/apache/pulsar/blob/82237d3684fe506bcb6426b3b23f413422e6e4fb/pulsar-functions/runtime/src/main/java/org/apache/pulsar/functions/runtime/RuntimeFactory.java#L35 [2] https://github.com/apache/pulsar/blob/f7c0b3c49c9ad8c28d0b00aa30d727850eb8bc04/pulsar-functions/instance/src/main/java/org/apache/pulsar/functions/instance/JavaInstanceRunnable.java#L116 > > Best Regards, > > -Lari
