Dear Pulsar Community Members, I would like to initiate a discussion on making the Pulsar Functions runtime "pluggable". In doing so, we can ensure that the addition of new runtime types becomes more straightforward.
This use case will allow us to add support for Pulsar Functions based on various platforms such as: * Pulsar Client Reactive * Node.js / JavaScript * WebAssembly (WASM) * Spring Pulsar & Reactive Spring One of the weak points in the current Pulsar Functions runtime is the default handling of messages individually. Individual message processing can be slow and inefficient in cases where the main function of the Pulsar Function (or Sink) is to do backend API calls. Although pipelining (processing multiple in-flight messages) is possible in current Pulsar Functions and Sinks, it often leads to complex and error-prone solutions, especially when there's a need to combine key-based ordered processing with retry and backoff implementations. The Reactive Pulsar Client provides an inbuilt solution for implementing pipelining. With its ReactiveMessagePipelineBuilder, we can configure concurrency levels with key-ordered processing support. This capability could potentially eliminate the need to use key-shared subscriptions to scale Pulsar processing. If a reactive Pulsar Function were primarily to serve as a router for API calls, we could adjust the concurrency level to hundreds or even thousands, provided the backend could handle the load. With a pluggable Pulsar Functions runtime, we could introduce new runtime types without the need for implementing each type in the upstream project. This strategy could likely lead to new opportunities for innovative ideas and contributions in this field. I am interested to know your thoughts on making the Pulsar Functions runtime pluggable so that we can add new runtime types. Best Regards, -Lari
