+1 to Enrico's suggestion about the API to allow sharing resources among multiple clients.
I think this could be a good way to hide the implementation details about event loop groups and prevent exposing Netty classes in the API. btw. The timer instance that would need to be shared besides the event loop instances. (the shared timer feature was added to PulsarClientImpl in https://github.com/apache/pulsar/pull/9802). This could also be handled as an internal detail in the PulsarClientGroup shared resources solution. -Lari On 2022/12/27 11:11:34 Enrico Olivelli wrote: > I generally support this proposal, > this is a problem we have in the Proxy and I have seen it on > applications that need to connect to multiple different tenants > and they need different authentication parameters, so they have to > create many PulsarClient instances. > > I have a suggestion: > > Exposing all the internals is a good idea for very advanced users, > but I believe that we should provide some simpler support. > > We should have an API to allow sharing resources among multiple > clients without entering the details. > > Interface PulsarClientGroup { > ... put here all the sharable things in the current version... > } > > PulsarClient client = ....newClient(). > .withSharedResources(pulsarClientGroup) > ... > > > I think that having a PulsarClientGroup is a good choice for future evolutions > because the internal thread pools may change: removed/added/change the > purpose. > > If we require users to deal with all the possible sharable resources > then we have a few risks: > - people can "forget" to share some resources > - upgrading the client may lead to not taking into account some new > "shareable" resources > > This is way I believe that we should provide an opaque > > Enrico > > Il giorno mar 27 dic 2022 alle ore 11:07 PengHui Li > <peng...@apache.org> ha scritto: > > > > Hi all, > > > > As discussed at > > https://lists.apache.org/thread/5obfm17g58n3dnbzyxg57vokgmwyp6hx > > I have created this proposal to support shared thread pool across multiple > > client instances > > Here is the proposal link https://github.com/apache/pulsar/issues/19074 > > > > Please help take a look, and look forward to your suggestions. > > > > Thanks, > > Penghui > > > > -------------------------------------------------- > > > > ### Motivation > > > > The Pulsar client mainly has three thread pools that cooperate with each > > other to complete the message publishing and consumption of messages. > > > > - IO threads - Used for handling network packets from the broker > > - Internal threads - Used for handling internal tasks such as moving the > > received messages to the internal receiver queue and pulling out the > > message from the receiver queue to return to users. And the Java client is > > optimized by the lock-free principle; each consumer will use a pinned > > internal thread to reduce the lock overhead. > > - External threads - Used by the message listener > > > > All the above thread pools will be created automatically after a Pulsar > > client instance has been created. > > > > But for some cases, users need to create multiple Pulsar client instances > > in a JVM process due to different authentications or others. Each client > > will have exclusive thread pools, which will cause unreasonable thread > > usage, waste memory, and potential performance degradation. > > > > It is not a serious problem for previous releases with the default > > configurations because the thread pool will only have 1 thread by default. > > But it also doesn't make sense that we only have one thread for each thread > > pool. We have discussed this part under this [thread]( > > https://lists.apache.org/thread/5obfm17g58n3dnbzyxg57vokgmwyp6hx) > > > > So this proposal will provide a new possibility for users that require > > multiple Pulsar client instances in one JVM process to use the shared > > thread pools across multiple Pulsar client instances. > > > > ### Goal > > > > Provide public API to use the shared thread pool across multiple Pulsar > > client instances in one JVM process > > > > - IO threads > > - Internal threads > > - External threads > > > > BTW, we already have such an ability internally. It was just hidden for > > users. Please take a look at #12037 and #13839 to get more details. > > > > ### API Changes > > > > The following APIs will be introduced to the Java Client when creating a > > Client instance > > > > ```java > > PulsarClient.builder() > > .eventLoopGroup(ioEventLoopGroup) > > .internalExecutorProvider(sharedInternalExecutorProvider) > > .externalExecutorProvider(sharedExternalExecutorProvider) > > .scheduledExecutorProvider(sharedScheduledExecutorProvider) > > ``` >
