I generally support this proposal,
this is a problem we have in the Proxy and I have seen it on
applications that need to connect to multiple different tenants
and they need different authentication parameters, so they have to
create many PulsarClient instances.
I have a suggestion:
Exposing all the internals is a good idea for very advanced users,
but I believe that we should provide some simpler support.
We should have an API to allow sharing resources among multiple
clients without entering the details.
Interface PulsarClientGroup {
... put here all the sharable things in the current version...
}
PulsarClient client = ....newClient().
.withSharedResources(pulsarClientGroup)
...
I think that having a PulsarClientGroup is a good choice for future evolutions
because the internal thread pools may change: removed/added/change the purpose.
If we require users to deal with all the possible sharable resources
then we have a few risks:
- people can "forget" to share some resources
- upgrading the client may lead to not taking into account some new
"shareable" resources
This is way I believe that we should provide an opaque
Enrico
Il giorno mar 27 dic 2022 alle ore 11:07 PengHui Li
<peng...@apache.org> ha scritto:
>
> Hi all,
>
> As discussed at
> https://lists.apache.org/thread/5obfm17g58n3dnbzyxg57vokgmwyp6hx
> I have created this proposal to support shared thread pool across multiple
> client instances
> Here is the proposal link https://github.com/apache/pulsar/issues/19074
>
> Please help take a look, and look forward to your suggestions.
>
> Thanks,
> Penghui
>
> --------------------------------------------------
>
> ### Motivation
>
> The Pulsar client mainly has three thread pools that cooperate with each
> other to complete the message publishing and consumption of messages.
>
> - IO threads - Used for handling network packets from the broker
> - Internal threads - Used for handling internal tasks such as moving the
> received messages to the internal receiver queue and pulling out the
> message from the receiver queue to return to users. And the Java client is
> optimized by the lock-free principle; each consumer will use a pinned
> internal thread to reduce the lock overhead.
> - External threads - Used by the message listener
>
> All the above thread pools will be created automatically after a Pulsar
> client instance has been created.
>
> But for some cases, users need to create multiple Pulsar client instances
> in a JVM process due to different authentications or others. Each client
> will have exclusive thread pools, which will cause unreasonable thread
> usage, waste memory, and potential performance degradation.
>
> It is not a serious problem for previous releases with the default
> configurations because the thread pool will only have 1 thread by default.
> But it also doesn't make sense that we only have one thread for each thread
> pool. We have discussed this part under this [thread](
> https://lists.apache.org/thread/5obfm17g58n3dnbzyxg57vokgmwyp6hx)
>
> So this proposal will provide a new possibility for users that require
> multiple Pulsar client instances in one JVM process to use the shared
> thread pools across multiple Pulsar client instances.
>
> ### Goal
>
> Provide public API to use the shared thread pool across multiple Pulsar
> client instances in one JVM process
>
> - IO threads
> - Internal threads
> - External threads
>
> BTW, we already have such an ability internally. It was just hidden for
> users. Please take a look at #12037 and #13839 to get more details.
>
> ### API Changes
>
> The following APIs will be introduced to the Java Client when creating a
> Client instance
>
> ```java
> PulsarClient.builder()
> .eventLoopGroup(ioEventLoopGroup)
> .internalExecutorProvider(sharedInternalExecutorProvider)
> .externalExecutorProvider(sharedExternalExecutorProvider)
> .scheduledExecutorProvider(sharedScheduledExecutorProvider)
> ```
