I have created a proposal for sharing the thread pool across client instances. Here is the PIP discussion thread
https://lists.apache.org/thread/5jw06hqlmwnrgvbn9lfom1vkwhwqwwd4 Thanks for all your suggestions -- Penghui On Mon, Dec 26, 2022 at 9:39 PM PengHui Li <peng...@apache.org> wrote: > > I think that we can change this default but we should provide an official > API to share the threadpools between the clients. > > That sounds good to me. > Since it will introduce a new API to the client side. > I will create a new proposal for the API changes. > > Thanks, > Penghui > > On Mon, Dec 26, 2022 at 8:21 PM Enrico Olivelli <eolive...@gmail.com> > wrote: > >> I am sorry, I am late on this train. >> >> Unfortunately if you have to connect using different authentication >> parameters you need to open many PulsarClients. >> >> I think that we can change this default but we should provide an official >> API to share the threadpools between the clients. >> >> There is no need to explicitly espose all the threadpools, that is >> difficult for users to deal with. >> We should have an API to allow sharing resources among multiple clients. >> >> Interface PulsarClientGroup .... >> >> >> PulsarClient client = ....newClient(). >> .withGroup(pulsarClientGroup) >> .... >> >> >> >> Enrico >> >> Il Lun 26 Dic 2022, 12:44 Ran Gao <r...@apache.org> ha scritto: >> >> > It seems this change will not affect the consumer, because the default >> > executor is a single-thread executor service, increasing the listener >> > threads number will allocate different single-thread executor services >> to >> > different consumers. >> > >> > On 2022/12/21 09:01:39 Yunze Xu wrote: >> > > I have a concern about the message ordering. If we have more than 1 >> > > listener thread by default, could messages from the same topic be >> > > passed to different listener threads? >> > > >> > > Thanks, >> > > Yunze >> > > >> > > On Wed, Dec 21, 2022 at 11:12 AM 丛搏 <bog...@apache.org> wrote: >> > > > >> > > > +1 >> > > > Our default configuration is best for most users. Multiple clients >> are >> > > > a few cases. >> > > > >> > > > Thanks, >> > > > Bo >> > > > >> > > > houxiaoyu <anonhx...@gmail.com> 于2022年12月20日周二 16:02写道： >> > > > > >> > > > > +1 >> > > > > >> > > > > This change might bring thread number increment in case users >> create >> > many >> > > > > clients, but too many pulsar clients run in one machine is not a >> > good use >> > > > > case I think, so this change looks good to me. >> > > > > >> > > > > Thanks, >> > > > > Xiaoyu Hou >> > > > > >> > > > > <mattisonc...@gmail.com> 于2022年12月20日周二 12:25写道： >> > > > > >> > > > > > +1 >> > > > > > >> > > > > > My concern is whether this change will affect some users who are >> > creating >> > > > > > many clients. I think we can wait for other users to confirm it. >> > (If this >> > > > > > will be affected, maybe we can give it a max_io_thread_num and >> > then expand >> > > > > > the size from 1 to max_io_thread_num when adding a new consumer >> or >> > producer) >> > > > > > >> > > > > > >> > > > > > Best, >> > > > > > Mattison >> > > > > > On Dec 20, 2022, 11:17 +0800, PengHui Li <peng...@apache.org>, >> > wrote: >> > > > > > > Hi all, >> > > > > > > >> > > > > > > I noticed the Java Client (I haven't checked other clients) >> uses >> > 1 IO >> > > > > > > thread and 1 listener >> > > > > > > thread by default. It will require users to update the thread >> > > > > > configuration >> > > > > > > if they have >> > > > > > > multiple cores and desired high throughput. >> > > > > > > >> > > > > > > Here is the example that we change to 16 IO threads in >> > > > > > > openmessaging benchmark >> > > > > > > >> > > > > > >> > >> https://github.com/openmessaging/benchmark/blob/master/driver-pulsar/pulsar.yaml#L22 >> > > > > > > >> > > > > > > We can apply the configuration of the threads based on the CPU >> > cores. So >> > > > > > > that for the >> > > > > > > most common cases, users don't need to touch the thread >> > configuration. >> > > > > > > >> > > > > > > ``` >> > > > > > > private int numIoThreads = >> > Runtime.getRuntime().availableProcessors(); >> > > > > > > private int numListenerThreads = >> > > > > > Runtime.getRuntime().availableProcessors(); >> > > > > > > ``` >> > > > > > > >> > > > > > > WDYT? >> > > > > > > >> > > > > > > Thanks, >> > > > > > > Penghui >> > > > > > >> > > >> > >> >
