Hello Ming, Thanks for bringing this to the community. Just to clarify your proposal, are you suggesting to use a separate port for fetch requests from all other requests including produce, but also e.g. metadata, commit/fetch-offsets, and other inter-broker requests? If yes that would mean the consumer metadata request should then be handled differently from producer metadata request so that they can return different port values, and also on the server side we'd need two socket servers as well. Maybe we can get a more concrete design on how this would work end-to-end for others to review.
Also I'm wondering if you have tried increasing the num.network.threads and see if that helps with the large fanout multiplexing issue other than separating them to different ports? Guozhang On Thu, Aug 20, 2020 at 7:03 PM Ming Liu <minga...@gmail.com> wrote: > Hi Kafka community, > I like to surface a KIP idea, which is to separate publish request from > the subscribe request using different ports. > > The context: We have some workload with over 5000 subscribers, the > latency on publish latency can be as high as 3000 ms second. After > investigation, we found the reason is mainly because there are too many > connections on socketserver and the multiplexing slows down the publish > latency. > > The proposal is somewhat similar to KIP-291: Separating controller > connections and requests from the data plane > > I like to check with experts here whether this is a viable idea to > continue pursuing or not? > > Thanks! > Ming > -- -- Guozhang
