Re: KIP idea: Separate publish request from the subscribe request

2020-08-28 Thread Ming Liu
Hi Guozhang, Yes, the goal is to reduce latency of produce latency from the consumer fetch requests. So the best separate in theory would be: 1. Socket server 1: All produce requests and all follower fetch requests (ACK=ALL) 2. Socket server 2: All other requests (metadata, commit/f

Re: KIP idea: Separate publish request from the subscribe request

2020-08-24 Thread Guozhang Wang
Hello Ming, Thanks for bringing this to the community. Just to clarify your proposal, are you suggesting to use a separate port for fetch requests from all other requests including produce, but also e.g. metadata, commit/fetch-offsets, and other inter-broker requests? If yes that would mean the co

KIP idea: Separate publish request from the subscribe request

2020-08-20 Thread Ming Liu
Hi Kafka community, I like to surface a KIP idea, which is to separate publish request from the subscribe request using different ports. The context: We have some workload with over 5000 subscribers, the latency on publish latency can be as high as 3000 ms second. After investigation, we fou