[
https://issues.apache.org/jira/browse/KAFKA-1912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14367331#comment-14367331
]
Andrii Biletskyi commented on KAFKA-1912:
-----------------------------------------
[~jkreps] thanks for adding the code.
A few questions / comments:
1. Regarding RequestRerouter shim: If we store and re-route unsent messages in
the separate thread, outside the main io thread pool, wouldn't it "blur"
num.io.threads setting? With this facility in place there will be a
RequestHandlerPool bounded by num.io.threads and a message queue with a worker
managing not-routed messages and unsent responses to clients.
2. On a message queue: if we "bombard" some broker with requests which require
re-routing to controller wouldn't it be strange when client receives
TimeoutException because _broker_ wasn't able to process that but not the
controller who is the real request "executor".
3. Is it possible for NetworkClient (on the client side) to know whether target
broker's message queue is full and thus broker may delay re-routing? Consider
there is a high priority admin task (like alter topic's config) but the target
broker is busy with produce / fetch requests.
4. This was mentioned by [~guozhang] in the mailing list. Let me add this
question too:
"the broker instance which carries the current controller is agnostic of its
existence,
and use KafkaApis to handle general Kafka requests. Having all admin
requests redirected to the controller instance will force the broker to be
aware of its carried controller, and access its internal data for handling
these requests. Plus with the number of clients out of Kafka's control,
this may easily cause the controller to be a hot spot in terms of request
load."
> Create a simple request re-routing facility
> -------------------------------------------
>
> Key: KAFKA-1912
> URL: https://issues.apache.org/jira/browse/KAFKA-1912
> Project: Kafka
> Issue Type: Improvement
> Reporter: Jay Kreps
> Fix For: 0.8.3
>
>
> We are accumulating a lot of requests that have to be directed to the correct
> server. This makes sense for high volume produce or fetch requests. But it is
> silly to put the extra burden on the client for the many miscellaneous
> requests such as fetching or committing offsets and so on.
> This adds a ton of practical complexity to the clients with little or no
> payoff in performance.
> We should add a generic request-type agnostic re-routing facility on the
> server. This would allow any server to accept a request and forward it to the
> correct destination, proxying the response back to the user. Naturally it
> needs to do this without blocking the thread.
> The result is that a client implementation can choose to be optimally
> efficient and manage a local cache of cluster state and attempt to always
> direct its requests to the proper server OR it can choose simplicity and just
> send things all to a single host and let that host figure out where to
> forward it.
> I actually think we should implement this more or less across the board, but
> some requests such as produce and fetch require more logic to proxy since
> they have to be scattered out to multiple servers and gathered back to create
> the response. So these could be done in a second phase.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
