[
https://issues.apache.org/jira/browse/KAFKA-4453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15702790#comment-15702790
]
ASF GitHub Bot commented on KAFKA-4453:
---------------------------------------
GitHub user onurkaraman opened a pull request:
https://github.com/apache/kafka/pull/2181
KAFKA-4453: add request prioritization
Today all requests (client requests, broker requests, controller requests)
to a broker are put into the same queue. They all have the same priority. So a
backlog of requests ahead of the controller request will delay the processing
of controller requests. This causes requests infront of the controller request
to get processed based on stale state.
Side effects may include giving clients stale metadata, rejecting
ProduceRequests and FetchRequests, and data loss (for some unofficial
definition of data loss in terms of messages beyond the high watermark).
We'd like to minimize the number of requests processed based on stale
state. With request prioritization, controller requests get processed before
regular queued up requests, so requests can get processed with up-to-date state.
Request prioritization can happen at the network layer with the
RequestChannel. The RequestChannel can categorize the request as regular or
prioritized based on the request id. If the incoming request id matches that of
UpdateMetadataRequest, LeaderAndIsrRequest, and StopReplicaRequest, the request
can get prioritized.
One solution is to simply add a prioritized request queue to supplement the
existing request queue in the RequestChannel and add request
prioritization-aware logic to both the sendRequest and receiveRequest
operations of RequestChannel. sendRequest puts the request into the respective
queue based on whether the request is prioritized or not. receiveRequest can
optimistically check the prioritized request queue and otherwise fallback to
the regular request queue. One subtlety here is whether to do a timed poll on
just the regular request queue or on both the prioritized request queue and
regular request queue sequentially. Only applying the timed poll to the regular
request queue punishes a prioritized request that arrives before a regular
request but moments after the prioritized request check. Applying the timed
poll to both queues sequentially results in a guaranteed latency increase on a
regular request.
An alternative is to replace RequestChannel’s existing request queue with a
prioritization-aware blocking queue. This approach avoids the earlier stated
subtlety by allowing the timed poll to apply to either prioritized or regular
requests in low-throughput scenarios while still allowing queued prioritized
requests to go ahead of queued regular requests.
This patch goes with the latter approach to avoid punishing late arriving
prioritized requests.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/onurkaraman/kafka KAFKA-4453
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/kafka/pull/2181.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2181
----
commit e7bff71dce307c3ad5e89f8627c63d546878e39e
Author: Onur Karaman <okara...@linkedin.com>
Date: 2016-11-28T18:54:35Z
KAFKA-4453: add request prioritization
Today all requests (client requests, broker requests, controller requests)
to a broker are put into the same queue. They all have the same priority. So a
backlog of requests ahead of the controller request will delay the processing
of controller requests. This causes requests infront of the controller request
to get processed based on stale state.
Side effects may include giving clients stale metadata, rejecting
ProduceRequests and FetchRequests, and data loss (for some unofficial
definition of data loss in terms of messages beyond the high watermark).
We'd like to minimize the number of requests processed based on stale
state. With request prioritization, controller requests get processed before
regular queued up requests, so requests can get processed with up-to-date state.
Request prioritization can happen at the network layer with the
RequestChannel. The RequestChannel can categorize the request as regular or
prioritized based on the request id. If the incoming request id matches that of
UpdateMetadataRequest, LeaderAndIsrRequest, and StopReplicaRequest, the request
can get prioritized.
One solution is to simply add a prioritized request queue to supplement the
existing request queue in the RequestChannel and add request
prioritization-aware logic to both the sendRequest and receiveRequest
operations of RequestChannel. sendRequest puts the request into the respective
queue based on whether the request is prioritized or not. receiveRequest can
optimistically check the prioritized request queue and otherwise fallback to
the regular request queue. One subtlety here is whether to do a timed poll on
just the regular request queue or on both the prioritized request queue and
regular request queue sequentially. Only applying the timed poll to the regular
request queue punishes a prioritized request that arrives before a regular
request but moments after the prioritized request check. Applying the timed
poll to both queues sequentially results in a guaranteed latency increase on a
regular request.
An alternative is to replace RequestChannel’s existing request queue with a
prioritization-aware blocking queue. This approach avoids the earlier stated
subtlety by allowing the timed poll to apply to either prioritized or regular
requests in low-throughput scenarios while still allowing queued prioritized
requests to go ahead of queued regular requests.
This patch goes with the latter approach to avoid punishing late arriving
prioritized requests.
----
> add request prioritization
> --------------------------
>
> Key: KAFKA-4453
> URL: https://issues.apache.org/jira/browse/KAFKA-4453
> Project: Kafka
> Issue Type: Bug
> Reporter: Onur Karaman
> Assignee: Onur Karaman
>
> Today all requests (client requests, broker requests, controller requests) to
> a broker are put into the same queue. They all have the same priority. So a
> backlog of requests ahead of the controller request will delay the processing
> of controller requests. This causes requests infront of the controller
> request to get processed based on stale state.
> Side effects may include giving clients stale metadata\[1\], rejecting
> ProduceRequests and FetchRequests, and data loss (for some unofficial\[2\]
> definition of data loss in terms of messages beyond the high watermark)\[3\].
> We'd like to minimize the number of requests processed based on stale state.
> With request prioritization, controller requests get processed before regular
> queued up requests, so requests can get processed with up-to-date state.
> \[1\] Say a client's MetadataRequest is sitting infront of a controller's
> UpdateMetadataRequest on a given broker's request queue. Suppose the
> MetadataRequest is for a topic whose partitions have recently undergone
> leadership changes and that these leadership changes are being broadcasted
> from the controller in the later UpdateMetadataRequest. Today the broker
> processes the MetadataRequest before processing the UpdateMetadataRequest,
> meaning the metadata returned to the client will be stale. The client will
> waste a roundtrip sending requests to the stale partition leader, get a
> NOT_LEADER_FOR_PARTITION error, and will have to start all over and query the
> topic metadata again.
> \[2\] The official definition of data loss in kafka is when we lose a
> "committed" message. A message is considered "committed" when all in sync
> replicas for that partition have applied it to their log.
> \[3\] Say a number of ProduceRequests are sitting infront of a controller's
> LeaderAndIsrRequest on a given broker's request queue. Suppose the
> ProduceRequests are for partitions whose leadership has recently shifted out
> from the current broker to another broker in the replica set. Today the
> broker processes the ProduceRequests before the LeaderAndIsrRequest, meaning
> the ProduceRequests are getting processed on the former partition leader. As
> part of becoming a follower for a partition, the broker truncates the log to
> the high-watermark. With weaker ack settings such as acks=1, the leader may
> successfully write to its own log, respond to the user with a success,
> process the LeaderAndIsrRequest making the broker a follower of the
> partition, and truncate the log to a point before the user's produced
> messages. So users have a false sense that their produce attempt succeeded
> while in reality their messages got erased. While technically part of what
> they signed up for with acks=1, it can still come as a surprise.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
