Janus Chow created HADOOP-17356:
-----------------------------------
Summary: RPC FairCallQueue for special users
Key: HADOOP-17356
URL: https://issues.apache.org/jira/browse/HADOOP-17356
Project: Hadoop Common
Issue Type: Improvement
Reporter: Janus Chow
Attachments: Implement 0.png, Implement 1.png, Implement 2.png,
Implement 3.png
In HADOOP-15016, the idea was first raised to support special users by
assigning each special user an independent queue with a share. The design was
intended for the user to better control the RPC schedule, but there is also a
risk that users may add a lot of items in the config of special-users, causing
a lot of queues in the RPCScheduler.
This ticket records some ideas to mitigate the risks while solving the
special-user problem based on HADOOP-15016.
0. The current implementation is as follows, all users will be treated equally,
_multiplexer_ will decide the call count in each queue.
!Implement 0.png!
1. The first idea is to amplify the weight of super-users and resue the initial
queues. This idea is easy to implement, but ordinary users and special users
would be affected by each other, and it would be difficult for the
_multiplexer_ to guarantee the traffic of super-suers.
!Implement 1.png!
2. The second idea is to set up one independent queue for all special users
with a config controlling the weight of all special-users. One concern for this
idea is that the scheduler between super-users' calls may not be fair.
!Implement 2.png!
3. The third idea is to also use priority queues for special-users based on
idea 2, ensuring the fair handling of all super-users. Another benefit of this
idea is we can use the queues to implement cost-based calculation.
!Implement 3.png!
I think Idea 3 should be a good balance of complexity and useability.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
