Yingqi Lu created HDFS-9861:
-------------------------------
Summary: Add SO_REUSEPORT into HDFS
Key: HDFS-9861
URL: https://issues.apache.org/jira/browse/HDFS-9861
Project: Hadoop HDFS
Issue Type: New Feature
Reporter: Yingqi Lu
Hello All,
SO_REUSEPORT is a kernel feature which enables multiple listen sockets to
listen to the same IP:PORT pair. This can potentially improve the scalability
and parallelism of network traffic handling, especially for big core count
systems or large sized clusters. The feature is enabled for both TCP and UDP
sockets. Linux, BSD and majority of other UNIX based OSes already have the
feature supported. Windows has not enabled it yet. For more details on the
feature itself, please refer to https://lwn.net/Articles/542629/.
Recently, SO_REUSEPORT is added into OpenJDK9 as one of the socket options. For
more details, please refer to
http://hg.openjdk.java.net/jdk9/dev/jdk/rev/f92af283ab18#l13.1.
Here, we are proposing to add SO_REUSEPORT support into HDFS for its NameNode
and DataNode implementations. We have a quick prototype which shows good
throughput and latency improvements on given micro workload (purely HDFS
workload, not MapReduce based). We will wrap up all the details together with a
patch and send them here soon for further discussion and review. At the
meantime, please let us know your comment and feedback.
Thanks,
Lucy
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
