Hi all, After apply the patch from HADOOP-6713 into our hadoop source tree, I did a benchmark on multi-readers RPC, and didn't found any throughput improvement . The attachment of this mail is a patch of my rpc benmark. Test scenario listed below.
Network: Gigabit LAN CPU: 8 core Intel(R) Xeon(R) CPU E5420 @ 2.50GHz Memory: 8GB ================== scenario 1 ======================= server side: # export HADOOP_OPS="-Xmx4096m -Xms256m" # hadoop/bin/hadoop jar hadoop/hadoop-0.19.1-test.jar rpcbench -server -handlers 1 -readers 10 -durationPerRPC 100 client side: # hadoop/bin/hadoop jar hadoop/hadoop-0.19.1-test.jar rpcbench -client -host hdp38 -p 38668 -req 100 -threads 10 -size 8 -arraySize 200000 10/12/17 12:25:07 INFO ipc.RPCBenchmark: Starting 100 rpc call(s). 10/12/17 12:26:02 INFO ipc.RPCBenchmark: # operations: 100 10/12/17 12:26:02 INFO ipc.RPCBenchmark: Elapsed Time: 55073 ms 10/12/17 12:26:02 INFO ipc.RPCBenchmark: Ops per sec: 1.815771793800955 10/12/17 12:26:02 INFO ipc.RPCBenchmark: Average Time: 5249 ms ================== scenario 2 ======================= server side: # export HADOOP_OPS="-Xmx4096m -Xms256m" # hadoop/bin/hadoop jar hadoop/hadoop-0.19.1-test.jar rpcbench -server -handlers 10 -readers 10 -durationPerRPC 100 client side: # hadoop/bin/hadoop jar hadoop/hadoop-0.19.1-test.jar rpcbench -client -host hdp38 -p 38668 -req 100 -threads 10 -size 8 -arraySize 200000 10/12/17 12:28:23 INFO ipc.RPCBenchmark: Starting 100 rpc call(s). 10/12/17 12:29:11 INFO ipc.RPCBenchmark: # operations: 100 10/12/17 12:29:11 INFO ipc.RPCBenchmark: Elapsed Time: 47784 ms 10/12/17 12:29:11 INFO ipc.RPCBenchmark: Ops per sec: 2.092750711535242 10/12/17 12:29:11 INFO ipc.RPCBenchmark: Average Time: 4581 ms ================== scenario 3 ======================= server side: # export HADOOP_OPS="-Xmx4096m -Xms256m" # hadoop/bin/hadoop jar hadoop/hadoop-0.19.1-test.jar rpcbench -server -handlers 10 -readers 1 -durationPerRPC 100 client side: # hadoop/bin/hadoop jar hadoop/hadoop-0.19.1-test.jar rpcbench -client -host hdp38 -p 38668 -req 100 -threads 10 -size 8 -arraySize 200000 10/12/17 12:31:19 INFO ipc.RPCBenchmark: Starting 100 rpc call(s). 10/12/17 12:32:08 INFO ipc.RPCBenchmark: # operations: 100 10/12/17 12:32:08 INFO ipc.RPCBenchmark: Elapsed Time: 49155 ms 10/12/17 12:32:08 INFO ipc.RPCBenchmark: Ops per sec: 2.034381039568711 10/12/17 12:32:08 INFO ipc.RPCBenchmark: Average Time: 4722 ms ================== scenario 4 ======================= server side: # export HADOOP_OPS="-Xmx4096m -Xms256m # hadoop/bin/hadoop jar hadoop/hadoop-0.19.1-test.jar rpcbench -server -handlers 1 -readers 1 -durationPerRPC 100 client side: # hadoop/bin/hadoop jar hadoop/hadoop-0.19.1-test.jar rpcbench -client -host hdp38 -p 38668 -req 100 -threads 10 -size 8 -arraySize 200000 10/12/17 12:36:04 INFO ipc.RPCBenchmark: Starting 100 rpc call(s). 10/12/17 12:36:59 INFO ipc.RPCBenchmark: # operations: 100 10/12/17 12:36:59 INFO ipc.RPCBenchmark: Elapsed Time: 55054 ms 10/12/17 12:36:59 INFO ipc.RPCBenchmark: Ops per sec: 1.816398445162931 10/12/17 12:36:59 INFO ipc.RPCBenchmark: Average Time: 5278 ms Statistics above indicates that a single reader can consume 200,000 8-bytes Strings very fast. Ten concurrent readers should consume the same data more quickly, but reader isn't the bottleneck of RPC at least in these scenarios. Can you give an example which scenario HADOOP-6713 will take affect? Thanks, Min -- My research interests are distributed systems, parallel computing and bytecode based virtual machine. My profile: http://www.linkedin.com/in/coderplay My blog: http://coderplay.javaeye.com
