Eric, Exactly what I was looking for! I found out that there was a high CPU consuming function on my server which was causing the crash under heavy load due to frequent context switching. After removing that and using the server config in AsyncServer.java the throughput substantially increased! I'll look at the different server and client configs in the qps_server/qps_client and see what works best for me.
Thanks so much again! On Friday, November 3, 2017 at 3:47:58 PM UTC-7, Eric Anderson wrote: > > On Thu, Nov 2, 2017 at 3:29 PM, <[email protected] <javascript:>> wrote: >> >> Is that 1000 QPS total, and all over one stream? Using a single stream >>> can't use much more than a single core of processing (excluding protobuf >>> and the application), so you may use some more streams. But 1k QPS is >>> really low. We see 750k QPS >>> <https://performance-dot-grpc-testing.appspot.com/explore?dashboard=5652536396611584> >>> ("Streaming >>> secure throughput QPS (8 core client to 8 core server)") between a client >>> and server with 8 cores each. Even with non-streaming RPCs we see 250k QPS. >> >> >> I'm trying 1000 QPS continuously for a minute, all over one stream. FWIW, >> 500 QPS works fine and I tried that up to 5 minutes. For those charts you >> showed, are you able to show the server and client configs/code that >> produced those metrics? Also, I am still using GRPC 1.5, and I know there >> were performance improvements from then but I still think 1000 QPS is still >> very low for this version. >> >> Anything else you could recommend? Thanks again. >> > > If you're running on a desktop or server class machine with at least a > couple cores available with ethernet running between the client and server, > then I don't know what would cause it to be that low. > > The performance test can be run with run_performance_test.py > <https://github.com/grpc/grpc/tree/master/tools/run_tests#performance-benchmarks-run_performance_testspy>... > > but... it's complicated. The Java code it uses is in the benchmarks > directory > <https://github.com/grpc/grpc-java/tree/master/benchmarks/src/main/java/io/grpc/benchmarks/driver> > . > > I'd instead suggest going into grpc-java's benchmark directory, running > `./gradlew -PskipCodegen=true installDist` and then running the binaries > in build/install/grpc-benchmarks/bin: qps_server, qps_client, and maybe > openloop_client. qps_client does ping-pong, so if you use only one RPC > you're just measuring latency. But I'm seeing 10k QPS in that setup on > localhost. > > In one terminal: > $ build/install/grpc-benchmarks/bin/qps_server --addr:1234 > > Then in another: > $ build/install/grpc-benchmarks/bin/qps_client --address=localhost:1234 > --duration=300 --channels=1 --outstanding_rpcs=1 --streaming_rpcs > > The qps_client, for each RPC, sends a message, waits for the response, > then sends another message. openloop_client is closer to what you want, but > it will is easier to "hold wrong." It does unary RPCs at a requested rate, > but it spin-waits and issues all RPCs from the same thread, so its maximum > QPS is artificially limited. > -- You received this message because you are subscribed to the Google Groups "grpc.io" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/grpc-io. To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/c0f9d824-00cf-4533-aaea-5ba8ca5c648d%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
