Hi Mark, On Tue, Sep 15, 2020 at 3:34 PM Mark Thomas <ma...@apache.org> wrote:
> On 15/09/2020 12:46, Martin Grigorov wrote: > > On Tue, Sep 15, 2020 at 2:37 PM Martin Grigorov <mgrigo...@apache.org> > > wrote: > > > >> Hi, > >> > >> I am running some load tests on Tomcat and I've noticed that when HTTP2 > is > >> enabled the throughput drops considerably. > >> > >> Here are the steps to reproduce: > >> > >> 1) Enable HTTP2, e.g. by commenting out this connector: > >> > >> > https://github.com/apache/tomcat/blob/d381d87005fa89d1f19d9091c0954f317c135d9d/conf/server.xml#L103-L112 > >> > >> 2) Download Vegeta load tool from: > >> https://github.com/tsenart/vegeta/releases/ > >> > >> 3) Run the load tests: > >> > >> 3.1) HTTP/1.1 > >> echo -e '{"method": "GET", "url": "http://localhost:8080/examples/"}' | > >> vegeta attack -format=json -rate=0 -max-workers=1000 -duration=10s | > >> vegeta encode > /tmp/http1.json; and vegeta report -type=json > >> /tmp/http1.json | jq . > >> > >> 3.2) HTTP2 > >> echo -e '{"method": "GET", "url": "https://localhost:8443/examples/"}' > | > >> vegeta attack -format=json -http2 -rate=0 -max-workers=1000 -insecure > >> -duration=10s | vegeta encode > /tmp/http2.json; and vegeta report > >> -type=json /tmp/http2.json | jq . > >> > >> As explained at https://github.com/tsenart/vegeta#-rate -rate=0 means > >> that Vegeta will try to send as many requests as possible with the > >> configured number of workers. > >> I use '-insecure' because I use self-signed certificate. > >> > >> On my machine I get around 14-15K reqs/sec for HTTP1.1 with only > responses > >> with code=200 . > >> But for HTTP2 Tomcat starts returning such kind of errors: > >> > >> "errors": [ > >> "Get \"https://localhost:8443/examples/\": http2: server sent > GOAWAY > >> and closed the connection; LastStreamID=9259, ErrCode=PROTOCOL_ERROR, > >> debug=\"Stream [9,151] has been closed for some time\"", > >> "http2: server sent GOAWAY and closed the connection; > >> LastStreamID=9259, ErrCode=PROTOCOL_ERROR, debug=\"Stream [9,151] has > been > >> closed for some time\"", > >> "Get \"https://localhost:8443/examples/\": http2: server sent > GOAWAY > >> and closed the connection; LastStreamID=239, ErrCode=PROTOCOL_ERROR, > >> debug=\"Stream [49] has been closed for some time\"" > >> ] > >> > >> when I ask for more than 2000 reqs/sec, i.e. -rate=2000/1s > > That indicates that the client has sent a frame associated with a stream > that the server closed previously and that that stream has been removed > from the Map of known streams to make room for new ones. See > Http2UpgardeHandler.pruneClosedStreams() > > It looks like the client is making assumptions about server behaviour > that go beyond the requirements of RFC 7540, section 5.3.4. > This is possible! I've just tested with two more HTTP2 impls: 1) Node.js http2-server.js =================================================== const http2 = require('http2'); const fs = require('fs'); const server = http2.createSecureServer({ key: fs.readFileSync('/path/to/server.key'), cert: fs.readFileSync('/path/to/server.crt') }); server.on('error', (err) => console.error(err)); server.on('stream', (stream, headers) => { // stream is a Duplex stream.respond({ 'content-type': 'text/plain; charset=utf-8', ':status': 200 }); stream.end('Hello world!'); }); server.listen(18080); =================================================== run with: node http2-server.js Runs fine with -rate=0 and gives around 8K reqs/sec 2) Rust Cargo.toml =================================================== [package] name = "my-http2-server" version = "0.0.1" publish = false authors = ["Martin Grigorov <mgrigo...@apache.org>"] license = "MIT/Apache-2.0" description = "Load test HTTP/2 " repository = "https://github.com/martin-g/http2-server-rust"; keywords = ["http2"] edition = "2018" [dependencies] actix-web = { version = "3", features = ["openssl"] } openssl = { version = "0.10", features = ["v110"] } =================================================== src/main.rs =================================================== use actix_web::{web, App, HttpRequest, HttpServer, Responder}; use openssl::ssl::{SslAcceptor, SslFiletype, SslMethod}; async fn index(_req: HttpRequest) -> impl Responder { "Hello world!" } #[actix_web::main] async fn main() -> std::io::Result<()> { let mut builder = SslAcceptor::mozilla_intermediate(SslMethod::tls()).unwrap(); builder .set_private_key_file("/path/to/server.key", SslFiletype::PEM) .unwrap(); builder.set_certificate_chain_file("/path/to/server.crt").unwrap(); HttpServer::new(|| App::new().route("/", web::get().to(index))) .bind_openssl("127.0.0.1:18080", builder)? .run() .await } =================================================== run with: cargo run Again no errors, throughput: 3K reqs/sec I will test with Netty tomorrow too, but so far it looks like only Tomcat fails under load. > >> All the access logs look like: > >> > >> 127.0.0.1 - - [15/Sep/2020:13:59:24 +0300] "GET /examples/ HTTP/2.0" 200 > >> 1126 > >> 127.0.0.1 - - [15/Sep/2020:13:59:24 +0300] "GET /examples/ HTTP/2.0" 200 > >> 1126 > >> 127.0.0.1 - - [15/Sep/2020:13:59:24 +0300] "GET /examples/ HTTP/2.0" 200 > >> 1126 > >> 127.0.0.1 - - [15/Sep/2020:13:59:24 +0300] "GET /examples/ HTTP/2.0" 200 > >> 1126 > >> 127.0.0.1 - - [15/Sep/2020:13:59:24 +0300] "GET /examples/ HTTP/2.0" 200 > >> 1126 > >> 127.0.0.1 - - [15/Sep/2020:13:59:24 +0300] "GET /examples/ HTTP/2.0" 200 > >> 1126 > >> > >> i.e. there are no error codes, just 200. > >> Vegeta reports the error with status code = 0. I think this just means > >> that it didn't get a proper HTTP response but just TCP error. > >> There are no errors in catalina.out. > >> > >> Are there any settings I can tune to get better throughput with HTTP2 ? > >> > >> Tomcat 10.0.0-M8. > > If you really want to maximise throughput then you need to reduce the > number of concurrent requests to (or a little above) the number of cores > available on the server. Go higher and you'll start to see throughput > tail off due to context switching. > > If you want to demonstrate throughput with a large number of clients > you'll probably need to experiment with both maxThreads, > I've forgot to say that I use maxThreads=8. > maxConcurrentStreams and maxConcurrentStreamExecution. > > If I had to guess, I'd expect maxConcurrentStreams == > maxConcurrentStreamExecution and low numbers for all of them to give the > best results. > I will check those tomorrow! Thanks! > > Mark > > > > Forgot to mention that I've also tested with JMeter + > > https://github.com/Blazemeter/jmeter-http2-plugin but there it fails > with > > OOM if I use more than 2000 virtual users. Increasing the memory still > does > > not give such good results as Vegeta for HTTP/1.1. Also JMeter uses > > sequential model. > > > > For comparison, I've also tested with a simple Golang based HTTP2 server: > > > > http2-server.go: > > ========================================================== > > package main > > > > import ( > > "fmt" > > "log" > > "net/http" > > "os" > > ) > > > > func main() { > > > > port := 8080 > > if port == "" { > > log.Fatal("Please specify the HTTP port as environment variable, > e.g. > > env PORT=8081 go run http-server.go") > > } > > > > tls_root := "/path/to/certs/" > > srv := &http.Server{Addr: ":" + port, Handler: > http.HandlerFunc(handle)} > > log.Fatal(srv.ListenAndServeTLS(tls_root + "server.crt", tls_root + > > "server.key")) > > } > > > > func handle(w http.ResponseWriter, r *http.Request) { > > // log.Printf("Got connection: %s", r.Proto) // prints HTTP/2.0 > > fmt.Fprintf(w, "Hello World") > > } > > ========================================================== > > > > Here Vegeta makes around 13K reqs/sec without error responses. > > > > To run this app do: go run http2-server.go > > > > > >> Regards, > >> Martin > >> > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: users-unsubscr...@tomcat.apache.org > For additional commands, e-mail: users-h...@tomcat.apache.org > >
