[Libevent-users] Problems with deferred HTTP handlers over SSL

[Amarin Phaosawasdi]

Hi All,

We're working on fixing an issue with one of our servers, which serves
HTTPS requests on one end, and makes asynchronous RPCs on the other end.

The server uses libevhtp to serve HTTPS requests. But since it is also
based on libevent and we suspect it might have something to do with
bufferevent_openssl, we thought it is appropriate to post it here.

When a HTTPS request comes in, a the server invokes an async RPC to an
internal server. When the RPC finishes, we then use the RPC results to
provide an HTTPS response to the browser.

However, almost all the time the browser gives an error.
- "SSL received a record with an incorrect Message Authentication Code.
(Error code: ssl_error_bad_mac_read)." (in Firefox)
- "Error 107 (net::ERR_SSL_PROTOCOL_ERROR): SSL protocol error." (in Chrome)
- "Error 324 (net::ERR_EMPTY_RESPONSE): The server closed the connection
without sending any data. (in Chrome)

Rarely, when the browser doesn't give an error, it either succeeds or
blocks forever.

Repeated fails normally end with a segmentation fault in the server.

When a synchronous RPC is used instead (everything is finished on the
callback stack), the error is gone and every request is successful.

Also if the server is configured to serve HTTP requests instead of HTTPS,
it also always succeeds regardless or the RPC calling mode.

Does anybody have an idea what might be going on?

The attached code can be used to reproduce this issue. When the server is
run, use a browser to connect to https://127.0.0.1:1025.

Tested with:
gcc/g++ 4.4.5
Both openssl 0.9.8 and 1.0.0f
libevhtp development branch
libevent master branch
Debian 2.6.39-bpo.2-amd64 x86_64

Thank you,
Amarin

libevhtp.key
Description: Binary data

libevhtp.pem
Description: application/x509-ca-cert

#include <unistd.h>
#include <pthread.h>
#include "evhtp.h"

void *DeferredHandler(void *arg) {
  sleep(1);
  evhtp_request_t *req = (evhtp_request_t*)arg;
  evhtp_headers_add_header(
      req->headers_out,
      evhtp_header_new("content-type", "text/plain; charset=utf-8", 0, 0));
  evbuffer_add(req->buffer_out, "hello", 5);
  evhtp_send_reply(req, EVHTP_RES_OK);
  pthread_exit(NULL);
}

void Handler(evhtp_request_t *req, void *) {
  pthread_t t;
  pthread_create(&t, NULL, DeferredHandler, (void*)req);
  pthread_detach(t);  // this fails
  // pthread_join(t, NULL);  // but this works
}

int main() {
  evbase_t *base = event_base_new();
  if (!base) return 1;

  evhtp_t *http = evhtp_new(base, NULL);
  if (!http) return 1;

  evhtp_set_gencb(http, Handler, NULL);
  evhtp_use_threads(http, NULL, 4, NULL);

  evhtp_ssl_cfg_t scfg;
  char ciphers[] = "RC4+RSA:HIGH:+MEDIUM:+LOW";
  char libevhtp_pemfile[] = "libevhtp.pem";
  scfg.pemfile = libevhtp_pemfile;
  char libevhtp_privfile[] = "libevhtp.key";
  scfg.privfile = libevhtp_privfile;
  scfg.cafile = NULL;
  scfg.capath = NULL;
  scfg.ciphers = ciphers;
  scfg.ssl_opts = SSL_OP_NO_SSLv2;
  scfg.verify_peer = SSL_VERIFY_NONE;
  scfg.verify_depth = 0;
  scfg.x509_verify_cb = NULL;
  scfg.x509_chk_issued_cb = NULL;
  scfg.store_flags = 0;
  scfg.scache_type = evhtp_ssl_scache_type_internal;
  scfg.scache_timeout = 1024;
  scfg.scache_size = 1024;
  scfg.scache_init = NULL;
  scfg.scache_add = NULL;
  scfg.scache_get = NULL;
  scfg.scache_del = NULL;
  scfg.args = NULL;
  evhtp_ssl_init(http, &scfg);

  // bind and serve
  if (evhtp_bind_socket(http, "127.0.0.1", 1025, 128) == -1) return 1;
  event_base_loop(base, 0);
  event_base_free(base);

  return 0;
}