Thanks for your analysis, Steffen. Dropping the Debian-specific patch is definitely the way to go for libwww/LWP. However I still believe IO::Socket::SSL should provide a way to clear SSL_MODE_AUTO_RETRY in order to fix applications relying on the former OpenSSL defaults, as suggested in the OpenSSL changelog:
“SSL_MODE_AUTO_RETRY is enabled by default. Applications that use
blocking I/O in combination with something like select() or poll()
will hang. This can be turned off again using SSL_CTX_clear_mode().”
Otherwise the “usual” way to write event loops in blocking I/O won't be
possible with IO::Socket::SSL.
On Sat, 11 May 2019 at 21:56:01 +0200, Steffen Ullrich wrote:
> As far as I can see it has nothing to do with SSL_MODE_AUTO_RETRY but
> instead is caused by expectations on the behavior of select which are
> wrong with TLS 1.3.
Please consider the enclosed netcat-like program. I don't think I'm
relying on any particular behavior of a specific TLS version, and
follow the practices for polling blocking sockets, as documented in
libssl, Net::SSLeay, and IO::Socket::SSL, namely:
- If SSL_pending() > 0, skip the (blocking) select() call and instead
call SSL_read() to process remaining bytes in the current SSL frame.
- If SSL_read() fails and sets SSL_ERROR_WANT_READ, don't treat it as
a read error.
The last point however relies on SSL_MODE_AUTO_RETRY being *unset*, like
it used to be with OpenSSL <1.1.1a. With SSL_MODE_AUTO_RETRY being set,
the program doesn't work properly. (*Not only for TLSv1.3, but also for
TLSv1.2*). This is expected with the new default:
“If the underlying BIO is blocking, a read function will only return
once the read operation has been finished or an error occurred,
except when a non-application data record has been processed and
SSL_MODE_AUTO_RETRY is not set. Note that if SSL_MODE_AUTO_RETRY is
set and only non-application data is available the call will hang.”
— https://www.openssl.org/docs/manmaster/man3/SSL_read.html
As seen below, this also breaks with ≤TLSv1.2; but only when the TLS
session is renegotiated, not during the initial handshake.
Generate a self-signed certificate:
$ openssl req -x509 -keyout /tmp/key.pem -out /tmp/cert.pem -subj
/CN=127.0.0.1 -nodes
Start a TLSv1.2 server on [127.0.0.1]:4433:
$ openssl s_server -accept 127.0.0.1:4433 -key /tmp/key.pem -cert
/tmp/cert.pem -tls1_2
Now start the enclosed program in another terminal. What's being
written in the s_server(1ssl) TTY is echoed on the netcat.pl side, and
vice versa. All good. Now trigger renegotiate the TLS session by
pressing ‘r\n’. The server prints
SSL_do_handshake -> 1
Read BLOCK
and netcat ends up being stuck in a blocking read(). So what's being
written client-side won't show up anymore in the server window, until
data is being sent from the server to the client and makes read()
return.
openssl s_server … -tls1_2 netcat.pl
----------------------------------- ---------
S: Using default temp DH parameters
S: ACCEPT
S: -----BEGIN SSL SESSION PARAMETERS-----
S: […]
S: ---
S: No server certificate CA names sent
S: CIPHER is ECDHE-RSA-AES128-GCM-SHA256
S: Secure Renegotiation IS supported
S: Entering loop...
C: can you hear me now?
S: can you hear me now?
C: yes
S: yes
C: good
S: good
C: starving you now
S: starving you now
C: r
S: SSL_do_handshake -> 1
S: Read BLOCK
C: meh, I'm muted
C: unstarving
S: meh, I'm muted
S: unstarving
(The ‘C: ’ prefix indicates a line written to the standard input, and
the ‘S: ’ prefix a line written to the standard output or error output.)
After renegotiation, the client is stuck in a blocking read() until the
server sends some data. Same thing with TLSv1.3, but of course without
the renegotiation part: this happens right at the begining.
openssl s_server … -tls1_3 netcat.pl
----------------------------------- ---------
S: Using default temp DH parameters
S: ACCEPT
S: -----BEGIN SSL SESSION PARAMETERS-----
S: […]
S: ---
S: No server certificate CA names sent
S: CIPHER is TLS_AES_256_GCM_SHA384
S: Secure Renegotiation IS supported
S: Entering loop...
C: can you hear me now?
C: I guess no...
C: unstarving
S: can you hear me now?
S: I guess no...
S: unstarving
Now the same trace, clearing SSL_MODE_AUTO_RETRY:
openssl s_server … -tls1_2 netcat.pl
----------------------------------- ---------
S: Using default temp DH parameters
S: ACCEPT
S: -----BEGIN SSL SESSION PARAMETERS-----
S: […]
S: ---
S: No server certificate CA names sent
S: CIPHER is ECDHE-RSA-AES128-GCM-SHA256
S: Secure Renegotiation IS supported
S: Entering loop...
C: can you hear me now?
S: can you hear me now?
C: yes
S: yes
C: good
S: good
C: starving you now
S: starving you now
C: r
S: SSL_do_handshake -> 1
S: Read BLOCK
S: SSL want read!
C: haha, I'm not muted
S: haha, I'm not muted
C: indeed you're not
S: indeed you're not
And for TLSv1.3:
openssl s_server … -tls1_3 netcat.pl
----------------------------------- ---------
S: Using default temp DH parameters
S: ACCEPT
S: -----BEGIN SSL SESSION PARAMETERS-----
S: […]
S: ---
S: No server certificate CA names sent
S: CIPHER is TLS_AES_256_GCM_SHA384
S: Secure Renegotiation IS supported
S: Entering loop...
S: SSL want read!
S: SSL want read!
C: can you hear me now?
S: can you here me now?
C: yup
S: yup
C: woo
S: woo
So with SSL_MODE_AUTO_RETRY being unset, SSL_read() fails and set
SSL_ERROR_WANT_READ (the “SSL want read!” messages above). That's AFAICT
the only expectation of that select() loop, and isn't depended on the
TLS version. As mentioned in the OpenSSL changelog, setting
SSL_MODE_AUTO_RETRY by default breaks programs using select() with
blocking I/O. For the convenience of “poorly written” programs that do
not properly handle non-application data records. (With the former
defaults they would break on renegotiation with ≤TLSv1.2; and right at
the beginning for TLSv1.3. For programs with event loops it's the other
way around.)
> But sysread on a blocking socket is supposed to only return if there
> was at least 1 byte read or if the peer closed the connection or if a
> permanent error occured. And since no application data were send it
> just hangs.
It doesn't do a direct sysread on the raw socket, does it? AFAICT
$socket->sysread() calls Net::SSLeay::read(), which might fail and
return SSL_ERROR_WANT_READ when receiving a non-application data record.
With SSL_MODE_AUTO_RETRY being cleared, that is.
As neither IO::Socket nor Net::SSLeay forbids the use of polling &
blocking sockets, and the new OpenSSL defaults breaks event loops in
blocking I/O (and that also for <v1.3), please provide an option to turn
off SSL_MODE_AUTO_RETRY on object creation, as suggested by the OpenSSL
developers :-)
--
Guilhem.
#!/usr/bin/perl
use warnings;
use strict;
use IO::Socket::SSL;
my $sock = IO::Socket::SSL->new(
PeerAddr => "127.0.0.1:4433",
SSL_ca_file => "/tmp/cert.pem",
#SSL_mode_auto_no_retry => 1,
) // die;
my ($std_buf, $std_off, $std_len) = ("", 0, 0);
my ($sock_buf, $sock_off, $sock_len) = ("", 0, 0);
sub ssl_read() {
if (defined (my $n = $sock->sysread($sock_buf, 4096, $sock_len))) {
exit 0 unless $n > 0;
$sock_len += $n;
} elsif ($SSL_ERROR == SSL_WANT_READ) {
print STDERR "SSL want read!\n";
} else {
die "sysread: $!";
}
}
print STDERR "Entering loop...\n";
while (1) {
if ((my $n = $sock->pending()) > 0) {
print STDERR "$n bytes left in the current SSL frame\n";
ssl_read();
next;
}
my $rin = "";
vec($rin, fileno(STDIN), 1) = 1;
vec($rin, $sock->fileno(), 1) = 1;
my $win = "";
vec($win, fileno(STDOUT), 1) = 1 if $sock_off < $sock_len;
vec($win, $sock->fileno(), 1) = 1 if $std_off < $std_len;
select(my $rout = $rin, my $wout = $win, undef, undef);
if (vec($rout, fileno(STDIN), 1) == 1) {
my $n = sysread(STDIN, $std_buf, 4096, $std_len) // die "sysread: $!";
exit 0 unless $n > 0;
$std_len += $n;
}
if (vec($rout, $sock->fileno(), 1) == 1) {
ssl_read();
}
if (vec($wout, fileno(STDOUT), 1) == 1) {
my $n = syswrite(STDOUT, $sock_buf, $sock_len-$sock_off, $sock_off) // die "syswrite: $!";
$sock_off += $n;
($sock_buf, $sock_off, $sock_len) = ("", 0, 0) unless $sock_off < $sock_len;
}
if (vec($wout, $sock->fileno(), 1) == 1) {
my $n = $sock->syswrite($std_buf, $std_len-$std_off, $std_off) // die "syswrite: $!";
$std_off += $n;
($std_buf, $std_off, $std_len) = ("", 0, 0) unless $std_off < $std_len;
}
}
signature.asc
Description: PGP signature
