> And, finally:
>
> 4. they DO NOT work when loaded by httpd
>
> I will be the first to admit that I don't really "know" much about
> public key cryptography and how openssl implements things. But, being
> simple, it seems to me that there are really only two possibilities.
>
> Either apache, pound, and openssl s_server are all flawed and are
> incorrectly using an invalid certificate/key pair for encryption; or
> there is a problem in httpd and how it deals with certificates and
> https.
>
> I will try things again tomorrow (later today) and see if I can get
> any info with tcpdump.
>
> If there is anything else to try, please let me know.
Please try 's_client' as I had suggested in an earlier email - it's not
the certificates themselves you should be testing (i.e. with 's_server'
and a web browser) but certificates/keys *with* httpd and a client which
will give you meaningful output ('s_client').
Like I had also mentioned earlier - I had generated a new certificate
and a key and tested it on one of my machines and it all works just
fine.
Last, but not least - if you hadn't done so already,please make sure you
are running the latest snapshot.
Regards,
Raf
----------
First, I installed the most recent snapshot:
OpenBSD 5.7-current (GENERIC.MP) #896: Thu Mar 26 14:56:12 MDT 2015
t...@amd64.openbsd.org:/usr/src/sys/arch/amd64/compile/GENERIC.MP
(sorry about the other dmesg snip - I pulled the snip off the top, and I was
not aware that on this system the message buffer
survives a reboot)
First, I started httpd as "httpd -d -v -v -v -v -v -v -v":
The terminal spits back:
startup
socket_rlimit: max open files 1024
socket_rlimit: max open files 1024
server_tls_load_keypair: using certificate /etc/ssl/server.crt
server_tls_load_keypair: using private key /etc/ssl/private/server.key
socket_rlimit: max open files 1024
server_privinit: adding server default
server_privinit: adding server default
server_launch: running server default
server_launch: running server default
server_launch: running server default
then, on an second terminal, I tried connecting with: openssl s_client
-connect 127.0.0.1:443
on the second (s_client) terminal, I get:
CONNECTED(00000003)
And that's it.
On the first (httpd) terminal, there is no output of any kind.
So, I waited about 10 seconds, nothing happened, and I shut down httpd. The
terminal says:
^C
logger exiting, pid 14644
server exiting, pid 21849
server exiting, pid 18400
server exiting, pid 10463
parent terminating, pid 9974
Then, I opened a s_server instance: openssl s_server -accept 443 -www -cert
/etc/ssl/server.crt -key /etc/ssl/private/server.key
It gives me:
Using auto DH parameters
Using default temp ECDH parameters
ACCEPT
And on the second terminal I try s_client again: openssl s_client -connect
127.0.0.1:443
And it connects. Here is the output (I XXX'ed some of the certificate info):
openssl s_client -connect 127.0.0.1:443
CONNECTED(00000003)
depth=0 C = US, ST = XXX, O = XXX, OU = XXX, L = XXX, CN = XXX, emailAddress =
XXX
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 C = US, ST = XXX, O = XXX, OU = XXX, L = XXX, CN = XXX, emailAddress =
XXX
verify error:num=27:certificate not trusted
verify return:1
depth=0 C = US, ST = XXX, O = XXX, OU = XXX, L = XXX, CN = XXX, emailAddress =
XXX
verify error:num=21:unable to verify the first certificate
verify return:1
---
Certificate chain
0 s:/C=US/ST=XXX/O=XXX/OU=XXX/L=XXX/CN=XXX/emailAddress=XXX
i:/C=US/ST=XXX/L=XXX/O=XXX/OU=XXX/CN=XXX/emailAddress=XXX
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIH6zCCBdOgAwIBAgIBATANBgkqhkiG9w0BAQsFADCBljELMAkGA1UEBhMCVVMx
ETAPBgNVBAgMCElsbGlub2lzMREwDwYDVQQHDAhXaW5uZXRrYTEUMBIGA1UECgwL
<... more certificate block ...>
6RUcfqhZ211+IvAnJVYAsz+1hzLGL57Ppct6HHf41xl36WakU+J3jlpVpIaA8jHh
5ThHy8QM1jeo90XENClcYD2W1OHD75Hchn5pEbA8BfpKJpvTwsosIFdZazWvHHO8
CU8P6Syj53sEw0MeooEt
-----END CERTIFICATE-----
subject=/C=US/ST=XXX/O=XXX/OU=XXX/L=XXX/CN=XXX/emailAddress=XXX
issuer=/C=US/ST=XXX/L=XX/O=XXX/OU=XXX/CN=XXX/emailAddress=XXX
---
No client certificate CA names sent
---
SSL handshake has read 2933 bytes and written 438 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-CHACHA20-POLY1305
Server public key is 4096 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-CHACHA20-POLY1305
Session-ID: FC424D25B814891FD0F881B1E20C6367547803E189FF2EB1D337201491CB078A
Session-ID-ctx:
Master-Key:
ADB4316898847559BAF6EE1188F1FCFAB0D741D36A73226D023458247CE26523F74EABE327755A7A12CFB9242AAA9413
TLS session ticket lifetime hint: 300 (seconds)
TLS session ticket:
0000 - 8c 5f 29 1e 1e a2 9f e3-f8 3e 62 1f 9f 10 ad 5c ._)......>b....\
0010 - be 8c 47 51 98 4c 93 66-bb a9 51 70 93 37 b3 4e ..GQ.L.f..Qp.7.N
0020 - 16 fc 38 fa f6 ea 37 73-9c d4 82 e9 1a 30 f9 44 ..8...7s.....0.D
0030 - eb 5e 4b 4f b2 c5 9e 00-f1 65 5e d5 b8 d7 76 c7 .^KO.....e^...v.
0040 - 59 24 9a 31 4b 9a a6 07-24 c6 53 ae fe 59 75 dd Y$.1K...$.S..Yu.
0050 - 0d d0 2d 7e cf 16 6b b2-4c 7a 57 e6 df 39 7a f6 ..-~..k.LzW..9z.
0060 - d6 d2 00 13 a0 9c ef 77-06 ba a4 36 e6 3d 1c fc .......w...6.=..
0070 - 1d bb a4 11 b8 be d9 e5-b0 f9 20 e9 b7 00 fe 95 .......... .....
0080 - 79 ff 25 bb 94 41 a8 57-2d 0b ea 43 5d fe 95 8e y.%..A.W-..C]...
0090 - c0 b6 8c 69 c6 c8 9e 9e-a0 3d dc 84 4a 19 60 91 ...i.....=..J.`.
Start Time: 1427479549
Timeout : 300 (sec)
Verify return code: 21 (unable to verify the first certificate)
---
That's it.
(I know that the CA for the cert is not currently trusted by the system above;
but the certificate does work for establishing the
connection with s_client and s_server)
Again, I don't really "know," but it seems to me that openssl s_server is able
to use the certificate without an issue.
However, httpd "loads" the certificate and key at startup, but then refuses to
respond to a connection attempt with that
certificate.
I have NOT tried using another certificate with httpd. I could, and a
different certificate may work just fine. But, that doesn't
seem to be a solution for the problem.
If the certificate is "good" (as demonstrated by s_client and s_server), then
should not httpd "just work" with it?
I don't understand why httpd does not provide any information about the https
connection attempts at all?
Thanks
---------------
Additional info:
I also ran tcpdump now.
First, I would like to say I am outside of my "understanding" zone at this
point, so, if I say something stupid, sorry.
Anyway, I ran tcpdump on the machine that is running the httpd server.
First, I ran s_server (because it works - as: openssl s_server -accept 443
-www -cert /etc/ssl/server.crt -key
/etc/ssl/private/server.key) and used a browser to initiate an https request.
This connection works (as previously described) and tcp dump (tcpdump -e -n -v
'tcp port 443 and host 10.0.128.67') shows:
tcpdump: listening on em0, link-type EN10MB
17:24:44.073781 00:25:90:cb:21:4c 00:25:90:dd:08:59 0800 66: 10.0.128.10.15136
> 10.0.128.67.443: S [tcp sum ok]
1401897034:1401897034(0) win 8192 <mss 1460,nop,wscale 2,nop,nop,sackOK> (DF)
(ttl 128, id 19309, len 52)
17:24:44.073811 00:25:90:dd:08:59 00:25:90:cb:21:4c 0800 66: 10.0.128.67.443 >
10.0.128.10.15136: S [tcp sum ok]
2504529513:2504529513(0) ack 1401897035 win 16384 <mss
1460,nop,nop,sackOK,nop,wscale 3> (DF) (ttl 64, id 21491, len 52)
17:24:44.074127 00:25:90:cb:21:4c 00:25:90:dd:08:59 0800 60: 10.0.128.10.15136
> 10.0.128.67.443: . [tcp sum ok] ack 1 win 16425
(DF) (ttl 128, id 19310, len 40)
17:24:44.074712 00:25:90:cb:21:4c 00:25:90:dd:08:59 0800 237: 10.0.128.10.15136
> 10.0.128.67.443: P 1:184(183) ack 1 win 16425 (DF)
(ttl 128, id 19311, len 223)
17:24:44.086336 00:25:90:dd:08:59 00:25:90:cb:21:4c 0800 1514: 10.0.128.67.443
> 10.0.128.10.15136: . 1:1461(1460) ack 184 win 2190
(DF) (ttl 64, id 10078, len 1500)
17:24:44.086342 00:25:90:dd:08:59 00:25:90:cb:21:4c 0800 1309: 10.0.128.67.443
> 10.0.128.10.15136: P 1461:2716(1255) ack 184 win
2190 (DF) (ttl 64, id 48629, len 1295)
17:24:44.086694 00:25:90:cb:21:4c 00:25:90:dd:08:59 0800 60: 10.0.128.10.15136
> 10.0.128.67.443: . [tcp sum ok] ack 2716 win 16425
(DF) (ttl 128, id 19312, len 40)
17:24:44.099337 00:25:90:cb:21:4c 00:25:90:dd:08:59 0800 180: 10.0.128.10.15136
> 10.0.128.67.443: P 184:310(126) ack 2716 win 16425
(DF) (ttl 128, id 19313, len 166)
17:24:44.099498 00:25:90:cb:21:4c 00:25:90:dd:08:59 0800 85: 10.0.128.10.15136
> 10.0.128.67.443: P [tcp sum ok] 310:341(31) ack
2716 win 16425 (DF) (ttl 128, id 19314, len 71)
17:24:44.099500 00:25:90:cb:21:4c 00:25:90:dd:08:59 0800 60: 10.0.128.10.15136
> 10.0.128.67.443: F [tcp sum ok] 341:341(0) ack 2716
win 16425 (DF) (ttl 128, id 19315, len 40)
<< ... and so on ...>>
Then, I started httpd (httpd -d -v -v -v) and used the same browser to initiate
a connection, with tcpdump watching (tcpdump -e -n
-v 'tcp port 443 and host 10.0.128.67'), and I see:
tcpdump: listening on em0, link-type EN10MB
17:25:39.059332 00:25:90:cb:21:4c 00:25:90:dd:08:59 0800 66: 10.0.128.10.15143
> 10.0.128.67.443: S [tcp sum ok]
2859303750:2859303750(0) win 8192 <mss 1460,nop,wscale 2,nop,nop,sackOK> (DF)
(ttl 128, id 19780, len 52)
17:25:39.059361 00:25:90:dd:08:59 00:25:90:cb:21:4c 0800 66: 10.0.128.67.443 >
10.0.128.10.15143: S [tcp sum ok]
1746765713:1746765713(0) ack 2859303751 win 16384 <mss
1460,nop,nop,sackOK,nop,wscale 3> (DF) (ttl 64, id 2718, len 52)
17:25:39.059659 00:25:90:cb:21:4c 00:25:90:dd:08:59 0800 60: 10.0.128.10.15143
> 10.0.128.67.443: . [tcp sum ok] ack 1 win 16425
(DF) (ttl 128, id 19781, len 40)
17:25:39.060293 00:25:90:cb:21:4c 00:25:90:dd:08:59 0800 571: 10.0.128.10.15143
> 10.0.128.67.443: P 1:518(517) ack 1 win 16425 (DF)
(ttl 128, id 19782, len 557)
17:25:39.254335 00:25:90:dd:08:59 00:25:90:cb:21:4c 0800 54: 10.0.128.67.443 >
10.0.128.10.15143: . [bad tcp cksum 1468! -> 36ee]
ack 518 win 2125 (DF) (ttl 64, id 45699, len 40)
17:25:49.251721 00:25:90:cb:21:4c 00:25:90:dd:08:59 0800 60: 10.0.128.10.15143
> 10.0.128.67.443: . [tcp sum ok] 517:518(1) ack 1
win 16425 (DF) (ttl 128, id 19859, len 41)
17:25:49.251739 00:25:90:dd:08:59 00:25:90:cb:21:4c 0800 54: 10.0.128.67.443 >
10.0.128.10.15143: . [bad tcp cksum 1468! -> 36ee]
ack 518 win 2125 (DF) (ttl 64, id 12769, len 40)
17:25:59.252283 00:25:90:cb:21:4c 00:25:90:dd:08:59 0800 60: 10.0.128.10.15143
> 10.0.128.67.443: . [tcp sum ok] 517:518(1) ack 1
win 16425 (DF) (ttl 128, id 19906, len 41)
17:25:59.252298 00:25:90:dd:08:59 00:25:90:cb:21:4c 0800 54: 10.0.128.67.443 >
10.0.128.10.15143: . [bad tcp cksum 1468! -> 36ee]
ack 518 win 2125 (DF) (ttl 64, id 8515, len 40)
17:26:00.976778 00:25:90:dd:08:59 00:25:90:cb:21:4c 0800 54: 10.0.128.67.443 >
10.0.128.10.15143: R [bad tcp cksum 1468! -> 3f37]
1:1(0) ack 518 win 0 (DF) (ttl 64, id 13205, len 40)
17:26:00.978213 00:25:90:cb:21:4c 00:25:90:dd:08:59 0800 66: 10.0.128.10.15149
> 10.0.128.67.443: S [tcp sum ok]
1972661198:1972661198(0) win 8192 <mss 1460,nop,wscale 2,nop,nop,sackOK> (DF)
(ttl 128, id 19925, len 52)
17:26:00.978236 00:25:90:dd:08:59 00:25:90:cb:21:4c 0800 54: 10.0.128.67.443 >
10.0.128.10.15149: R [bad tcp cksum 1468! -> 7d37]
0:0(0) ack 1972661199 win 0 (DF) (ttl 64, id 46183, len 40)
17:26:01.229126 00:25:90:cb:21:4c 00:25:90:dd:08:59 0800 66: 10.0.128.10.15150
> 10.0.128.67.443: S [tcp sum ok]
4037716682:4037716682(0) win 8192 <mss 1460,nop,wscale 2,nop,nop,sackOK> (DF)
(ttl 128, id 19926, len 52)
17:26:01.229142 00:25:90:dd:08:59 00:25:90:cb:21:4c 0800 54: 10.0.128.67.443 >
10.0.128.10.15150: R [bad tcp cksum 1468! -> c323]
0:0(0) ack 4037716683 win 0 (DF) (ttl 64, id 21545, len 40)
17:26:01.477399 00:25:90:cb:21:4c 00:25:90:dd:08:59 0800 66: 10.0.128.10.15149
> 10.0.128.67.443: S [tcp sum ok]
1972661198:1972661198(0) win 8192 <mss 1460,nop,wscale 2,nop,nop,sackOK> (DF)
(ttl 128, id 19927, len 52)
<< ... and it keeps going on with "R [bad tcp cksum]" and "S" packets
alternating ... >>
So, based on my _very_limited_ understanding, what I see is that with s_server
running, the browser sends an syn packet, the
s_server responds with a syn/ack, and then the browser sends a ack and starts
pushing data, which the s_server answers correctly,
and a server cert and other stuff gets transmitted.
But, with httpd running, the syn, syn/ack, ack sequence occurs (so the
connection is established), but after the first push of data
from the browser to the httpd server, the server's response is somehow
malformed (I base this statement on the "bad tcp cksum" from
tcpdump in the server's reply to the first push of data. The "bad tcp cksum"
is a comment from tcpdump itself, right?).
Since tcpdump is watching the flow of data at the same interface (em0) that the
server is using, this excludes problems with errors
from transmission of the packet on the network, right?
If my "analysis" is correct, this would lead to the conclusion that httpd is
somehow mangling the https connection. But, I really
don't know.
I would be happy to forward any additional information. Should this be
considered a "bug," or am I doing something wrong?
Thanks again
