FYI, I took some time out this afternoon and wrote a detailed certificate configuration guide. Hopefully this will help.
https://review.openstack.org/613454 Reviews would be welcome! Michael On Thu, Oct 25, 2018 at 7:00 AM Tobias Urdin <tobias.ur...@binero.se> wrote: > > Might as well throw it out here. > > After a lot of troubleshooting we were able to narrow our issue down to > our test environment running qemu virtualization, we moved our compute > node to hardware and > used kvm full virtualization instead. > > We could properly reproduce the issue where generating a CSR from a > private key and then trying to verify the CSR would fail complaining about > "Signature did not match the certificate request" > > We suspect qemu floating point emulation caused this, the same OpenSSL > function that validates a CSR is the one used when validating the SSL > handshake which caused our issue. > After going through the whole stack, we have Octavia working flawlessly > without any issues at all. > > Best regards > Tobias > > On 10/23/2018 04:31 PM, Tobias Urdin wrote: > > Hello Erik, > > > > Could you specify the DNs you used for all certificates just so that I > > can rule it out on my side. > > You can redact anything sensitive with some to just get the feel on how > > it's configured. > > > > Best regards > > Tobias > > > > On 10/22/2018 04:47 PM, Erik McCormick wrote: > >> On Mon, Oct 22, 2018 at 4:23 AM Tobias Urdin <tobias.ur...@binero.se> > >> wrote: > >>> Hello, > >>> > >>> I've been having a lot of issues with SSL certificates myself, on my > >>> second trip now trying to get it working. > >>> > >>> Before I spent a lot of time walking through every line in the DevStack > >>> plugin and fixing my config options, used the generate > >>> script [1] and still it didn't work. > >>> > >>> When I got the "invalid padding" issue it was because of the DN I used > >>> for the CA and the certificate IIRC. > >>> > >>> > 19:34 < tobias-urdin> 2018-09-10 19:43:15.312 15032 WARNING > >>> octavia.amphorae.drivers.haproxy.rest_api_driver [-] Could not connect > >>> to instance. Retrying.: SSLError: ("bad handshake: Error([('rsa > >>> routines', 'RSA_padding_check_PKCS1_type_1', 'block type is not 01'), > >>> ('rsa routines', 'RSA_EAY_PUBLIC_DECRYPT', 'padding check failed'), > >>> ('SSL routines', 'ssl3_get_key_exchange', 'bad signature')],)",) > >>> > 19:47 < tobias-urdin> after a quick google "The problem was that my > >>> CA DN was the same as the certificate DN." > >>> > >>> IIRC I think that solved it, but then again I wouldn't remember fully > >>> since I've been at so many different angles by now. > >>> > >>> Here is my IRC logs history from the #openstack-lbaas channel, perhaps > >>> it can help you out > >>> http://paste.openstack.org/show/732575/ > >>> > >> Tobias, I owe you a beer. This was precisely the issue. I'm deploying > >> Octavia with kolla-ansible. It only deploys a single CA. After hacking > >> the templates and playbook to incorporate a separate server CA, the > >> amphorae now load and provision the required namespace. I'm adding a > >> kolla tag to the subject of this in hopes that someone might want to > >> take on changing this behavior in the project. Hopefully after I get > >> through Upstream Institute in Berlin I'll be able to do it myself if > >> nobody else wants to do it. > >> > >> For certificate generation, I extracted the contents of > >> octavia_certs_install.yml (which sets up the directory structure, > >> openssl.cnf, and the client CA), and octavia_certs.yml (which creates > >> the server CA and the client certificate) and mashed them into a > >> separate playbook just for this purpose. At the end I get: > >> > >> ca_01.pem - Client CA Certificate > >> ca_01.key - Client CA Key > >> ca_server_01.pem - Server CA Certificate > >> cakey.pem - Server CA Key > >> client.pem - Concatenated Client Key and Certificate > >> > >> If it would help to have the playbook, I can stick it up on github > >> with a huge "This is a hack" disclaimer on it. > >> > >>> ----- > >>> > >>> Sorry for hijacking the thread but I'm stuck as well. > >>> > >>> I've in the past tried to generate the certificates with [1] but now > >>> moved on to using the openstack-ansible way of generating them [2] > >>> with some modifications. > >>> > >>> Right now I'm just getting: Could not connect to instance. Retrying.: > >>> SSLError: [SSL: BAD_SIGNATURE] bad signature (_ssl.c:579) > >>> from the amphoras, haven't got any further but I've eliminated a lot of > >>> stuck in the middle. > >>> > >>> Tried deploying Ocatavia on Ubuntu with python3 to just make sure there > >>> wasn't an issue with CentOS and OpenSSL versions since it tends to lag > >>> behind. > >>> Checking the amphora with openssl s_client [3] it gives the same one, > >>> but the verification is successful just that I don't understand what the > >>> bad signature > >>> part is about, from browsing some OpenSSL code it seems to be related to > >>> RSA signatures somehow. > >>> > >>> 140038729774992:error:1408D07B:SSL routines:ssl3_get_key_exchange:bad > >>> signature:s3_clnt.c:2032: > >>> > >>> So I've basicly ruled out Ubuntu (openssl-1.1.0g) and CentOS > >>> (openssl-1.0.2k) being the problem, ruled out signing_digest, so I'm > >>> back to something related > >>> to the certificates or the communication between the endpoints, or what > >>> actually responds inside the amphora (gunicorn IIUC?). Based on the > >>> "verify" functions actually causing that bad signature error I would > >>> assume it's the generated certificate that the amphora presents that is > >>> causing it. > >>> > >>> I'll have to continue the troubleshooting to the inside of the amphora, > >>> I've used the test-only amphora image before but have now built my own > >>> one that is > >>> using the amphora-agent from the actual stable branch, but same issue > >>> (bad signature). > >>> > >>> For verbosity this is the config options set for the certificates in > >>> octavia.conf and which file it was copied from [4], same here, a > >>> replication of what openstack-ansible does. > >>> > >>> Appreciate any feedback or help :) > >>> > >>> Best regards > >>> Tobias > >>> > >>> [1] > >>> https://github.com/openstack/octavia/blob/master/bin/create_certificates.sh > >>> [2] http://paste.openstack.org/show/732483/ > >>> [3] http://paste.openstack.org/show/732486/ > >>> [4] http://paste.openstack.org/show/732487/ > >>> > >>> On 10/20/2018 01:53 AM, Michael Johnson wrote: > >>>> Hi Erik, > >>>> > >>>> Sorry to hear you are still having certificate issues. > >>>> > >>>> Issue #2 is probably caused by issue #1. Since we hot-plug the tenant > >>>> network for the VIP, one of the first steps after the worker connects > >>>> to the amphora agent is finishing the required configuration of the > >>>> VIP interface inside the network namespace on the amphroa. > >>>> > >> Thanks for the hint on the workflow of this. I hadn't gotten deep > >> enough into the code to find that yet, but I suspected it was blocking > >> since the namespace never got created either. Thanks > >> > >>>> If I remember correctly, you are attempting to configure Octavia with > >>>> the dual CA option (which is good for non-development use). > >>>> > >>>> This is what I have for notes: > >>>> > >>>> [certificates] gets the following: > >>>> cert_generator = local_cert_generator > >>>> ca_certificate = server CA's "server.pem" file > >>>> ca_private_key = server CA's "server.key" file > >>>> ca_private_key_passphrase = pass phrase for ca_private_key > >>>> [controller_worker] > >>>> client_ca = Client CA's ca_cert file > >>>> [haproxy_amphora] > >>>> client_cert = Client CA's client.pem file (I think with it's key > >>>> concatenated is what rm_work said the other day) > >>>> server_ca = Server CA's ca_cert file > >>>> > >> This is all very helpful. It's a bit difficult to know what goes where > >> the way the documentation is written presently. For something that's > >> going to be the defacto standard for loadbalancing, we as a community > >> need to do a better job of documenting how to set up, configure, and > >> manage this in production. I'm trying to capture my lessons learned > >> and processes as I go to help with that if I can. > >> > >> -Erik > >> > >>>> That said, I can probably run through this and write something up next > >>>> week that is more step-by-step/detailed. > >>>> > >>>> Michael > >>>> > >>>> On Fri, Oct 19, 2018 at 2:31 PM Erik McCormick > >>>> <emccorm...@cirrusseven.com> wrote: > >>>>> Apologies for cross-posting, but in the event that these might be > >>>>> worth filing as bugs, I wanted the Octavia devs to see it as well... > >>>>> > >>>>> I've been wrestling with getting Octavia up and running and have > >>>>> become stuck on two issues. I'm hoping someone has run into these > >>>>> before. My google foo has come up empty. > >>>>> > >>>>> Issue 1: > >>>>> When the Octavia controller tries to poll the amphora instance, it > >>>>> tries repeatedly and eventually fails. The error on the controller > >>>>> side is: > >>>>> > >>>>> 2018-10-19 14:17:39.181 26 ERROR > >>>>> octavia.amphorae.drivers.haproxy.rest_api_driver [-] Connection > >>>>> retries (currently set to 300) exhausted. The amphora is unavailable. > >>>>> Reason: HTTPSConnectionPool(host='10.7.0.112', port=9443): Max retries > >>>>> exceeded with url: /0.5/plug/vip/10.250.20.15 (Caused by > >>>>> SSLError(SSLError("bad handshake: Error([('rsa routines', > >>>>> 'RSA_padding_check_PKCS1_type_1', 'invalid padding'), ('rsa routines', > >>>>> 'rsa_ossl_public_decrypt', 'padding check failed'), ('asn1 encoding > >>>>> routines', 'ASN1_item_verify', 'EVP lib'), ('SSL routines', > >>>>> 'tls_process_server_certificate', 'certificate verify > >>>>> failed')],)",),)): SSLError: HTTPSConnectionPool(host='10.7.0.112', > >>>>> port=9443): Max retries exceeded with url: /0.5/plug/vip/10.250.20.15 > >>>>> (Caused by SSLError(SSLError("bad handshake: Error([('rsa routines', > >>>>> 'RSA_padding_check_PKCS1_type_1', 'invalid padding'), ('rsa routines', > >>>>> 'rsa_ossl_public_decrypt', 'padding check failed'), ('asn1 encoding > >>>>> routines', 'ASN1_item_verify', 'EVP lib'), ('SSL routines', > >>>>> 'tls_process_server_certificate', 'certificate verify > >>>>> failed')],)",),)) > >>>>> > >>>>> On the amphora side I see: > >>>>> [2018-10-19 17:52:54 +0000] [1331] [DEBUG] Error processing SSL request. > >>>>> [2018-10-19 17:52:54 +0000] [1331] [DEBUG] Invalid request from > >>>>> ip=::ffff:10.7.0.40: [SSL: SSL_HANDSHAKE_FAILURE] ssl handshake > >>>>> failure (_ssl.c:1754) > >>>>> > >>>>> I've generated certificates both with the script in the Octavia git > >>>>> repo, and with the Openstack Ansible playbook. I can see that they are > >>>>> present in /etc/octavia/certs. > >>>>> > >>>>> I'm using the Kolla (Queens) containers for the control plane so I'm > >>>>> sure I've satisfied all the python library constraints. > >>>>> > >>>>> Issue 2: > >>>>> I"m not sure how it gets configured, but the tenant network interface > >>>>> (ens6) never comes up. I can spawn other instances on that network > >>>>> with no issue, and I can see that Neutron has the port attached to the > >>>>> instance. However, in the instance this is all I get: > >>>>> > >>>>> ubuntu@amphora-33e0aab3-8bc4-4fcb-bc42-b9b36afb16d4:~$ ip a > >>>>> 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN > >>>>> group default qlen 1 > >>>>> link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 > >>>>> inet 127.0.0.1/8 scope host lo > >>>>> valid_lft forever preferred_lft forever > >>>>> inet6 ::1/128 scope host > >>>>> valid_lft forever preferred_lft forever > >>>>> 2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc pfifo_fast > >>>>> state UP group default qlen 1000 > >>>>> link/ether fa:16:3e:30:c4:60 brd ff:ff:ff:ff:ff:ff > >>>>> inet 10.7.0.112/16 brd 10.7.255.255 scope global ens3 > >>>>> valid_lft forever preferred_lft forever > >>>>> inet6 fe80::f816:3eff:fe30:c460/64 scope link > >>>>> valid_lft forever preferred_lft forever > >>>>> 3: ens6: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group > >>>>> default qlen 1000 > >>>>> link/ether fa:16:3e:89:a2:7f brd ff:ff:ff:ff:ff:ff > >>>>> > >>>>> There's no evidence of the interface anywhere else including udev rules. > >>>>> > >>>>> Any help with either or both issues would be greatly appreciated. > >>>>> > >>>>> Cheers, > >>>>> Erik > >>>>> > >>>>> __________________________________________________________________________ > >>>>> OpenStack Development Mailing List (not for usage questions) > >>>>> Unsubscribe: > >>>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > >>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >>>> __________________________________________________________________________ > >>>> OpenStack Development Mailing List (not for usage questions) > >>>> Unsubscribe: > >>>> openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >>>> > >>> __________________________________________________________________________ > >>> OpenStack Development Mailing List (not for usage questions) > >>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >> __________________________________________________________________________ > >> OpenStack Development Mailing List (not for usage questions) > >> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > >> > > > > __________________________________________________________________________ > > OpenStack Development Mailing List (not for usage questions) > > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > > > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev _______________________________________________ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
