Hi Erik,
Thanks for reaching out for assistance on this topic. I’ll try to answer your
questions in order.
> Short Version:
> When running the NiFi toolkit ../bin/tls-toolkit.sh server, how do I get the
> server to include an additional public certificate of authority in the
> returned truststore.jks file (../bin/tls-toolkit.sh client calls)
The quick answer is that there is currently a feature availability difference
between the standalone mode and the client/server mode of the TLS Toolkit. The
behavior you are requesting is currently in the standalone mode but not in the
client/server mode. This is due to a difference in their implementation logic,
which is being addressed as part of a larger refactoring under NIFI-5462 [1],
which is a component of the TLS Toolkit epic NIFI-5458 [2].
If you need that functionality out of the box right now, using the standalone
mode with the --additionalCACertificate flag [3] will allow you to provide
further CA certificates into the signing process.
However, there is a current workaround for the client/server mode which may be
acceptable to you. Rather than manually signing each issued node certificate
with your enterprise CA, you can sign the NiFi CA certificate with the
enterprise CA (following the same CSR steps you already do), and then
subsequent signing operations using the toolkit should propagate this
information via the certificate chain. There are explicit instructions for
performing these actions in the Admin Guide under "Using An Existing
Intermediate Certificate Authority (CA)” [4] and "Signing with
Externally-signed CA Certificates” [5]. I’ve also included example output below
showing a keystore containing a private key entry with 2 certificates
demonstrating the Root CA signing an Intermediate CA, which signed the node
certificate.
Your further questions are regarding the configuration of shared trust with the
NiFi Registry instance. I’ll offer some quick background in response to your
last statements and then try to address each component directly.
The NiFi TLS Toolkit was designed to facilitate the deployment and
configuration of TLS behavior _in an environment lacking an enterprise security
team_. By design, the toolkit assumes a completely isolated environment where
an individual admin was deploying NiFi as a proof of concept, development
system. The toolkit is _not_ designed to replace an enterprise security team,
and in environments where this expertise and these services are available, that
process should be followed. I am sorry if the documentation feels lacking to
you. These are very complicated processes, and it’s written to be targeted to
an “advanced beginner” — someone who understands the general concepts but isn’t
familiar with every openssl setting, and benefits from some level of
abstraction. With that in mind, we still tried to make the steps clear for the
common use cases we expect users to encounter. If you have suggestions for
improving the documentation (specific steps that were unclear, etc.) we welcome
those.
In response to the specific issue you are encountering, I am surprised this is
occurring. The toolkit is designed to issue certificates properly signed with
the CA, so any “node” should be trusted automatically.
I understand this seems pedantic, but definitions and terminology are critical
in security, so I want to clarify that your command to issue a certificate for
NiFi Registry is not issuing a _client_ certificate. It is issuing a server
certificate which will be used in a server role when the Registry is hosting a
web service (API/UI) and external entities connect to it (Registry Client in
NiFi, a user via browser, the NiFi CLI toolkit, etc.). The command you’re using
to issue a cert for the CLI is doing the same thing. The instructions on
configuring security for the CLI [6] assume you are using an actual _client_
certificate, for example one issued for a human user which has been loaded into
the browser to allow authenticated access. The differences in issuing a
_client_ certificate include the command-line flags for the toolkit (-C), the
key usages and roles assigned, the output format (JKS vs PKCS12), etc.
I am sorry the PKIX error message is unclear. The error is a Java-native
exception and we can make an effort to capture it and provide a more
user-friendly message in the future. However, searching "PKIX path building
failed nifi” [7] does return a number of helpful articles and Q/A posts with
instructions to resolve the issue.
Your statement about needing to run a meddler-in-the-middle is confusing to me.
It appears that the issued truststore does not contain the specific CA entry
you need in order to form a complete certificate chain and authenticate the
service. A MitM won’t resolve this, and the commands you included don’t run a
MitM; rather you make a TLS connection to the server using openssl s_client and
retrieve the server’s public certificate, then import that into the truststore.
This process makes sense but should be unnecessary. Can you share the command
you use to generate the truststore initially, and the state of the truststore
upon generation? Any truststore generated by the (same instance of the) toolkit
should contain the public certificate of the CA, which is what signed all node
certificates in the environment, so they should all be cross-trusted. Needing
to import the public certificate of an external CA goes back to the first part
of your email. The local/NiFi CA certificate should be present by default. The
capability to capture the external CA is already recorded in NIFI-5460 [8].
I’ve marked the issue you reported as a duplicate of that. Please provide any
feedback if you feel this is incorrect.
>> This is a mess of undocumented black magic.
>>
>> I dont expect the public root CA to change.
>>
>> Is there a way to simplify this process? Cant we ignore this whole chain
>> thing? Maybe include the public CA cert in the nifi-ca-keystore.jks on the
>> server running the
>> ../bin/tls-toolkit.sh server
>>
>> This voodoo process really is too much guys and needs to be simplified.
>> Suggestions are welcome.
Again, I apologize that the documentation wasn’t helpful to you during this
time. Your statement on not expecting the public root CA to change feels
specific to your environment. Other enterprise organizations rotate that
certificate on regular intervals and on any kind of compromise, so building in
static preset values is not something we are prepared to do. Similarly,
automatically importing the JRE cacerts default truststore would provide
out-of-the-box compatibility with many publicly accessible commercial CA
services, but this would invisibly expose many additional risks, as a user
could be expecting to send data to an internal service, signed only by an
internal CA, and accidentally misconfigure the processor/reporting task to
validate a certificate signed by any arbitrary provider on the internet,
leading to massive, invisible data leakage. We try to align to the principle of
least surprise, and requiring users in a secured environment to perform some
manual (while automatable) tasks is preferable to exposing these risks out of
the box to all users.
Many users of Apache NiFi are deploying to large enterprise environments using
the currently available tools. While we do try to simplify this process for all
users of NiFi, if your needs are such that the current tooling is insufficient
to manage this and your existing corporate procedures do not align with what is
available, you may be well served in investigating commercial vendor support.
I encourage you to share further thoughts on concrete steps to improve this
process that do not open users up to unseen security risks, and to follow the
Jiras discussed above for further developments in improving the behavior and
ease-of-use of the TLS Toolkit.
------
Example keytool output for keystore with external CA public certificate in
chain:
...resto/Workspace/scratch/certs/externalCA (master) 😉
🔒 3s @ 19:01:57 $ keytool -list -v -keystore
intermediate_signed.nifi.apache.org/keystore.jks
Enter keystore password:
Keystore type: jks
Keystore provider: SUN
Your keystore contains 1 entry
Alias name: nifi-key
Creation date: Aug 2, 2018
Entry type: PrivateKeyEntry
Certificate chain length: 2
Certificate[1]:
Owner: CN=intermediate_signed.nifi.apache.org, OU=NIFI
Issuer: CN=Intermediate CA
Serial number: 164fd79074700000000
Valid from: Thu Aug 02 18:48:44 PDT 2018 until: Sun Aug 01 18:48:44 PDT 2021
Certificate fingerprints:
MD5: BA:0F:14:50:71:7A:DD:D3:10:CD:DA:B8:09:2A:6D:FE
SHA1: EE:D6:76:25:68:64:45:ED:74:21:9C:20:7B:05:13:C0:7A:BB:A3:E9
SHA256:
C2:EA:C4:EC:B4:2D:13:E6:86:0D:E7:4D:48:B8:98:E1:18:2B:AB:3D:F1:0E:AD:58:CE:3D:AC:C1:83:31:9F:2B
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 2048-bit RSA key
Version: 3
Extensions:
...
#5: ObjectId: 2.5.29.17 Criticality=false
SubjectAlternativeName [
DNSName: intermediate_signed.nifi.apache.org
]
...
Certificate[2]:
Owner: CN=Intermediate CA
Issuer: CN=Root CA
Serial number: 164fd68928600000000
Valid from: Thu Aug 02 18:30:46 PDT 2018 until: Fri Aug 03 18:30:46 PDT 2018
Certificate fingerprints:
MD5: 85:70:4E:B4:CB:FC:E7:89:DE:13:36:6E:B0:A0:74:51
SHA1: 09:6B:94:51:DA:BD:75:6E:35:EC:E4:5A:ED:6B:D0:BA:A4:69:58:82
SHA256:
D7:67:4F:3A:4F:40:7B:D2:1A:E5:A2:84:39:19:15:73:84:13:43:E2:46:3E:2A:52:2E:74:06:65:B6:79:CE:84
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 2048-bit RSA key
Version: 3
...
*******************************************
[1] https://issues.apache.org/jira/browse/NIFI-5462
[2] https://issues.apache.org/jira/browse/NIFI-5458
<https://issues.apache.org/jira/browse/NIFI-5458>
[3]
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#standalone
<https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#standalone>
[4]
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#using-an-existing-intermediate-certificate-authority-ca
[5]
https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#signing-with-externally-signed-ca-certificates
<https://nifi.apache.org/docs/nifi-docs/html/administration-guide.html#signing-with-externally-signed-ca-certificates>
[6]
https://github.com/apache/nifi/tree/master/nifi-toolkit/nifi-toolkit-cli#security-configuration
<https://github.com/apache/nifi/tree/master/nifi-toolkit/nifi-toolkit-cli#security-configuration>
[7] https://www.google.com/search?q=PKIX+path+building+failed+nifi
[8] https://issues.apache.org/jira/browse/NIFI-5460
<https://issues.apache.org/jira/browse/NIFI-5460>
Andy LoPresto
[email protected]
[email protected]
PGP Fingerprint: 70EC B3E5 98A6 5A3F D3C4 BACE 3C6E F65B 2F7D EF69
> On Feb 8, 2019, at 8:56 AM, Erik Anderson <[email protected]> wrote:
>
> I am not sure if this is a NiFi dev or NiFi user question. Seems more dev
> based
>
> Short Version:
> When running the NiFi toolkit ../bin/tls-toolkit.sh server, how do I get the
> server to include an additional public certificate of authority in the
> returned truststore.jks file (../bin/tls-toolkit.sh client calls)
>
> Long version:
> When running tls-toolkit.sh as the CA for NiFi instances
> ../bin/tls-toolkit.sh server -D "CN=nifi-ca.blah.blah.bloomberg.com, OU=NIFI"
> -t mysupersecrettoken-p 8443 -d 1826
>
> And I call the CA to request a
> /bb/nifi-toolkit/nifi-toolkit-1.7.1/bin/tls-toolkit.sh client -c
> nifi-ca.blah.com -t mysupersecrettoken-p 8443 -D
> "CN=nifi-myproject.blah.blah.bloomberg.com, OU=NIFI, O=Bloomberg L.P., L=New
> York, ST=New York, C=US"
>
> The toolkit gives me a keystore.jks
>
> I need to get this certificate signed by an external entity so I dont have
> issues with the browsers.
>
> Basically I do
> keytool -certreq -alias nifi-key -file csr.txt -keystore keystore.jks
>
> I the the certificate signing request, csr.txt, I feed it to the public
> certificate of authority, I get a signed certificate back.
>
> I convert the format of the certificate like so
> openssl x509 -outform der -in new_cert.pem -out new_cert.crt
>
> Now I import the certificate back into my keystore.
> keytool -import -alias nifi-key -file new_cert.crt -keystore keystore.jks
>
> Now I have solved the certificate warning errors in the browser when I
> connect to the NiFi UI.
>
> ------------
>
> Now I want to use the NiFi cli to to automate data flows between the
> registry and the NiFi instances so I generate a client cert
> 2) ../bin/tls-toolkit.sh client -c nifi-ca.blah.blah.bloomberg.com -t
> mysupersecrettoken-p 8443 -D "CN=nifi-cli, OU=NIFI" -T PKCS12
>
> I setup nifi.prop, I also add "CN=nifi-cli, OU=NIFI" as a user to the NiFi UI
>
> Now I call the NiFi instane as so
> ../bin/cli.sh nifi pg-list -verbose -p
> /bb/nifi-toolkit/nifi-toolkit-1.7.1/nifi.prop
>
> And I get this error
> org.apache.nifi.toolkit.cli.api.CommandException: Error executing command
> 'pg-list' : sun.security.validator.ValidatorException: PKIX path building
> failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to
> find valid certification path to requested target
>
> ***
> BTW, this error message and it gives you little to go on. Really sucks. Its
> not covered in any mailing list, topics, howto, nothing. This is a massive
> issue at an enterprise company.
>
> Regardless I figure this out.
> ***
>
> To fix this problem, anytime I call the
> ../bin/tls-toolkit.sh client
>
> I need to install my own man-in-the-middle and extract the root certificate
> like so
>
> openssl s_client -connect nifi-myproject.blah.blah.bloomberg.com:443
> -showcerts
>
> Extract the root CA certificate
>
> post modify the truststore.jks file to include the root CA certificate that
> signed the .csr
>
> cat > /bb/nifi-toolkit/nifi-toolkit-1.7.1/certs/digicert.crt <<EOF
> -----BEGIN CERTIFICATE-----
> MIIElDCCA3ygAwIBAgIQAf2j627KdciIQ4tyS8+8kTANBgkqhkiG9w0BAQsFADBh
> MQswCQYDVQQGEwJVUzEVMBMGA1UEChMMRGlnaUNlcnQgSW5jMRkwFwYDVQQLExB3
> d3cuZGlnaWNlcnQuY29tMSAwHgYDVQQDExdEaWdpQ2VydCBHbG9iYWwgUm9vdCBD
> QTAeFw0xMzAzMDgxMjAwMDBaFw0yMzAzMDgxMjAwMDBaME0xCzAJBgNVBAYTAlVT
> MRUwEwYDVQQKEwxEaWdpQ2VydCBJbmMxJzAlBgNVBAMTHkRpZ2lDZXJ0IFNIQTIg
> U2VjdXJlIFNlcnZlciBDQTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
> ANyuWJBNwcQwFZA1W248ghX1LFy949v/cUP6ZCWA1O4Yok3wZtAKc24RmDYXZK83
> nf36QYSvx6+M/hpzTc8zl5CilodTgyu5pnVILR1WN3vaMTIa16yrBvSqXUu3R0bd
> KpPDkC55gIDvEwRqFDu1m5K+wgdlTvza/P96rtxcflUxDOg5B6TXvi/TC2rSsd9f
> /ld0Uzs1gN2ujkSYs58O09rg1/RrKatEp0tYhG2SS4HD2nOLEpdIkARFdRrdNzGX
> kujNVA075ME/OV4uuPNcfhCOhkEAjUVmR7ChZc6gqikJTvOX6+guqw9ypzAO+sf0
> /RR3w6RbKFfCs/mC/bdFWJsCAwEAAaOCAVowggFWMBIGA1UdEwEB/wQIMAYBAf8C
> AQAwDgYDVR0PAQH/BAQDAgGGMDQGCCsGAQUFBwEBBCgwJjAkBggrBgEFBQcwAYYY
> aHR0cDovL29jc3AuZGlnaWNlcnQuY29tMHsGA1UdHwR0MHIwN6A1oDOGMWh0dHA6
> Ly9jcmwzLmRpZ2ljZXJ0LmNvbS9EaWdpQ2VydEdsb2JhbFJvb3RDQS5jcmwwN6A1
> oDOGMWh0dHA6Ly9jcmw0LmRpZ2ljZXJ0LmNvbS9EaWdpQ2VydEdsb2JhbFJvb3RD
> QS5jcmwwPQYDVR0gBDYwNDAyBgRVHSAAMCowKAYIKwYBBQUHAgEWHGh0dHBzOi8v
> d3d3LmRpZ2ljZXJ0LmNvbS9DUFMwHQYDVR0OBBYEFA+AYRyCMWHVLyjnjUY4tCzh
> xtniMB8GA1UdIwQYMBaAFAPeUDVW0Uy7ZvCj4hsbw5eyPdFVMA0GCSqGSIb3DQEB
> CwUAA4IBAQAjPt9L0jFCpbZ+QlwaRMxp0Wi0XUvgBCFsS+JtzLHgl4+mUwnNqipl
> 5TlPHoOlblyYoiQm5vuh7ZPHLgLGTUq/sELfeNqzqPlt/yGFUzZgTHbO7Djc1lGA
> 8MXW5dRNJ2Srm8c+cftIl7gzbckTB+6WohsYFfZcTEDts8Ls/3HB40f/1LkAtDdC
> 2iDJ6m6K7hQGrn2iWZiIqBtvLfTyyRRfJs8sjX7tN8Cp1Tm5gr8ZDOo0rwAhaPit
> c+LJMto4JQtV05od8GiG7S5BNO98pVAdvzr508EIDObtHopYJeS4d60tbvVS3bR0
> j6tJLp07kzQoH3jOlOrHvdPJbRzeXDLz
> -----END CERTIFICATE-----
> EOF
>
> keytool -import -v -trustcacerts -alias digicert-ca -file digicert.crt
> -keystore truststore.jks
>
>
> Now, finally,
>
> ../bin/cli.sh nifi pg-list -verbose -p
> /bb/nifi-toolkit/nifi-toolkit-1.7.1/nifi.prop
>
> Works! Now I am happy.
>
> This is a mess of undocumented black magic.
>
> I dont expect the public root CA to change.
>
> Is there a way to simplify this process? Cant we ignore this whole chain
> thing? Maybe include the public CA cert in the nifi-ca-keystore.jks on the
> server running the
> ../bin/tls-toolkit.sh server
>
> This voodoo process really is too much guys and needs to be simplified.
> Suggestions are welcome.
>
> Erik Anderson
> Bloomberg
