Hi Harsha/Colin
I did the sample with a custom Provider for TrustStoreManager and tried
using ssl.provider Kafka config AND the way KIP-492 is suggesting (by
adding Provider programmatically instead of relying on
ssl.provider+java.security. The below sample is followed by my detailed
findings. I'll appreciate if you can go through it carefully and see if you
see my point.
package providertest;
import java.security.Provider;
public class MyProvider extends Provider {
private static final String name = "MyProvider";
private static double version = 1.0d;
private static String info = "Maulin's SSL Provider v"+version;
public MyProvider() {
super(name, version, info);
this.put("TrustManagerFactory.PKIX",
"providertest.MyTrustManagerFactory");
}
}
*Details:*
KIP-492 documents that it will use Security.addProvider() assuming it will
add it as position '0' which is not a correct assumption. The
addProvider()'s documentation says it will add it to the last available
position. You may want to correct that to say
Security.insertProviderAt(provider, 1).
Now coming back to our specific discussion,
1. SPIFFE example uses Custom Algorithm - spiffe. Hence when you add that
provider in the provider list via Security.addProvider() the position where
it gets added doesn't matter (even if you don't end up adding it as first
entry) since that is the ONLY provider for SPIFFE specific algorithm you
might have.
We do *not* have custom algorithm for Key/Trust StoreMangers. Which means
we have to use X509, PKIX etc "Standard Algorithms" ((
https://docs.oracle.com/javase/7/docs/technotes/guides/security/StandardNames.html))
in our provider to override the TrustStoreManager (see my sample code) and
KeyStoreManger and KeyManager. This creates another challenge mentioned in
the below point.
2. In order to make our Provider for loading custom TrustStore work, we
have to add the provider as 'first' in the list since there are others with
the same algorithm.
However, the programatic way of adding provider
(Security.insertProviderAt()) is *not* deterministic for ordering since
different code can call the method for a different provider and depending
upon the order of the call our provider can be first or pushed down the
list. This can happen very well in any client application using Kafka. This
is specially problematic for a case when you want to guarantee order for a
Provider having "Standard Algorithms".
If we add our provider in java.security file that definitely guarantees the
order(unless somebody calls removeProvider() which is less likely). But if
we add our provider in java.security file it will defeat the purpose of the
KIP-492.
In the gist - Apache Kafka must not rely on "un-deterministic" method to
rely on Provider ordering.
3. If we just use existing ssl.provider kafka configuration then our
provider will be used in SSLContext.getInstance(protocol, provider) call in
SslFactory.java and if our provider does not have implementation for
SSLContext.TLS/TLSv1.1/TLSv1.2 etc it breaks (we tested it). Example: In
MyProvider sample above you see that I didn't add SSLContext.TLSv1 as
"Service+Algorithm" and that didn't work for me. In SPIFFE provider you
don't have this challenge since you are planning to bypass ssl.provider as
you mention in the KIP-492.
*Overall summary,*
1. Any provider based mechanisms- a) existing ssl.provider and b)KIP-492,
for loading key/trust store using "Standard Algorithms" do not work
2. Approach suggested in our KIP-486 works without any issue and it is *not*
our context specific solve
3. Based on above we feel KIP-492 and KIP-486 are complimentary changes and
not contradicting or redundent.
If you want we can do a joint session somehow to walk through the sample I
have and various experiments I did. I would encourage you to do similar
exercise by writing a Provider for "Standard Algorithm" for
TrustStoreManager (like our needs) and see what you find since only writing
samples can bring out the complexity/challenges we face.
Thanks
Maulin
On Wed, Aug 14, 2019 at 11:15 PM Maulin Vasavada <maulin.vasav...@gmail.com>
wrote:
> Just to update - still working on it. Get to work only on and off on it :(
>
> On Thu, Aug 8, 2019 at 4:05 PM Maulin Vasavada <maulin.vasav...@gmail.com>
> wrote:
>
>> Hi Harsha
>>
>> Let me try to write samples and will let you know.
>>
>> Thanks
>> Maulin
>>
>> On Thu, Aug 8, 2019 at 4:00 PM Harsha Ch <harsha...@gmail.com> wrote:
>>
>>> Hi Maulin,
>>> With java security providers can be as custom you would like it
>>> to
>>> be. If you only want to to implement a custom way of loading the
>>> keystore and truststore and not implement any protocol/encryption
>>> handling
>>> you can leave them empty and no need to implement.
>>> Have you looked into the links I pasted before?
>>>
>>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.java
>>>
>>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeTrustManager.java
>>>
>>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.java
>>>
>>> Can you please tell me which methods are too complex in above to
>>> implement
>>> or unnecessary? You are changing anything in SSL/TLS implementations
>>> provided by
>>>
>>> All of the implementations delegating the checks to the default
>>> implementation anyway.
>>> Spire agent is an example, its nothing but a GRPC server listening on a
>>> unix domain socket . Above code is making a RPC call to the local daemon
>>> to
>>> get the certificate and keys. The mechanics are pretty much same as what
>>> you are asking for.
>>>
>>> Thanks,
>>> Harsha
>>>
>>> On Thu, Aug 8, 2019 at 3:47 PM Maulin Vasavada <
>>> maulin.vasav...@gmail.com>
>>> wrote:
>>>
>>> > Imagine a scenario like - We know we have a custom KMS and as a Kafka
>>> owner
>>> > we want to comply to using that KMS source to load keys/certs. As a
>>> Kafka
>>> > owner we know how to integrate with KMS but doesn't necessarily have to
>>> > know anything about cipher suites, algorithms, and SSL/TLS
>>> implementation.
>>> > Going the Provider way requires to know lot more than we should, isn't
>>> it?
>>> > Not that we would have concern/shy-away knowing those details - but if
>>> we
>>> > don't have to - why should we?
>>> >
>>> > On Thu, Aug 8, 2019 at 3:23 PM Maulin Vasavada <
>>> maulin.vasav...@gmail.com>
>>> > wrote:
>>> >
>>> > > Hi Harsha
>>> > >
>>> > > We don't have spire (or similar) agents and we do not have keys/certs
>>> > > locally on any brokers. To elaborate more on my previous email,
>>> > >
>>> > > I agree that Java security Providers are used in much broader sense
>>> - to
>>> > > have a particular implementation of an algorithm, use specific cipher
>>> > > suites for SSL , OR in our current team's case have a particular
>>> way to
>>> > > leverage pre-generated SSL sessions. However, the scope of our KIP
>>> (486)
>>> > is
>>> > > much restricted than that. We merely intend to provide a custom
>>> > > keystore/truststore for our SSL connections and not really worry
>>> about
>>> > > underlying specific SSL/TLS implementation. This simplifies it a
>>> lot for
>>> > > us to keep the concerns separate and clear.
>>> > >
>>> > > I feel our approach is more complimentary such that it allows for
>>> using
>>> > > keystores of choice while retaining the flexibility to use any
>>> > > underlying/available Provider for actually making the SSL call.
>>> > >
>>> > > We agree with KIP-492's approach based on Providers (and Java's
>>> > > recommendation), but also strongly believe that our approach can
>>> > compliment
>>> > > it very effectively for reasons explained above.
>>> > >
>>> > > Thanks
>>> > > Maulin
>>> > >
>>> > > On Thu, Aug 8, 2019 at 3:05 PM Harsha Chintalapani <ka...@harsha.io>
>>> > > wrote:
>>> > >
>>> > >> Hi Maulin,
>>> > >>
>>> > >> On Thu, Aug 08, 2019 at 2:04 PM, Maulin Vasavada <
>>> > >> maulin.vasav...@gmail.com>
>>> > >> wrote:
>>> > >>
>>> > >> > Hi Harsha
>>> > >> >
>>> > >> > The reason we rejected the SslProvider route is that - we only
>>> needed
>>> > a
>>> > >> > custom way to load keys/certs. Not touch any policy that existing
>>> > >> Providers
>>> > >> > govern like SunJSSE Provider.
>>> > >> >
>>> > >>
>>> > >> We have exactly the same requirements to load certs and keys through
>>> > spire
>>> > >> agent. We used security.provider to do that exactly. I am not sure
>>> why
>>> > you
>>> > >> would be modifying any policies provided by default SunJSSE
>>> provider.
>>> > Can
>>> > >> you give me an example of having custom provider that will override
>>> an
>>> > >> existing policy in SunJSSE provider.
>>> > >>
>>> > >> As pointed out earlier, this kip
>>> > >>
>>> > >>
>>> >
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
>>> > >> allows
>>> > >> you to load security.provider through config.
>>> > >> Take a look at the examples I gave before
>>> > >>
>>> > >>
>>> >
>>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.java
>>> > >> It registers KeyManagerFactory and TrustManagerFactory and Keystore
>>> > >> algorithm.
>>> > >> Implement your custom way of loading Keystore in here
>>> > >>
>>> > >>
>>> >
>>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore.java
>>> > >>
>>> > >> and Trust manager like here
>>> > >>
>>> > >>
>>> >
>>> https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeTrustManager.java
>>> > >>
>>> > >> In your Kafka client you can set the security.provider to your
>>> custom
>>> > >> implementation and with this fix
>>> > >> https://issues.apache.org/jira/browse/KAFKA-8191 you can set
>>> > >> keyManagerAlgorigthm and trustManagerAlgorithm configs.
>>> > >>
>>> > >> All of this is in your clients and broker side and do not need to
>>> touch
>>> > >> any
>>> > >> policy changes at JVM level. You'll register the providers in the
>>> > priority
>>> > >> order and can still have SunJSSE provider and have your custom
>>> provider
>>> > to
>>> > >> implement the key and trust managers.
>>> > >>
>>> > >>
>>> > >>
>>> > >> The ask here is different than KIP-492. We don't have any need to
>>> > >> > modify/specify the algorithm parameter. Does that make sense?
>>> > >> >
>>> > >>
>>> > >> The ask in KIP is introducing new interfaces where the KIP's
>>> > >> goal/motivation can be achieved through the security.provider and we
>>> > >> worked
>>> > >> on similar goal without touching any Keystore or Truststore
>>> interfaces.
>>> > >> My advise is against changing or introducing new interfaces when it
>>> can
>>> > >> work through security.provider.
>>> > >>
>>> > >> Thanks,
>>> > >> Harsha
>>> > >>
>>> > >>
>>> > >> Thanks
>>> > >> > Maulin
>>> > >> >
>>> > >> > On Thu, Aug 8, 2019 at 7:48 AM Harsha Chintalapani <
>>> ka...@harsha.io>
>>> > >> > wrote:
>>> > >> >
>>> > >> > In your KIP you added security. provider as rejected alternative
>>> and
>>> > >> > specified "its not the correct way". Do you mind explaining why
>>> its
>>> > >> not? I
>>> > >> > didn't find any evidence in Java docs to say so. Contrary to your
>>> > >> statement
>>> > >> > it does say in the java docs
>>> > >> > " However, please note that a provider can be used to implement
>>> any
>>> > >> > security service in Java that uses a pluggable architecture with a
>>> > >> choice
>>> > >> > of implementations that fit underneath."
>>> > >> >
>>> > >> > Java Security Providers have been used by other projects to
>>> provide
>>> > such
>>> > >> > integration . I am not sure if you looked into Spiffe project to
>>> > >> > efficiently distribute certificates but here is an example of Java
>>> > >> provider
>>> > >> >
>>> > >> > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/
>>> > >> >
>>> > >>
>>> >
>>> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider.
>>> > >> > java which
>>> > >> > obtains certificates from local daemons.
>>> > >> > These integrations are being used in Tomcat, Jetty etc.. We are
>>> also
>>> > >> using
>>> > >> > Security provider to do the same in our Kafka clusters. So unless
>>> I
>>> > see
>>> > >> > more evidence why security.provider doesn't work for you adding
>>> new
>>> > >> > interfaces while there exists more cleaner way of achieving the
>>> goals
>>> > of
>>> > >> > this KIP is unnecessary and breaks the well known security
>>> interfaces
>>> > >> > provided by Java itself.
>>> > >> >
>>> > >> > Thanks,
>>> > >> > Harsha
>>> > >> >
>>> > >> > On Thu, Aug 08, 2019 at 6:54 AM, Harsha Chintalapani <
>>> ka...@harsha.io
>>> > >
>>> > >> > wrote:
>>> > >> >
>>> > >> > Hi Maulin,
>>> > >> > Not sure if you looked at my previous replies. This
>>> > >> >
>>> > >> > changes
>>> > >> >
>>> > >> > are not required as there is already security Provider to do what
>>> you
>>> > >> are
>>> > >> > proposing. This KIP
>>> > https://cwiki.apache.org/confluence/display/KAFKA/
>>> > >> > KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config
>>> also
>>> > >> > addresses easy registration of such providers.
>>> > >> >
>>> > >> > Thanks,
>>> > >> > Harsha
>>> > >> >
>>> > >> > On Wed, Aug 07, 2019 at 11:31 PM, Maulin Vasavada
>>> > >> <maulin.vasavada@gmail.
>>> > >> > com> wrote:
>>> > >> >
>>> > >> > Bump! Can somebody please review this?
>>> > >> >
>>> > >> > On Tue, Jul 16, 2019 at 1:51 PM Maulin Vasavada <
>>> > >> >
>>> > >> > maulin.vasav...@gmail.com>
>>> > >> >
>>> > >> > wrote:
>>> > >> >
>>> > >> > Bump! Can somebody please review this?
>>> > >> >
>>> > >> >
>>> > >>
>>> > >
>>> >
>>>
>>
