I thought about it more. I feel that no matter how we refactor the code (with or without KIP-383 integrated), ultimately the need of customizing loading for keys and certs will still remain. Whenever that need arises we might end up thinking about the solution suggested by our KIP-486. Hence regardless of the other KIPs and configurations "if we do need to customize loading of keys/certs, we will need the code changes suggested by this KIP".
Let me know what you guys think. Harsha, we are working on changing the interfaces for key/trust store loaders with Certificate and PrivateKey objects. Will probably be able to update it later today or tomorrow. Thanks Maulin On Thu, Aug 29, 2019 at 2:30 PM Maulin Vasavada <maulin.vasav...@gmail.com> wrote: > On that, I actually looked at KIP-383 before briefly. However, that > sounded like lot of changes suggested. > > One "key" thing we have to keep in mind is - IF we need lot of > customization Kafka already allows you to use your SslProvider via > ssl.providers or the changes done by KIP-492 and > SSLContext.getInstance(protocol, provider) call allows us to return the > SSLContext with "ALL" the details we would like to customize. Hence I am > not sure that customization suggested by KIP-383 would be worth the effort. > We also have similar SSLContext customization outside of Kafka. > > Thanks > Maulin > > > > > > On Thu, Aug 29, 2019 at 12:47 PM Pellerin, Clement < > clement_pelle...@ibi.com> wrote: > >> KIP-383 in its present form was vetoed because it was not possible to add >> validation of custom properties in a future KIP. The solution to that is >> the first proposal I wrote for KIP-383 which made the whole SslFactory >> pluggable. That first solution was also vetoed hence the deadlock. >> >> Replacing the whole factory was a much nicer solution. It was vetoed >> because doing this almost invariably meant the replacement lost all the >> complex validation code in the default SslFactory. >> >> My current idea is to extract the validation code into another public API >> that SslFactory would call. I did not look at the newly refactored code and >> I did not study how to do this yet. KIP-383 was not popular at the time and >> designing a new solution is a lot of work. >> >> Is there interest from 3 binding voters for something like this? >> >> -----Original Message----- >> From: Rajini Sivaram [mailto:rajinisiva...@gmail.com] >> Sent: Thursday, August 29, 2019 2:57 PM >> To: dev >> Subject: Re: [DISCUSS] KIP-486 Support for pluggable KeyStore and >> TrustStore >> >> Hi Maulin, >> >> In SSL scenarios, I imagine security providers introduced by KIP-492 are >> likely to be most useful when you want to use third party providers. The >> biggest advantage of the config from that KIP is that you don't need to >> write much code to integrate existing security providers into Kafka >> brokers >> or clients. As I understand it, KIP-486 is a more convenient option for >> the >> specific problem of loading keystores/truststores differently. It can be >> achieved in theory with KIP-492, but KIP-486 is a much simpler option for >> this case. >> >> My concern about KIP-486 is that it introduces yet another interface into >> our already complex security code, while only solving one particular use >> case. Have you looked at >> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-383%3A++Pluggable+interface+for+SSL+Factory >> ? >> The goal was to make >> org.apache.kafka.common.security.ssl.SslEngineBuilder pluggable. >> The code has already been refactored by Colin after that KIP was written, >> making it easier to implement KIP-383. This should enable you to load your >> keystores and truststores differently. Using a pluggable SslEngineBuilder >> will also solve several other use cases at the same time. KIP-383 hasn't >> been voted through yet, but perhaps you could take a look and we could >> revive that instead if it solves your use case as well? >> >> Regards, >> >> Rajini >> >> >> On Thu, Aug 29, 2019 at 6:42 PM Maulin Vasavada < >> maulin.vasav...@gmail.com> >> wrote: >> >> > Hi Harsha >> > >> > Thank you. Appreciate your time and support on this. Let me go back do >> some >> > more research and get back to you on the KeyStore interface part. >> > Basically, if we return certs and keys in the interface then Kafka code >> > will have to build KeyStore object - which is also reasonable. >> > >> > Thanks >> > Maulin >> > >> > On Thu, Aug 29, 2019 at 10:01 AM Harsha Chintalapani <ka...@harsha.io> >> > wrote: >> > >> > > Hi Maulin, >> > > Use cases are clear now. I am +1 for moving >> forward >> > > with the discussions on having such configurable option for users. But >> > the >> > > interfaces is proposed doesn't look right to me. We are still talking >> > about >> > > keystore interfaces. Given keystore's are used as filebased way of >> > > transporting certificates I am not sure it will help the rest of the >> > > user-base. >> > > In short, I am +1 on the KIP's motivation and only >> have >> > > questions around returning keystores instead of returning certs, >> private >> > > keys etc. . If others in the community are ok with such interface we >> can >> > > move forward. >> > > >> > > Thanks, >> > > Harsha >> > > >> > > >> > > On Wed, Aug 28, 2019 at 1:51 PM, Maulin Vasavada < >> > > maulin.vasav...@gmail.com> >> > > wrote: >> > > >> > > > Hi Harsha >> > > > >> > > > As we synced-up offline on this topic, we hope you don't have any >> more >> > > > clarifications that you are seeking. If that is the case, can you >> > please >> > > > help us move this forward and discuss what changes you would expect >> on >> > > the >> > > > KIP design in order to make it valuable contribution? >> > > > >> > > > Just FYI - we verified our primary design change with the author of >> > Sun's >> > > > X509 Trustmanager implementation and the outcome is that what we are >> > > > proposing makes sense at the heart of it - "Instead of writing >> > > TrustManager >> > > > just plugin the Trust store". We are open to discuss additional >> changes >> > > > that you/anybody else would like to see on the functionality >> however. >> > > > >> > > > Thanks >> > > > Maulin >> > > > >> > > > On Thu, Aug 22, 2019 at 9:12 PM Maulin Vasavada < >> > > maulin.vasav...@gmail.com> >> > > > wrote: >> > > > >> > > > Hi Harsha >> > > > >> > > > Any response on my question? I feel this KIP is worth accommodating. >> > Your >> > > > help is much appreciated. >> > > > >> > > > Thanks >> > > > Maulin >> > > > >> > > > On Tue, Aug 20, 2019 at 11:52 PM Maulin Vasavada < >> > maulin.vasavada@gmail. >> > > > com> wrote: >> > > > >> > > > Hi Harsha >> > > > >> > > > I've examined the SPIFFE provider more and have one question - >> > > > >> > > > If SPIFFE didn't have a need to do checkSpiffeId() call at the below >> > > > location, would you really still write the Provider? *OR* Would you >> > just >> > > > use TrustManagerFactory.init(KeyStore) signature to pass the >> KeyStore >> > > from >> > > > set of certs returned by spiffeIdManager. getTrustedCerts()? >> > > > >> > > > >> > https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/ >> > > > provider/CertificateUtils.java#L100 >> > > > >> > > > /** >> > > > >> > > > * Validates that the SPIFFE ID is present and matches the SPIFFE ID >> > > > configured in >> > > > * the java.security property ssl.spiffe.accept >> > > > * >> > > > * If the authorized spiffe ids list is empty any spiffe id is >> > authorized >> > > > * >> > > > * @param chain an array of X509Certificate that contains the Peer's >> > SVID >> > > > to be validated >> > > > * @throws CertificateException when either the certificates doesn't >> > have >> > > a >> > > > SPIFFE ID or the SPIFFE ID is not authorized >> > > > */ >> > > > static void checkSpiffeId(X509Certificate[] chain) throws >> > > > CertificateException { >> > > > >> > > > Thanks >> > > > Maulin >> > > > >> > > > On Tue, Aug 20, 2019 at 4:49 PM Harsha Chintalapani < >> ka...@harsha.io> >> > > > wrote: >> > > > >> > > > Maulin, >> > > > The code parts you are pointing are specific for Spiffe and if >> > > > you are talking about validate method which uses PKIX check like any >> > > other >> > > > provider does. >> > > > If you want to default to SunJSSE everywhere you can do so by >> > delegating >> > > > the calls in these methods to SunJSSE provider. >> > > > >> > > > TrustManagerFactory tmf = TrustManagerFactory >> > > > .getInstance(TrustManagerFactory.getDefaultAlgorithm());and use >> > > > tmf.chekServerTrusted() >> > > > or use >> > > > https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/ >> > > > TrustManagerFactory.html#getInstance(java.lang.String)if you want a >> > > > specific provider. >> > > > >> > > > -Harsha >> > > > >> > > > On Tue, Aug 20, 2019 at 4:26 PM, Maulin Vasavada < >> > maulin.vasavada@gmail. >> > > > com> >> > > > wrote: >> > > > >> > > > Okay, so I take that you guys agree that I have to write a 'custom' >> > > > algorithm and a provider to make it work , correct? >> > > > >> > > > Now, for Harsha's comment "Here the 'Custom' Algorithm is not an >> > > > implementation per say , ..." , I diagree. You can refer to https:// >> > > > >> > > > >> > >> github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/provider/ >> > > > >> > > > SpiffeTrustManager.java#L91 <http://spiffetrustmanager.java/#L91> >> and >> > > > >> > > > >> > https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/ >> > > > >> > > > provider/CertificateUtils.java#L100 >> > > > >> > > > "that code" is the customization you have for the custom way to >> check >> > > > something on top of regular checks. That method is NOT doing custom >> > > > truststore loading. It is validating/verifying something in the >> > > > >> > > > "custom" >> > > > >> > > > way with spiffeId. >> > > > I bet that without that you won't have a need of the custom >> algorithm >> > > > >> > > > in >> > > > >> > > > the first place. >> > > > >> > > > Let me know if you agree to this. >> > > > >> > > > Thanks >> > > > Maulin >> > > > >> > > > On Tue, Aug 20, 2019 at 2:08 PM Sandeep Mopuri <mpr...@gmail.com> >> > > > >> > > > wrote: >> > > > >> > > > Hi Maulin, thanks for the discussion. As Harsha pointed out, to use >> the >> > > > KIP492, you need to create a new provider, register a *new* custom >> > > > algorithm for your keymanager and trustmanager factory >> implementations. >> > > > After this, the kafka server configuration can be done as given >> below >> > > > >> > > > # Register the provider class with custom algorithm, say CUSTOM >> > > > >> > > > security. >> > > > >> > > > provider.classes=com.company.security.CustomProvider >> > > > <http://provider.classes=com.company.security.customprovider/> >> > > > <http://security.provider.classes >> > > > >> > > > =com.company.security.customprovider/> >> > > > >> > > > # Register the keymanager and trustmanager algorithms >> > > > # These algorithms indicate that the Keymanager and Trustmanagers >> > > > registered under the algorithm "CUSTOM" needs to be used >> > > > ssl.trustmanager.algorithm=CUSTOM >> > > > ssl.keymanager.algorithm=CUSTOM >> > > > >> > > > And the customprovider looks like this... >> > > > >> > > > public class CustomProvider extends Provider { >> > > > public CustomProvider() { >> > > > super("NEW_CUSTOM_PROVIDER", 0.1, "Custom KeyStore and TrustStore"); >> > > > super.put("KeyManagerFactory.CUSTOM", "customKeyManagerFactory"); >> > > > super.put("TrustManagerFactory.CUSTOM", >> > > > "customTrustManagerFactory"); >> > > > } >> > > > } >> > > > >> > > > The PR for this is in review and can be found here: >> > > > >> > > > https://github.com/ >> > > > >> > > > apache/kafka/pull/7090 >> > > > This PR includes the fixed insertProviderAt function call. >> > > > >> > > > On Tue, Aug 20, 2019 at 9:56 AM Harsha Chintalapani < >> ka...@harsha.io> >> > > > wrote: >> > > > >> > > > Answers are in-line >> > > > >> > > > On Mon, Aug 19, 2019 at 10:45 PM, Maulin Vasavada < >> > > > >> > > > maulin.vasavada@gmail. >> > > > >> > > > com >> > > > >> > > > wrote: >> > > > >> > > > Hi Colin >> > > > >> > > > When I refer to "standard" or "custom" algorithms I am following >> Java >> > > > security Provider Terminology. You can refer to >> > > > https://docs.oracle.com/javase/7/docs/technotes/guides/security/ >> > > > StandardNames.html#TrustManagerFactory link I provided earlier in >> the >> > > > emails. It says PKIX is the default Algorithm for >> TrustManagerFactory. >> > > > >> > > > 1. For SPIFFE, I am not sure why you are saying 'it does not >> implement >> > > > custom algorithms' because the following file clearly indicates >> that it >> > > > does use custom algorithm- >> > > > >> > > > >> > https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/ >> > > > >> > > > provider/SpiffeProvider.java#L17 >> > > > >> > > > Algorithm value: >> > > > >> > > > >> > https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/ >> > > > >> > > > provider/SpiffeProviderConstants.java#L6 >> > > > >> > > > @Harsha do you want to chime in since you use that provider? >> > > > >> > > > Here the "Custom" Algorithm is not an implementation per say , >> rather >> > > > >> > > > used >> > > > >> > > > to invoke the custom trust store factory and key manager factory. >> You >> > > > >> > > > are >> > > > >> > > > not moving away from "standard" alogrithms that are available. >> > > > >> > > > >> > https://github.com/spiffe/java-spiffe/blob/master/src/main/java/spiffe/ >> > > > >> > > > provider/SpiffeTrustManager.java >> > > > >> > > > As you can see it delegates all the calls of verification of >> > > > >> > > > certificate >> > > > >> > > > to >> > > > >> > > > the default implementation available. >> > > > So in our implementation we still use PKIX to verify the certificate >> > > > chain. So you are not losing anything here and Spiffe is not >> > > > >> > > > reimplementing >> > > > >> > > > the verification process. >> > > > >> > > > 2. I already mentioned in my 3rd point, in my previous post, why >> using >> > > > >> > > > ssl.provider does NOT work. I updated KIP-486 in "rejected >> > > > >> > > > alternatives" >> > > > >> > > > also why ssl.provider does not work. >> > > > >> > > > As mentioned before , provider is the right way to go. I am still >> not >> > > > understanding the gap is. >> > > > If I understand correctly your argument is , provider is going to >> ask >> > > > >> > > > to >> > > > >> > > > implement a custom algorithm. >> > > > My answer to that is , no you are not re-implementing the algorithm. >> > > > >> > > > Please >> > > > >> > > > check the above link , TrustManager implementation it delegates the >> > > > >> > > > calls >> > > > >> > > > back. There is no need to implement your own here. >> > > > >> > > > 3. Security.insertProviderAt() comments were based on assumption if >> > > > >> > > > KIP-492 >> > > > >> > > > changes are done and we use that mechanism to configure providers >> > > > >> > > > instead >> > > > >> > > > of ssl.provider configuration. >> > > > >> > > > KIP-492 has patch available and going through review. >> > > > >> > > > Can you read my all the points, I mentioned in my previous post, >> very >> > > > >> > > > carefully? I am covering all the aspects in explaining. I am open to >> > > > >> > > > still >> > > > >> > > > discuss more to clarify any doubts. >> > > > >> > > > "3. If we just use existing ssl.provider kafka configuration then >> our >> > > > provider will be used in SSLContext.getInstance(protocol, provider) >> > > > >> > > > call >> > > > >> > > > in >> > > > >> > > > SslFactory.java <http://sslfactory.java/> <http://sslfactory.java/> >> < >> > > > http://sslfactory.java/> >> > > > >> > > > and >> > > > >> > > > if our provider does not have >> > > > implementation for SSLContext.TLS/TLSv1.1/TLSv1.2 etc it breaks (we >> > > > >> > > > tested >> > > > >> > > > it). Example: In MyProvider sample above you see that I didn't add >> > > > SSLContext.TLSv1 as >> > > > "Service+Algorithm" and that didn't work for me. In SPIFFE provider >> you >> > > > don't have this challenge since you are planning to bypass >> ssl.provider >> > > > >> > > > as >> > > > >> > > > you mention in the KIP-492." >> > > > >> > > > Yes here you need to pass the protocol that your >> > > > >> > > > KeyManager/TrustManager >> > > > >> > > > registered with and in no way its deviating from TLS RFC spec. >> > > > >> > > > >> > https://github.com/srisatish/openjdk/blob/master/jdk/src/share/classes/ >> > > > >> > > > javax/net/ssl/SSLContext.java#L134 >> > > > >> > > > My suggestion here is for you to implement a simple Security >> Provider >> > > > >> > > > as >> > > > >> > > > you did before and register a Algorithm. You can use the existing >> > > > implementation in SpiffeProvider and plug in changes where you need >> to >> > > > retrieve the certificates from by making RPC call. >> > > > >> > > > Run an end-to-end test with Kafka broker coming up with your >> provider >> > > > making calls to RPC call. You do need to pass the "custom algorithm" >> > > > >> > > > that >> > > > >> > > > you registered in your key manager to invoke the provider. I think >> your >> > > > concern is this approach is replacing the existing known >> ciphersuites >> > > > >> > > > and >> > > > >> > > > certificate validation provided by java. Which its not. >> > > > >> > > > Now test the TLS connection you can do so via openssl -s_client >> options >> > > > >> > > > to >> > > > >> > > > test if everything else is working. >> > > > >> > > > I am happy to share configs that we used if you are interested. >> > > > >> > > > Thanks, >> > > > Harsha >> > > > >> > > > Thanks >> > > > Maulin >> > > > >> > > > On Mon, Aug 19, 2019 at 9:52 AM Colin McCabe <cmcc...@apache.org> >> > > > >> > > > wrote: >> > > > >> > > > Hi Maulin, >> > > > >> > > > A lot of JSSE providers don't implement custom algorithms. Spire is >> a >> > > > >> > > > good >> > > > >> > > > example of a JSSE provider that doesn't, and yet is still useful to >> > > > >> > > > many >> > > > >> > > > people. Your JSSE provider can work fine even if it doesn't >> implement a >> > > > custom algorithm. >> > > > >> > > > Maybe I'm missing something, but I don't understand the discussion >> of >> > > > Security.insertProviderAt() that you included. SslEngineBuilder >> doesn't >> > > > >> > > > use >> > > > >> > > > that API to get the security provider. Instead, it calls >> > > > "SSLContext.getInstance(protocol, provider)", where provider is the >> > > > >> > > > name >> > > > >> > > > of the provider. >> > > > >> > > > best, >> > > > Colin >> > > > >> > > > On Sat, Aug 17, 2019, at 20:13, Maulin Vasavada wrote: >> > > > >> > > > On top of everything above I feel strongly to add the 4th point >> which >> > > > >> > > > is >> > > > >> > > > based on Java APIs for TrustManagerFactory.init(KeyStore) ( >> > > > >> > > > https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/ >> > > > TrustManagerFactory.html#init(java.security.KeyStore >> > > > <http://java.security.keystore/> >> > > > <http://java.security.keystore/>) >> > > > ) >> > > > >> > > > and KeyManagerFactory.init(KeyStore, char[]) ( >> > > > >> > > > >> > > >> > >> https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/KeyManagerFactory >> > > > . >> > > > >> > > > html#init(java.security.KeyStore <http://java.security.keystore/> >> > > <http:// >> > > > java.security.keystore/ >> > > > ,%20char[]) >> > > > >> > > > ). >> > > > >> > > > 4. The above APIs are intended to support providing "trust/key >> > > > >> > > > material" >> > > > >> > > > from the user without having to write their own >> > > > >> > > > TrustManager/KeyManagers. >> > > > >> > > > To quote from the TrustManagerFactory.init()'s documentation >> > > > >> > > > "Initializes >> > > > >> > > > this factory with a source of certificate authorities and related >> trust >> > > > material." >> > > > To quote from the KeyManagerFactory.init()'s documentation >> "Initializes >> > > > this factory with a source of key material." >> > > > >> > > > Based on this it is clear that there is a flexibility provided by >> Java >> > > > >> > > > to >> > > > >> > > > to enable developers to provide the required trust/key material >> loaded >> > > > >> > > > from >> > > > >> > > > "anywhere" without requiring them to write custom provider OR >> trust/key >> > > > managers. This same flexibility is reflected in Kafka code also >> where >> > > > >> > > > it >> > > > >> > > > loads the trust/keys from a local file and doesn't require writing a >> > > > Provider necessarily. If we do NOT have a custom algorithm, it makes >> > > > >> > > > less >> > > > >> > > > sense to write a Provider. >> > > > >> > > > Thanks >> > > > Maulin >> > > > >> > > > On Thu, Aug 15, 2019 at 11:45 PM Maulin Vasavada < >> > > > >> > > > maulin.vasav...@gmail.com> >> > > > >> > > > wrote: >> > > > >> > > > Hi Harsha/Colin >> > > > >> > > > I did the sample with a custom Provider for TrustStoreManager and >> tried >> > > > using ssl.provider Kafka config AND the way KIP-492 is suggesting >> (by >> > > > adding Provider programmatically instead of relying on >> > > > >> > > > ssl.provider+java. >> > > > >> > > > security. The below sample is followed by my detailed findings. I'll >> > > > appreciate if you can go through it carefully and see >> > > > >> > > > if you >> > > > >> > > > see my point. >> > > > >> > > > package providertest; >> > > > >> > > > import java.security.Provider <http://java.security.provider/> >> > <http:// >> > > > java.security.provider/> >> > > > >> > > > <http:// >> > > > >> > > > java.security.provider/>; >> > > > >> > > > public class MyProvider extends Provider { >> > > > >> > > > private static final String name = "MyProvider"; private static >> double >> > > > version = 1.0d; >> > > > private static String info = "Maulin's SSL Provider v"+version; >> > > > >> > > > public MyProvider() { >> > > > super(name, version, info); >> > > > this.put("TrustManagerFactory.PKIX", >> > > > >> > > > "providertest.MyTrustManagerFactory"); >> > > > >> > > > } >> > > > } >> > > > >> > > > *Details:* >> > > > >> > > > KIP-492 documents that it will use Security.addProvider() assuming >> it >> > > > >> > > > will >> > > > >> > > > add it as position '0' which is not a correct assumption. The >> > > > addProvider()'s documentation says it will add it to the last >> available >> > > > position. You may want to correct that to say >> > > > Security.insertProviderAt(provider, 1). >> > > > >> > > > Now coming back to our specific discussion, >> > > > >> > > > 1. SPIFFE example uses Custom Algorithm - spiffe. Hence when you add >> > > > >> > > > that >> > > > >> > > > provider in the provider list via Security.addProvider() the >> position >> > > > >> > > > where >> > > > >> > > > it gets added doesn't matter (even if you don't end up adding it as >> > > > >> > > > first >> > > > >> > > > entry) since that is the ONLY provider for SPIFFE specific algorithm >> > > > >> > > > you >> > > > >> > > > might have. >> > > > >> > > > We do *not* have custom algorithm for Key/Trust StoreMangers. Which >> > > > >> > > > means >> > > > >> > > > we have to use X509, PKIX etc "Standard Algorithms" (( >> > > > >> > > > https://docs.oracle.com/javase/7/docs/technotes/guides/security/ >> > > > StandardNames.html >> > > > )) >> > > > >> > > > in our provider to override the TrustStoreManager (see my sample >> code) >> > > > >> > > > and >> > > > >> > > > KeyStoreManger and KeyManager. This creates another challenge >> > > > >> > > > mentioned in >> > > > >> > > > the below point. >> > > > >> > > > 2. In order to make our Provider for loading custom TrustStore >> work, we >> > > > have to add the provider as 'first' in the list since there are >> others >> > > > >> > > > with >> > > > >> > > > the same algorithm. >> > > > >> > > > However, the programatic way of adding provider >> > > > (Security.insertProviderAt()) is *not* deterministic for ordering >> since >> > > > different code can call the method for a different provider and >> > > > >> > > > depending >> > > > >> > > > upon the order of the call our provider can be first or pushed down >> the >> > > > list. This can happen very well in any client application using >> Kafka. >> > > > >> > > > This >> > > > >> > > > is specially problematic for a case when you want to guarantee order >> > > > >> > > > for a >> > > > >> > > > Provider having "Standard Algorithms". >> > > > >> > > > If we add our provider in java.security file that definitely >> guarantees >> > > > the order(unless somebody calls removeProvider() which is less >> > > > >> > > > likely). But >> > > > >> > > > if we add our provider in java.security file it will defeat the >> > > > >> > > > purpose of >> > > > >> > > > the KIP-492. >> > > > >> > > > In the gist - Apache Kafka must not rely on "un-deterministic" >> method >> > > > >> > > > to >> > > > >> > > > rely on Provider ordering. >> > > > >> > > > 3. If we just use existing ssl.provider kafka configuration then our >> > > > provider will be used in SSLContext.getInstance(protocol, provider) >> > > > >> > > > call in >> > > > >> > > > SslFactory.java <http://sslfactory.java/> <http://sslfactory.java/> >> < >> > > > http://sslfactory.java/> >> > > > >> > > > and >> > > > >> > > > if our provider does not have implementation for >> > > > SSLContext.TLS/TLSv1.1/TLSv1.2 etc it breaks >> > > > >> > > > (we >> > > > >> > > > tested it). Example: >> > > > >> > > > In >> > > > >> > > > MyProvider sample above you see that I didn't add SSLContext.TLSv1 >> as >> > > > "Service+Algorithm" and that didn't work for me. In SPIFFE provider >> you >> > > > don't have this challenge since you are planning to bypass >> > > > >> > > > ssl.provider as >> > > > >> > > > you mention in the KIP-492. >> > > > >> > > > *Overall summary,* >> > > > >> > > > 1. Any provider based mechanisms- a) existing ssl.provider and >> > > > >> > > > b)KIP-492, >> > > > >> > > > for loading key/trust store using "Standard Algorithms" do not work >> > > > >> > > > 2. Approach suggested in our KIP-486 works without any issue and it >> is >> > > > *not* our context specific solve >> > > > >> > > > 3. Based on above we feel KIP-492 and KIP-486 are complimentary >> changes >> > > > and not contradicting or redundent. >> > > > >> > > > If you want we can do a joint session somehow to walk through the >> > > > >> > > > sample I >> > > > >> > > > have and various experiments I did. I would encourage you to do >> similar >> > > > exercise by writing a Provider for "Standard Algorithm" for >> > > > TrustStoreManager (like our needs) and see what you find since only >> > > > >> > > > writing >> > > > >> > > > samples can bring out the complexity/challenges we face. >> > > > >> > > > Thanks >> > > > Maulin >> > > > >> > > > On Wed, Aug 14, 2019 at 11:15 PM Maulin Vasavada < >> > > > >> > > > maulin.vasavada@gmail. >> > > > >> > > > com> wrote: >> > > > >> > > > Just to update - still working on it. Get to work only on and off on >> > > > >> > > > it :( >> > > > >> > > > On Thu, Aug 8, 2019 at 4:05 PM Maulin Vasavada < >> > > > >> > > > maulin.vasav...@gmail.com> >> > > > >> > > > wrote: >> > > > >> > > > Hi Harsha >> > > > >> > > > Let me try to write samples and will let you know. >> > > > >> > > > Thanks >> > > > Maulin >> > > > >> > > > On Thu, Aug 8, 2019 at 4:00 PM Harsha Ch <harsha...@gmail.com> >> > > > >> > > > wrote: >> > > > >> > > > Hi Maulin, >> > > > With java security providers can be as custom you would >> > > > >> > > > like >> > > > >> > > > it to >> > > > be. If you only want to to implement a custom way of loading the >> > > > >> > > > keystore >> > > > >> > > > and truststore and not implement any protocol/encryption handling >> you >> > > > >> > > > can >> > > > >> > > > leave them empty and no need to implement. Have you looked into the >> > > > >> > > > links I >> > > > >> > > > pasted before? >> > > > >> > > > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/ >> > > > >> > > > >> > > >> > >> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore. >> > > > >> > > > java >> > > > >> > > > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/ >> > > > spiffe-security-provider/src/main/java/spiffe/api/provider/ >> > > > SpiffeTrustManager.java <http://spiffetrustmanager.java/> <http:// >> > > > spiffetrustmanager.java/> >> > > > >> > > > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/ >> > > > >> > > > >> > > >> > >> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider. >> > > > >> > > > java >> > > > >> > > > Can you please tell me which methods are too complex in above to >> > > > >> > > > implement >> > > > >> > > > or unnecessary? You are changing anything in SSL/TLS implementations >> > > > provided by >> > > > >> > > > All of the implementations delegating the checks to the default >> > > > implementation anyway. >> > > > Spire agent is an example, its nothing but a GRPC server listening >> > > > >> > > > on a >> > > > >> > > > unix domain socket . Above code is making a RPC call to the local >> > > > >> > > > daemon >> > > > >> > > > to >> > > > get the certificate and keys. The mechanics are pretty much same as >> > > > >> > > > what >> > > > >> > > > you are asking for. >> > > > >> > > > Thanks, >> > > > Harsha >> > > > >> > > > On Thu, Aug 8, 2019 at 3:47 PM Maulin Vasavada < >> > > > >> > > > maulin.vasav...@gmail.com> >> > > > >> > > > wrote: >> > > > >> > > > Imagine a scenario like - We know we have a custom KMS and as a >> > > > >> > > > Kafka >> > > > >> > > > owner >> > > > >> > > > we want to comply to using that KMS source to load keys/certs. As >> > > > >> > > > a >> > > > >> > > > Kafka >> > > > >> > > > owner we know how to integrate with KMS but doesn't necessarily >> > > > >> > > > have >> > > > >> > > > to >> > > > >> > > > know anything about cipher suites, algorithms, and SSL/TLS >> > > > >> > > > implementation. >> > > > >> > > > Going the Provider way requires to know lot more than we should, >> > > > >> > > > isn't it? >> > > > >> > > > Not that we would have concern/shy-away knowing those details - >> > > > >> > > > but >> > > > >> > > > if we >> > > > >> > > > don't have to - why should we? >> > > > >> > > > On Thu, Aug 8, 2019 at 3:23 PM Maulin Vasavada < >> > > > >> > > > maulin.vasav...@gmail.com> >> > > > >> > > > wrote: >> > > > >> > > > Hi Harsha >> > > > >> > > > We don't have spire (or similar) agents and we do not have >> > > > >> > > > keys/certs >> > > > >> > > > locally on any brokers. To elaborate more on my previous email, >> > > > >> > > > I agree that Java security Providers are used in much broader >> > > > >> > > > sense >> > > > >> > > > - to >> > > > >> > > > have a particular implementation of an algorithm, use specific >> > > > >> > > > cipher >> > > > >> > > > suites for SSL , OR in our current team's case have a >> > > > >> > > > particular >> > > > >> > > > way to >> > > > >> > > > leverage pre-generated SSL sessions. However, the scope of our >> > > > >> > > > KIP >> > > > >> > > > (486) >> > > > >> > > > is >> > > > >> > > > much restricted than that. We merely intend to provide a custom >> > > > keystore/truststore for our SSL connections and not really worry >> > > > >> > > > about >> > > > >> > > > underlying specific SSL/TLS implementation. This simplifies it >> > > > >> > > > a >> > > > >> > > > lot for >> > > > >> > > > us to keep the concerns separate and clear. >> > > > >> > > > I feel our approach is more complimentary such that it allows >> > > > >> > > > for >> > > > >> > > > using >> > > > >> > > > keystores of choice while retaining the flexibility to use any >> > > > underlying/available Provider for actually making the SSL call. >> > > > >> > > > We agree with KIP-492's approach based on Providers (and Java's >> > > > recommendation), but also strongly believe that our approach can >> > > > >> > > > compliment >> > > > >> > > > it very effectively for reasons explained above. >> > > > >> > > > Thanks >> > > > Maulin >> > > > >> > > > On Thu, Aug 8, 2019 at 3:05 PM Harsha Chintalapani < >> > > > >> > > > ka...@harsha.io >> > > > >> > > > wrote: >> > > > >> > > > Hi Maulin, >> > > > >> > > > On Thu, Aug 08, 2019 at 2:04 PM, Maulin Vasavada < >> > > > >> > > > maulin.vasavada@gmail. >> > > > >> > > > com> >> > > > wrote: >> > > > >> > > > Hi Harsha >> > > > >> > > > The reason we rejected the SslProvider route is that - we >> > > > >> > > > only >> > > > >> > > > needed >> > > > >> > > > a >> > > > >> > > > custom way to load keys/certs. Not touch any policy that >> > > > >> > > > existing >> > > > >> > > > Providers >> > > > >> > > > govern like SunJSSE Provider. >> > > > >> > > > We have exactly the same requirements to load certs and keys >> > > > >> > > > through >> > > > >> > > > spire >> > > > >> > > > agent. We used security.provider to do that exactly. I am not >> > > > >> > > > sure >> > > > >> > > > why >> > > > >> > > > you >> > > > >> > > > would be modifying any policies provided by default SunJSSE >> > > > >> > > > provider. >> > > > >> > > > Can >> > > > >> > > > you give me an example of having custom provider that will >> > > > >> > > > override an >> > > > >> > > > existing policy in SunJSSE provider. >> > > > >> > > > As pointed out earlier, this kip >> > > > >> > > > https://cwiki.apache.org/confluence/display/KAFKA/ >> > > > KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config >> > > > >> > > > allows >> > > > you to load security.provider through config. >> > > > Take a look at the examples I gave before >> > > > >> > > > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/ >> > > > >> > > > >> > > >> > >> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider. >> > > > >> > > > java >> > > > >> > > > It registers KeyManagerFactory and TrustManagerFactory and >> > > > >> > > > Keystore >> > > > >> > > > algorithm. >> > > > Implement your custom way of loading Keystore in here >> > > > >> > > > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/ >> > > > >> > > > >> > > >> > >> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeKeyStore. >> > > > >> > > > java >> > > > >> > > > and Trust manager like here >> > > > >> > > > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/ >> > > > spiffe-security-provider/src/main/java/spiffe/api/provider/ >> > > > SpiffeTrustManager.java <http://spiffetrustmanager.java/> <http:// >> > > > spiffetrustmanager.java/> >> > > > >> > > > In your Kafka client you can set the security.provider to your >> > > > >> > > > custom >> > > > >> > > > implementation and with this fix >> > > > https://issues.apache.org/jira/browse/KAFKA-8191 you can set >> > > > keyManagerAlgorigthm and trustManagerAlgorithm configs. >> > > > >> > > > All of this is in your clients and broker side and do not need >> > > > >> > > > to >> > > > >> > > > touch >> > > > >> > > > any >> > > > policy changes at JVM level. You'll register the providers in >> > > > >> > > > the >> > > > >> > > > priority >> > > > >> > > > order and can still have SunJSSE provider and have your custom >> > > > >> > > > provider >> > > > >> > > > to >> > > > >> > > > implement the key and trust managers. >> > > > >> > > > The ask here is different than KIP-492. We don't have any need >> > > > >> > > > to >> > > > >> > > > modify/specify the algorithm parameter. Does that make sense? >> > > > >> > > > The ask in KIP is introducing new interfaces where the KIP's >> > > > goal/motivation can be achieved through the security.provider >> > > > >> > > > and >> > > > >> > > > we >> > > > >> > > > worked >> > > > on similar goal without touching any Keystore or Truststore >> > > > >> > > > interfaces. >> > > > >> > > > My advise is against changing or introducing new interfaces >> > > > >> > > > when >> > > > >> > > > it can >> > > > >> > > > work through security.provider. >> > > > >> > > > Thanks, >> > > > Harsha >> > > > >> > > > Thanks >> > > > >> > > > Maulin >> > > > >> > > > On Thu, Aug 8, 2019 at 7:48 AM Harsha Chintalapani < >> > > > >> > > > ka...@harsha.io> >> > > > >> > > > wrote: >> > > > >> > > > In your KIP you added security. provider as rejected >> > > > >> > > > alternative >> > > > >> > > > and >> > > > >> > > > specified "its not the correct way". Do you mind explaining >> > > > >> > > > why >> > > > >> > > > its >> > > > >> > > > not? I >> > > > >> > > > didn't find any evidence in Java docs to say so. Contrary to >> > > > >> > > > your >> > > > >> > > > statement >> > > > >> > > > it does say in the java docs >> > > > " However, please note that a provider can be used to >> > > > >> > > > implement >> > > > >> > > > any >> > > > >> > > > security service in Java that uses a pluggable architecture >> > > > >> > > > with >> > > > >> > > > a >> > > > >> > > > choice >> > > > >> > > > of implementations that fit underneath." >> > > > >> > > > Java Security Providers have been used by other projects to >> > > > >> > > > provide >> > > > >> > > > such >> > > > >> > > > integration . I am not sure if you looked into Spiffe >> > > > >> > > > project to >> > > > >> > > > efficiently distribute certificates but here is an example of >> > > > >> > > > Java >> > > > >> > > > provider >> > > > >> > > > https://github.com/spiffe/spiffe-example/blob/master/java-spiffe/ >> > > > >> > > > >> > > >> > >> spiffe-security-provider/src/main/java/spiffe/api/provider/SpiffeProvider. >> > > > >> > > > java which >> > > > obtains certificates from local daemons. >> > > > These integrations are being used in Tomcat, Jetty etc.. We >> > > > >> > > > are >> > > > >> > > > also >> > > > >> > > > using >> > > > >> > > > Security provider to do the same in our Kafka clusters. So >> > > > >> > > > unless I >> > > > >> > > > see >> > > > >> > > > more evidence why security.provider doesn't work for you >> > > > >> > > > adding >> > > > >> > > > new >> > > > >> > > > interfaces while there exists more cleaner way of achieving >> > > > >> > > > the >> > > > >> > > > goals >> > > > >> > > > of >> > > > >> > > > this KIP is unnecessary and breaks the well known security >> > > > >> > > > interfaces >> > > > >> > > > provided by Java itself. >> > > > >> > > > Thanks, >> > > > Harsha >> > > > >> > > > On Thu, Aug 08, 2019 at 6:54 AM, Harsha Chintalapani < >> > > > >> > > > ka...@harsha.io >> > > > >> > > > wrote: >> > > > >> > > > Hi Maulin, >> > > > Not sure if you looked at my previous replies. This >> > > > >> > > > changes >> > > > >> > > > are not required as there is already security Provider to do >> > > > >> > > > what you >> > > > >> > > > are >> > > > >> > > > proposing. This KIP >> > > > >> > > > https://cwiki.apache.org/confluence/display/KAFKA/ >> > > > >> > > > KIP-492%3A+Add+java+security+providers+in+Kafka+Security+config >> > > > >> > > > also >> > > > >> > > > addresses easy registration of such providers. >> > > > >> > > > Thanks, >> > > > Harsha >> > > > >> > > > On Wed, Aug 07, 2019 at 11:31 PM, Maulin Vasavada >> > > > >> > > > <maulin.vasavada@gmail. >> > > > >> > > > com> wrote: >> > > > >> > > > Bump! Can somebody please review this? >> > > > >> > > > On Tue, Jul 16, 2019 at 1:51 PM Maulin Vasavada < >> > > > >> > > > maulin.vasav...@gmail.com> >> > > > >> > > > wrote: >> > > > >> > > > Bump! Can somebody please review this? >> > > > >> > > > -- >> > > > Thanks, >> > > > M.Sai Sandeep >> > > > >> > > > >> > > >> > >> >
