Securing Kafka with zookeeper 3.5.5+ and mTLS

Hi, I was just wondering if the following article: https://docs.confluent.io/current/kafka/incremental-security-upgrade.html is still valid when using Zookeeper 3.5.5 with mTLS rather than kerberos? If it is still valid, what principle is used for the ACL? Thanks! ttyl Dima -- dbrod...@sales

Re: Securing Kafka - Keystore and Truststore question

Raghav, *My guess about the problem is that I was generate a csr (certificate signing request), which is different from actually extracting certificate. Please correct me if I am wrong.* Yes, that is correct. Use "keytool -exportcert" to extract the certificate. *To actually address our problem

Re: Securing Kafka - Keystore and Truststore question

Raghav, If you send me the full command sequence, I can take a look. Also, which JRE are you using? Regards, Rajini On Thu, May 18, 2017 at 12:19 PM, Raghav wrote: > Rajini > > I just tried this. It turns out that I can't import cert-file by itself in > trust store until it is signed by a CA.

Re: Securing Kafka - Keystore and Truststore question

Rajini I just tried this. It turns out that I can't import cert-file by itself in trust store until it is signed by a CA. Could be because of the format ? Any idea here ... In the above steps, if I sign the server-cert-file and client-cert-file by a private CA then I can add them to trust store a

Re: Securing Kafka - Keystore and Truststore question

Raghav, Perhaps what you want to do is: *You do (for the brokers):* Generate key-pair for broker: keytool -keystore kafka.server.keystore.jks -alias localhost -validity 365 -genkey Export certificate to a file to send to your customers: keytool -exportcert -file server-cert-file -keystore kaf

Re: Securing Kafka - Keystore and Truststore question

Rajini, Sure, will submit a PR shortly. Your answer is very helpful, but I think I did not put the question correctly. Pardon my ignore but I am still trying to get my ways around Kafka security. I was trying to understand, can we (Kafka Broker) just add the certificate (unsigned or signed) from

Re: Securing Kafka - Keystore and Truststore question

Raghav, Yes, you can create a truststore with your customers' certificates and vice-versa. It will be best to give your CA certificate to your customers and get the CA certificate from each of your customers and add them to your broker's truststore. You can both then create additional certificates

Re: Securing Kafka - Keystore and Truststore question

Another quick question: Say we chose to add our customer's certificates directly to our brokers trust store and vice verse, could that work ? There is no documentation on Kafka or Confluent site for this ? Thanks. On Wed, May 17, 2017 at 1:56 PM, Rajini Sivaram wrote: > Raghav, > > 1. Yes, yo

Re: Securing Kafka - Keystore and Truststore question

Raghav, 1. Yes, your customers can use certificates signed by a trusted authority. You can simply omit the truststore configuration for your broker in server.properties, and Kafka would use the default, which will trust the client certificates. If your brokers are using SSL for inter-broker commun

Re: Securing Kafka - Keystore and Truststore question

One follow up questions Rajini: 1. Can we use some other mechanism like have our customer's use a well known CA which JKS understands, and in that case we don't have to ask our customers to do this certificate-in and certificate-out thing ? I am just trying to understand if we can make our custome

Re: Securing Kafka - Keystore and Truststore question

Many thanks, Rajini. On Tue, May 16, 2017 at 8:43 AM, Rajini Sivaram wrote: > Hi Raghav, > > If your Kafka broker is configured with *ssl.client.auth=required,* your > customer's clients need to provide a keystore. In any case, they need a > truststore since your broker is using SSL. For the tru

Re: Securing Kafka - Keystore and Truststore question

Hi Raghav, If your Kafka broker is configured with *ssl.client.auth=required,* your customer's clients need to provide a keystore. In any case, they need a truststore since your broker is using SSL. For the truststore, you can given them ca-cert, as you mentioned. Client keystore contains a certif

Re: Securing Kafka - Keystore and Truststore question

Hi Rajini This was very helpful. I have another questions on similar lines. We host Kafka Broker, and we also have our own private CA. We want our customers to setup their Kafka Clients (Producer and Consumer) using SSL using *ssl.client.auth=required*. Is there a way, we can generate certificat

Re: Securing Kafka - Keystore and Truststore question

Raqhav, 1. Clients need a keystore if you are using TLS client authentication. To enable client authentication, you need to configure ssl.client.auth in server.properties. This can be set to required|requested|none. If you don't enable client authentication, any client will be able to connect to y

Securing Kafka - Keystore and Truststore question

Hi I read the documentation here: https://kafka.apache.org/documentation/#security_ssl I have few questions about trust store and keystore based on this scenario: We have 5 Kafka Brokers in our cluster. We want our clients to write to our Kafka brokers in a secure way. Suppose, we also host a pr

reporting the controller error in securing kafka cluster

HI, I'm building a safe Kafka cluster using SASL_PLAINTEXT。 It can create the topic, but not producing and consuming data。 The following is an related log : less controller.log [2016-03-08 11:43:44,164] INFO [Controller 0]: Controller starting up (kafka.controller.KafkaController) [2016-03-

Re: Securing kafka

I'm definitely interested in this too. On Fri, Aug 30, 2013 at 6:03 PM, Jay Kreps wrote: > Yeah if nobody else does it first linkedin will definitely do kerberos/ssl > + unix permissions at the topic level soonish. If folks already have a head > start on the auth piece we would love to have tha

Re: Securing kafka

Yeah if nobody else does it first linkedin will definitely do kerberos/ssl + unix permissions at the topic level soonish. If folks already have a head start on the auth piece we would love to have that contribution. On Fri, Aug 30, 2013 at 5:25 AM, Maxime Brugidou wrote: > We would love to see k

Re: Securing kafka

Please contribute that back!, Would potentially be huge for mirroring clusters across Amazon Regions, for instance. On Thu, Aug 29, 2013 at 8:22 PM, Rajasekar Elango wrote: > We have made changes to kafka code to support certificate based mutual SSL > authentication. So the clients and broker wi

Re: Securing kafka

That's sounds very interesting. Looking forward to it! On Aug 29, 2013 11:23 PM, "Rajasekar Elango" wrote: > We have made changes to kafka code to support certificate based mutual SSL > authentication. So the clients and broker will exchange trusted > certificates for successful communication. Th

Re: Securing kafka

We would love to see kerberos authentication + some unix-like permission system for topics (where one topic is a file and users/groups have read and/or write access). I guess this is not high-priority but it enables some sort of kafka-as-a-service possibility with multi tenancy. You could integrat

Re: Securing kafka

No certificates are not per topic. It is for entire broker. Thanks, Raja. On Thu, Aug 29, 2013 at 11:33 PM, Joe Stein wrote: > are the certificate stores by topic? very interesting!!! looking forward to > trying it out and review it > > /*** > Joe Stein

Re: Securing kafka

are the certificate stores by topic? very interesting!!! looking forward to trying it out and review it /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop

Re: Securing kafka

We have made changes to kafka code to support certificate based mutual SSL authentication. So the clients and broker will exchange trusted certificates for successful communication. This provides both authentication and ssl encryption. Planning to contribute that code back to kafka soon. Thanks, R

Re: Securing kafka

One use case I have been discussing recently with a few clients is verifying the digital signature of a message as part of the acceptance criteria of it being committed to the log and/or when it is consumed. I would be very interested in discussing different scenarios such as Kafka as a service, p

Re: Securing kafka

+1 We don't have any application-level security at this time so the answer is whatever you can do at the network/system level. -Jay On Thu, Aug 29, 2013 at 10:09 AM, Benjamin Black wrote: > IP filters on the hosts. > On Aug 29, 2013 10:03 AM, "Calvin Lei" wrote: > > > Is there a way to stop

RE: Securing kafka

bject: Re: Securing kafka IP filters on the hosts. On Aug 29, 2013 10:03 AM, "Calvin Lei" wrote: > Is there a way to stop a malicious user to connect directly to a kafka > broker and send any messages? Could we have the brokers to accept a > message to a list of know IPs? >

Re: Securing kafka

IP filters on the hosts. On Aug 29, 2013 10:03 AM, "Calvin Lei" wrote: > Is there a way to stop a malicious user to connect directly to a kafka > broker and send any messages? Could we have the brokers to accept a message > to a list of know IPs? >

Securing kafka

Is there a way to stop a malicious user to connect directly to a kafka broker and send any messages? Could we have the brokers to accept a message to a list of know IPs?

Re: Securing Kafka

Yes securing the content of log messages at rest is important to us - which favors message encryption. Thanks for the responses. Fergal. On Tue, Apr 23, 2013 at 7:31 PM, Chris Curtin wrote: > Also keep in mind that anything done at the transport (SSL for example) > layer won't solve your 'at re

Re: Securing Kafka

Also keep in mind that anything done at the transport (SSL for example) layer won't solve your 'at rest' problems. All messages are written to disk, so unless the broker does some encryption logic you haven't solved the data visibility issues. I also think this should be a producer/consumer probl

Re: Securing Kafka

The other issue with this model is that you're Kafka servers are available on the internet ... so anyone else can push data into them. At least, if you are running in a cross-cloud/cross-datacenter environment. We're struggling with the same design issues right now. What we've hoped for is that

Re: Securing Kafka

Yes, I think encryption at the message level is a workable solution, as long as you don't care about exposing the meta data that goes with it (e.g. topic names, kafka broker/zk server locations, etc.). Jason On Tue, Apr 23, 2013 at 10:02 AM, Fergal Somers wrote: > Hi > > We are planning to use

Securing Kafka

Hi We are planning to use Kafka, but like others on this list we have a need to be able to secure communication. The approaches people have suggested on this list are: - Encrypt the messages at the producer (e.g http://search-hadoop.com/m/1AfXKcZIk52/message+encryption&subj=Re+Secure+commu