tname:9092
ssl.trustStore.locations=/home/kafka/kafka/ssl/kafka.truststore.jks
ssl.trustStore.password=password
ssl.keyStore.location=/home/kafka/kafka/ssl/kafka.keystore.jks
ssl.keyStore.password=password
ssl.key.password=password
security.inter.broker.protocol=SASL_SSL
ssl.client.auth=required
sasl.enabled.mechan
Hi Kafka Team
Recently I moved Kafka cluster from CentOS8 to UbuntuServer20.04, same Kafka
version(2.13-3.0.0), same Kafka configuration(check below), same
JDK(openjdk-11-jdk) in server, but I get python client failed to connect.
# SASL-SSL
security.inter.broker.protocol=SASL_SSL
sasl.enabled.me
Hi ,
I have been trying to enable Kafka security ssl authentication using
certificates and encryption. but i am getting errors when i try to create a
topic and Kafka status fails whereas zookeeper is running fine.
Note :- The screenshot of the error I get when I try to create a topic is
attached
Hi Team,
we need some help regarding ca certificate authority change in kafka .
Currently we are connecting in kafka using ssl implementation.
kafka version used is 1.1.1
below is server.properties
listeners=INT://$PVT_HOST_NAME:9094,EXT://$PVT_HOST_NAME:9092
advertised.listeners=INT://$PVT_HO
Hi ,
I have been trying to enable Kafka security ssl authentication using
certificates and encryption. but i am getting errors when i try to create a
topic and Kafka status fails whereas zookeeper is running fine.
Note :- The screenshot of the error I get when I try to create a topic is
attached
t;
>> I setup Kafka and client SSL config by taking reference of
>> Apache Kafka<https://kafka.apache.org/documentation/#security_ssl>
>> Apache Kafka TLS encryption & authentication - Azure HDInsight |
>> Microsoft Docs<
>> https://docs.microsoft.com/en-us/azu
ake failed).
>
>
> I setup Kafka and client SSL config by taking reference of
> Apache Kafka<https://kafka.apache.org/documentation/#security_ssl>
> Apache Kafka TLS encryption & authentication - Azure HDInsight | Microsoft
> Docs<
> https://docs.microsoft.com/e
lt;https://kafka.apache.org/documentation/#security_ssl>
Apache Kafka TLS encryption & authentication - Azure HDInsight | Microsoft
Docs<https://docs.microsoft.com/en-us/azure/hdinsight/kafka/apache-kafka-ssl-encryption-authentication>
And I can verify my Kafka cluster SSL with below command:
op
alizer.class=com.org.KafkaJsonSerializer
>
> kafka.producer.topic.audit=Audit
>
> kafka.producer.topic.audit.test=audit-trail-test
>
> kafka.producer.topic.crl=certificate-revocation
>
> kafka.test.to.test.topic.t=Aer
>
> kafka.producer.topic.data=compacted
>
/data/test/ssl/keystore/kafka.keystore.jks
ssl.truststore.location=/
test.com/data/kafka/ssl/truststore/kafka.truststore.jks
ssl.key.password=**
ssl.keystore.password=**
ssl.truststore.password=**
security.protocol=SSL
ssl.protocol=TLS
ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
ssl.keystore
Hi team,
Any update on the below issue.
Regards,
Soumya
From: Nayak, Soumya R.
Sent: Wednesday, July 31, 2019 11:37 AM
To: users@kafka.apache.org
Subject: Kafka SSL Issue Observed
Hi team,
I am using the SSL and SASL PLAIN on the kafka brokers (cluster of 4 nodes).
The version of kafka
Hi team,
I am using the SSL and SASL PLAIN on the kafka brokers (cluster of 4 nodes).
The version of kafka - 1.0.0 . I am observing the below issue with regards to
SSL. Why this issue is happening?
Is this issue addressed in the latest versions ?
[2019-07-30 06:11:35,629] WARN Failed to send SS
If you can access the remote file via a mounted filesystem, you can specify
'/mountpoint/truststore.jks’ as the value for ssl.truststore.location. You
cannot use a url to specify a remote resource.
> On May 2, 2019, at 11:38 AM, anurag wrote:
>
> Hi All,
>
> Is it possible to set the value o
Hi All,
Is it possible to set the value of ssl.truststore.location to a location on
remote host. Basically I have ssl certificates available on remote host and
i would like my docker kafka container to read and use certificates from
remote location. If this is possible can you please provide an ex
I am trying to set up a three node Kafka v1.1 cluster with SSL. I can consume
messages via the SSL port but I cannot produce. The command is timing out after
60 seconds with the following error message:
ERROR Error when sending message to topic myTopic with key: null, value: 5
bytes with error:
t: Thursday, February 14, 2019 5:41 AM
To: users@kafka.apache.org
Subject: Kafka SSL and multiple domain names
Hello,
We need to have the same Kafka cluster bound to multiple DNS aliases/domain
names.
However, for some poor reason, we can't have a single SSL certificate with
subject alt names
Hello,
We need to have the same Kafka cluster bound to multiple DNS aliases/domain
names.
However, for some poor reason, we can't have a single SSL certificate with
subject alt names matching all DNS aliases.
Is it possible to use different SSL certs depending on the hostname used by
the client?
___
From: sham singh
Sent: Thursday, December 21, 2017 4:06 PM
To: users@kafka.apache.org
Subject: Re: Kafka SSL error
Ted - i'm not seeing any difference in the Non-w
hello,
here is the update on this ..
seems the script ->
*/usr/hdp/2.5.3.0-37/kafka/bin/kafka-producer-perf-test.sh *
has an issue in SSL mode, it seems to not be able to recognize the
security-protocol=SSL & the config file passed i.e. when the truststore,
password is passed through the config f
Ted - i'm not seeing any difference in the Non-working & working clusters ..
Another thing, seem like there is some issue with the connectivity .. the
console consumer gets disconnected
/usr/hdp/2.5.3.0-37/kafka/bin/kafka-console-consumer.sh --new-consumer
--topic mmtest1 --bootstrap-server host1
Since you're using a Vendor's distro, can you post on their community page ?
BTW do you notice any difference in settings between the working cluster
and this cluster ?
Cheers
On Thu, Dec 21, 2017 at 12:27 PM, sham singh
wrote:
> Hello All -
> I'm getting this error, when publishing messages t
Hello All -
I'm getting this error, when publishing messages to Kafka topic using SSL
mode,
Command to publish messages :
*/usr/hdp/2.5.3.0-37/kafka/bin/kafka-producer-perf-test.sh --messages
100 --message-size 1000 --topics mmtest4 \*
*--broker-list :9093,:9093,:9093, \*
*--threads 1 --compr
Thanks Jakub .. for your inputs & help in this !
I was able to get this to work last week..
On Thu, Sep 21, 2017 at 12:22 AM, Jakub Scholz wrote:
> Hi,
>
> If you want the Kafka broker to present the whole chain you have to use the
> chain when creating the PKCS12 file (use the chain instead of
Hi,
If you want the Kafka broker to present the whole chain you have to use the
chain when creating the PKCS12 file (use the chain instead of the host
certificate). As you mentioned, the chain should be in the order 1) server
cert, 2) intermediate cert and 3) root cert. It will be then automatical
Another point ..
on adding the chain.p12 (PKCS#12 format) .. order of the certs in the
keystore is as shown below
Alias : 1
Cert[1] -> server cert
Cert[2] -> Intermediate cert
Cert[3] -> Root cert
mentioning that, since one of the articles i read mentioned that the certs
imported should be in fol
Hi Jakub,
Thanks for the detailed note...
here is the update ->
I was able to convert the host.cert.pem to PKCS#12 & import the cert into
the kafka.server.keystore.jks
(also into kafka.server.truststore.jks)
wrt the host.root.pem & host.intermed.pem certs - i'm assuming i need to
convert them to
ad 1) The problem is that the signed certificate (host.cert.pem) which the
CA provides is only the public key. You have to combine it with the private
key which you created when requesting the signed certificate. The private
key is never sent to the CA so they cannot provide it back. You or whoever
Hello
- thanks for the response
Here is the update on the issue.
I'm using certs signed/provided by org-wide CA (geotrust, not a self-signed
cert)
The Signed(by the CA - geotrust) cert provided has 3 certificates
- host.chain.pem (certificate chain - contains the Root, Intermediate,
Signed Server
Hi,
Looking at your commands it looks as if you generated a self signed key for
server, self signed key for client and then imported the CA keys public
keys into the truststores. I don’t think this will work because now you
have two different self signed keys in the keystores and the presumably th
Hello All -
I was able to set up SSL for the Kafka brokers, using OpenSSL.
however, I'm having issues with setting up SSL using the pem file (i.e. SSL
certificate - certified by CA, provided by the company)
Here is what i've done -
created the server/client keystore & truststore files and importe
Hi All -
I've Kafka 0.10 .. I've enabled SSL(Non-kerberized) for Kafka Broker on
Node 4, and *i'm able to produce/consume messages using console-producer &
console-consumer from Node 4.*
However, i'm having issues enabling ssl connection between Node 4 & Node 5
& try to consume messages from Node
Hi All ,
How can I avoid using password for keystore creation ?
We are currently passing keystore password while accessing TLS enabled
Kafka instance .
I would like to use either passwordless keystore or avoid password for
clients accessing Kafka .
From: Stephane Maarek
Sent: Tuesday, December 20, 2016 7:11 PM
To: Rajini Sivaram
Cc: users@kafka.apache.org
Subject: Re: Kafka SSL encryption plus external CA
Thanks Rajini.
I used a CNAME broker-bootstrap-A.example.com t
Stephane,
I believe that should work, though I haven't tried it myself.
On Wed, Dec 21, 2016 at 12:11 AM, Stephane Maarek <
steph...@simplemachines.com.au> wrote:
> Thanks Rajini.
>
> I used a CNAME broker-bootstrap-A.example.com that round robins to the
> actual brokers broker-1.example.com, br
Thanks Rajini.
I used a CNAME broker-bootstrap-A.example.com that round robins to the
actual brokers broker-1.example.com, broker-2.example.com (etc etc).
Therefore no brokers advertises the bootstrap DNS name we’re using. Is that
an issue? The SSL certificate wildcard will match both boostrap CNA
Stephane,
Bootstrap brokers are also verified by the client in exactly the same way,
so they should also match the wildcard of their certificate. Basically,
clients need to make a secure SSL connection to one of the bootstrap
brokers to obtain advertised hostnames of brokers, so they need to compl
Thanks Rajini!
Also, I currently have each broker advertising as broker1.mydomain.com,
broker2.mydomain.com broker6.mydomain.com etc…
I have setup CNAME with round robin fashion to group brokers by
availability zone i.e. broker-a.mydomain.com broker-b.mydomain.com
broker-c.mydomain.com. I use them
Stephane,
If you are using a trusted CA like Verisign, clients don't need to specify
a truststore. The host names specified in advertised.listeners in the
broker must match the wildcard DNS names in the certificates if clients
configure ssl.endpoint.identification.algorithm=https. If
ssl.endpoint.
Hi,
I have read the docs extensively but yet there are a few answers I can’t
find. It has to do with external CA
Please confirm my understanding if possible:
I can create my own CA to sign all the brokers and clients certificates.
Pros:
- cheap, easy, automated. I need to find a way to access tha
mon.security.plain.PlainLoginModule required
>>
>> username="someuser"
>>
>> user_kafka="somePassword"
>>
>> password="kafka-password";
>>
>> };
>>
>>
>> The fact that I can no longer even consume f
PLAINTEXT
> (which is a regression of where I was before we started trying to add SSL)
> tells me there is something wrong in either server.properties or jaas.conf.
> I've checked the Kafka broker logs (server.log) each time I try connecting
> and this is the only line that gets
_______
From: Rajini Sivaram
Sent: Monday, November 21, 2016 11:03:14 AM
To: users@kafka.apache.org
Subject: Re: Can Kafka/SSL be terminated at a load balancer?
Rule #1 and Rule #2 cannot co-exist. You are basically configuring your LB
to point to a Kafka broker and you are poin
ing was: does that exception
> (ClosedChannelException) indicate bad configs on the Kafka broker?
>
>
> From: Zac Harvey
> Sent: Thursday, November 17, 2016 4:44:06 PM
> To: users@kafka.apache.org
> Subject: Can Kafka/SSL be terminated at a load bal
n) indicate bad configs on the Kafka broker?
From: Zac Harvey
Sent: Thursday, November 17, 2016 4:44:06 PM
To: users@kafka.apache.org
Subject: Can Kafka/SSL be terminated at a load balancer?
We have two Kafka nodes and for reasons outside of this question, would li
ow
I could troubleshoot it?
Thanks again!
Best,
Zac
From: Rajini Sivaram
Sent: Monday, November 21, 2016 10:11:00 AM
To: users@kafka.apache.org
Subject: Re: Can Kafka/SSL be terminated at a load balancer?
A load balancer that balances the load across the brokers wouldn&#
rtin Gainty
> wrote:
>
> >
> >
> >
> >
> >
> > From: Zac Harvey
> > Sent: Monday, November 21, 2016 8:59 AM
> > To: users@kafka.apache.org
> > Subject: Re: Can Kafka/SSL be terminated at a load balancer?
> >
> > Thanks again Raji
gain, just still a little uncertain about the traffic/ports coming
> into the load balancer!
>
>
> Best,
>
> Zac
>
>
> From: Rajini Sivaram
> Sent: Monday, November 21, 2016 8:48:41 AM
> To: users@kafka.apache.org
> Subject: Re:
wrote:
>
>
>
>
>
> From: Zac Harvey
> Sent: Monday, November 21, 2016 8:59 AM
> To: users@kafka.apache.org
> Subject: Re: Can Kafka/SSL be terminated at a load balancer?
>
> Thanks again Rajini,
>
>
> Using these configs,
From: Zac Harvey
Sent: Monday, November 21, 2016 8:59 AM
To: users@kafka.apache.org
Subject: Re: Can Kafka/SSL be terminated at a load balancer?
Thanks again Rajini,
Using these configs, would clients connect to the load balancer over SSL/9093?
And then
balancer!
Best,
Zac
From: Rajini Sivaram
Sent: Monday, November 21, 2016 8:48:41 AM
To: users@kafka.apache.org
Subject: Re: Can Kafka/SSL be terminated at a load balancer?
Zac,
Yes, that is correct. Ruby clients will not be authenticated by Kafka. They
talk SSL to
ct?
>
>
> Thanks again for all the great help so far, you've already helped me more
> than you know!
>
>
> Zac
>
>
> From: Rajini Sivaram
> Sent: Monday, November 21, 2016 3:53:47 AM
> To: users@kafka.apache.org
>
ed to authenticate, correct?
Thanks again for all the great help so far, you've already helped me more than
you know!
Zac
From: Rajini Sivaram
Sent: Monday, November 21, 2016 3:53:47 AM
To: users@kafka.apache.org
Subject: Re: Can Kafka/SSL be terminated
fka01.example.com:9092
>
> Thanks again!
> Zac
>
>
>
>
>
>
> From: Rajini Sivaram
> Sent: Friday, November 18, 2016 9:57:22 AM
> To: users@kafka.apache.org
> Subject: Re: Can Kafka/SSL be terminated at a load balancer?
&
ram
Sent: Friday, November 18, 2016 9:57:22 AM
To: users@kafka.apache.org
Subject: Re: Can Kafka/SSL be terminated at a load balancer?
You should set advertised.listeners rather than the older
advertised.host.name property in server.properties:
- listeners=SSL://:9093
- advertised.l
er client-side configs that will need to be made for the Ruby
> clients to connect over SSL?
>
>
> Thank you enormously here!
>
>
> Best,
>
> Zac
>
>
>
> From: Rajini Sivaram
> Sent: Friday, November 18, 2016 5:15:13 AM
> To:
rmously here!
Best,
Zac
From: Rajini Sivaram
Sent: Friday, November 18, 2016 5:15:13 AM
To: users@kafka.apache.org
Subject: Re: Can Kafka/SSL be terminated at a load balancer?
Zac,
Kafka has its own built-in load-balancing mechanism based on partition
assignment. Requests are processed by parti
Zac,
Kafka has its own built-in load-balancing mechanism based on partition
assignment. Requests are processed by partition leaders, distributing load
across the brokers in the cluster. If you want to put a proxy like HAProxy
with SSL termination in front of your brokers for added security, you ca
We have two Kafka nodes and for reasons outside of this question, would like to
set up a load balancer to terminate SSL with producers (clients). The SSL cert
hosted by the load balancer will be signed by trusted/root CA that clients
should natively trust.
Is this possible to do, or does Kafka
Aha , got it. So thats where I got confused.
> On Feb 1, 2016, at 3:04 PM, Ismael Juma wrote:
>
> Hi Nazario,
>
> The problem in the original post is that you were setting
> advertised.host.name, which means that advertised.listeners won't fall back
> to listeners anymore. Yes, it's bit con
Hi Nazario,
The problem in the original post is that you were setting
advertised.host.name, which means that advertised.listeners won't fall back
to listeners anymore. Yes, it's bit confusing given how the configs
evolved over time.
I have configured several clusters to use SSL by setting listen
On Mon, Feb 1, 2016 at 7:15 PM, Nazario Parsacala
wrote:
> So it looks like you need both listeners and advertised.listeners ..?
>
No, you always need to set `listeners` (`advertised.listeners` defaults to
`listeners`). If you want `advertised.listeners` to be different than
`listeners`, then yo
I dont think that is the behavior I have seen. If I set listeners only ( as
per my original post) , SSL will never get registered.
[2016-02-01 11:27:49,712] INFO Registered broker 0 at path /brokers/ids/0 with
addresses: PLAINTEXT -> EndPoint(servername,9092,PLAINTEXT)
(kafka.utils.ZkUtils)
So it looks like you need both listeners and advertised.listeners ..?
When I set both configs .. It finally worked.
Maybe we can update the docs ..?
> On Feb 1, 2016, at 1:59 PM, Nazario Parsacala wrote:
>
> So I made the port 9092 but SSL. But it seems like it is just openning it for
> PL
So I made the port 9092 but SSL. But it seems like it is just openning it for
PLAINTEXT. Even though it has registered it as SSL
[2016-02-01 13:42:20,536] INFO Registered broker 0 at path /brokers/ids/0 with
addresses: SSL -> EndPoint(reactor.us.cixsoft.net,9092,SSL)
(kafka.utils.ZkUtils)
ope
Hmm. So I removed port 9092 and just use port 9093. So no PLAINTEXT just SSL
advertised.listeners=SSL://reactor.us.cixsoft.net:9093
Cleared Zookeeper and Kafka store and restart ..
You see that it is registering 9093 onbly
[2016-02-01 13:35:51,729] INFO Registered broker 0 at path /brokers/ids/0
Ok, This is getting interesting .. On the broker side, it is saying that it is
registering 9092 as PLAINTEXT and 9093 as SSL
[2016-02-01 13:26:33,796] INFO Registered broker 0 at path /brokers/ids/0 with
addresses: PLAINTEXT -> EndPoint(servername,9092,PLAINTEXT),SSL ->
EndPoint(servername,909
No juice.
/kafka-topics.sh --describe --topic anotherone --zookeeper localhost:2181
Topic:anotheronePartitionCount:4ReplicationFactor:1 Configs:
Topic: anotherone Partition: 0Leader: 0 Replicas: 0
Isr: 0
Topic: anotherone Partition: 1
Hello Nazario,
Could you try it by creating a new topic?
Thank you,
Anirudh
That works. At least it is saying that it is registering now with the SSL
side.
[2016-02-01 12:29:40,184] INFO Registered broker 0 at path /brokers/ids/0
with addresses: PLAINTEXT -> EndPoint(servername,9092,PLAINTEXT),
That works. At least it is saying that it is registering now with the SSL side.
[2016-02-01 12:29:40,184] INFO Registered broker 0 at path /brokers/ids/0 with
addresses: PLAINTEXT -> EndPoint(servername,9092,PLAINTEXT),SSL ->
EndPoint(servername,9093,SSL) (kafka.utils.ZkUtils)
Thank you.
Now
Please use advertised.listeners instead of advertised.host.name. See this
comment:
https://github.com/apache/kafka/pull/793#issuecomment-174287124
Ismael
On Mon, Feb 1, 2016 at 4:44 PM, Nazario Parsacala
wrote:
> Hi,
>
> We were using kafka for a while now. We have been using the binary releas
Hi,
We were using kafka for a while now. We have been using the binary release
2.10-0.8.2.1 . But we have been needing a encrypted communication between our
publishers and subscribers. So we got 2.10-0.9.0.0. This works very well with
no SSL enabled. But currently have issues with SSL enabled.
71 matches
Mail list logo
