if the source kafka cluster is SSL-enabled, then the consumer of mm2 should be
configured to read from SSL-enabled cluster
if the target kafka cluster is SSL-enabled, then the producer of mm2 should be
configured to write to SSL-enabled cluster.
On 2021/04/16 03:23:21, Men Lim <zulu...@gmail.com> wrote:
> well the 405 is due to a syntax error in the connector.json. after fixing
> that, passing the -k switch, it started. but when looking at the
> connect.log, mm2 is only talking in plaintext rather than SSL. After a
> while it timed out because the port 9094 is ssl while mm2 is trying to use
> plaintext. at least got past the curl problem.
>
> On Wed, Apr 14, 2021 at 12:11 PM Men Lim <zulu...@gmail.com> wrote:
>
> > I read thru the security_ssl page when I started this, it doesn't apply
> > much to me because I'm running this in AWS MSK, where I can't access the
> > broker. so my hands are tied there when it come to certificate.
> >
> > however, this morning, I decided to work on creating a self sign cert for
> > the CURL command. I was able to get past the ssl handshake error and now
> > running into another issue.
> >
> > * Trying 127.0.0.1...
> > * TCP_NODELAY set
> > * Connected to localhost (127.0.0.1) port 8443 (#0)
> > * ALPN, offering h2
> > * ALPN, offering http/1.1
> > * Cipher selection:
> > ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
> > * successfully set certificate verify locations:
> > * CAfile: /etc/pki/tls/certs/ca-bundle.crt
> > CApath: none
> > * TLSv1.2 (OUT), TLS header, Certificate Status (22):
> > * TLSv1.2 (OUT), TLS handshake, Client hello (1):
> > * TLSv1.2 (IN), TLS handshake, Server hello (2):
> > * TLSv1.2 (IN), TLS handshake, Certificate (11):
> > * TLSv1.2 (OUT), TLS alert, unknown CA (560):
> >
> > ** SSL certificate problem: self signed certificate* Closing connection 0*
> >
> > so I tried passing the -k flag and got the following message
> >
> > * Trying 127.0.0.1...
> > * TCP_NODELAY set
> > * Connected to localhost (127.0.0.1) port 8443 (#0)
> > * ALPN, offering h2
> > * ALPN, offering http/1.1
> > * Cipher selection:
> > ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
> > * successfully set certificate verify locations:
> > * CAfile: /etc/pki/tls/certs/ca-bundle.crt
> > CApath: none
> > * TLSv1.2 (OUT), TLS header, Certificate Status (22):
> > * TLSv1.2 (OUT), TLS handshake, Client hello (1):
> > * TLSv1.2 (IN), TLS handshake, Server hello (2):
> > * TLSv1.2 (IN), TLS handshake, Certificate (11):
> > * TLSv1.2 (IN), TLS handshake, Server key exchange (12):
> > * TLSv1.2 (IN), TLS handshake, Server finished (14):
> > * TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
> > * TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
> > * TLSv1.2 (OUT), TLS handshake, Finished (20):
> > * TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
> > * TLSv1.2 (IN), TLS handshake, Finished (20):
> > * SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
> > * ALPN, server did not agree to a protocol
> > * Server certificate:
> > * subject: C=us; ST=il; L=Unknown; O=usfoods; OU=Unknown; CN=Unknown
> > * start date: Apr 14 16:52:07 2021 GMT
> > * expire date: Apr 12 16:52:07 2031 GMT
> > * issuer: C=US; ST=MI; L=Unknown; O=FUD; OU=Unknown; CN=Unknown
> > * SSL certificate verify result: self signed certificate (18), continuing
> > anyway.
> > > PUT /connectors HTTP/1.1
> > > Host: localhost:8443
> > > User-Agent: curl/7.61.1
> > > Accept: */*
> > > Content-Type: application/json
> > > Content-Length: 1251
> > > Expect: 100-continue
> > >
> > < HTTP/1.1 100 Continue
> > * We are completely uploaded and fine
> > < HTTP/1.1 405 Method Not Allowed
> > < Date: Wed, 14 Apr 2021 19:07:01 GMT
> > < Content-Length: 58
> > < Server: Jetty(9.4.33.v20201020)
> > <
> > * Connection #0 to host localhost left intact
> > *{"error_code":405,"message":"HTTP 405 Method Not Allowed"}*
> >
> > with the error, the mm2 process still isn't running. i'm not sure how to
> > go about troubleshooting this HTTP 405 msg. Maybe the cert can't be self
> > signed and need an official one.
> >
> >
> > On Tue, Apr 13, 2021 at 8:41 PM Ning Zhang <ning2008w...@gmail.com> wrote:
> >
> >> assume your target / destination kafka cluster is SSL enabled. If your
> >> MM2 wants to write to such cluster, you may have the following config in
> >> your MM2:
> >>
> >>
> >> https://github.com/ning2008wisc/minikube-mm2-demo/blob/master/kafka-mm/values.yaml#L79-L80
> >>
> >> on the broker (even client) side, you may refer to:
> >>
> >> http://kafka.apache.org/090/documentation.html#security_ssl
> >>
> >> On 2021/04/09 15:01:26, Men Lim <zulu...@gmail.com> wrote:
> >> > Hi Ning,
> >> >
> >> > thanks for the response. This self sign cert stays on the ec2 instance,
> >> > specifically for the curl command and I don't have to share it with the
> >> > brokers correct?
> >> >
> >> > thanks,
> >> >
> >> >
> >> >
> >> > On Fri, Apr 9, 2021 at 7:55 AM Ning Zhang <ning2008w...@gmail.com>
> >> wrote:
> >> >
> >> > > Hi Men,
> >> > >
> >> > > I used to deploy MM2 on EC2 with SSL and IIRC, probably give a try of
> >> > > self-signing certs and key for testing purpose:
> >> > > https://linuxize.com/post/creating-a-self-signed-ssl-certificate/
> >> > >
> >> > > On 2021/04/09 03:14:30, Men Lim <zulu...@gmail.com> wrote:
> >> > > > Hi Ryanne,
> >> > > >
> >> > > > thanks for the reply. My kafka clusters are on AWS, their
> >> serverless
> >> > > > platform, MSK. I'm stuck with using the default java cacerts
> >> unless I
> >> > > use
> >> > > > their AWS PCA which is pretty pricey.
> >> > > >
> >> > > > I ran the CURL command yesterday with the -v and --tlsv1.2 flag and
> >> got
> >> > > the
> >> > > > following verbose message:
> >> > > >
> >> > > > curl -s -X POST -H 'Content-Type: application/json' --data
> >> > > @connector.json
> >> > > > https://localhost:8443/connectors -v --tlsv1.2
> >> > > > * Trying 127.0.0.1...
> >> > > > * TCP_NODELAY set
> >> > > > * Connected to localhost (127.0.0.1) port 8443 (#0)
> >> > > > * ALPN, offering h2
> >> > > > * ALPN, offering http/1.1
> >> > > > * Cipher selection:
> >> > > > ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
> >> > > > * successfully set certificate verify locations:
> >> > > > * CAfile: /etc/pki/tls/certs/ca-bundle.crt
> >> > > > CApath: none
> >> > > > * TLSv1.2 (OUT), TLS header, Certificate Status (22):
> >> > > > * TLSv1.2 (OUT), TLS handshake, Client hello (1):
> >> > > > * TLSv1.2 (IN), TLS header, Unknown (21):
> >> > > > * TLSv1.2 (IN), TLS alert, handshake failure (552):
> >> > > > * error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert
> >> > > handshake
> >> > > > failure
> >> > > >
> >> > > > Thanks
> >> > > >
> >> > > > On Mon, Apr 5, 2021 at 7:26 AM Ryanne Dolan <ryannedo...@gmail.com>
> >> > > wrote:
> >> > > >
> >> > > > > Yes it's possible. The most common issue in my experience is the
> >> > > location
> >> > > > > of the trust store and key store being different or absent on some
> >> > > hosts.
> >> > > > > You need to make sure that these locations are consistent across
> >> all
> >> > > hosts
> >> > > > > in your Connect cluster, or use a ConfigProvider to provide the
> >> > > location
> >> > > > > dynamically. Otherwise, a task will get scheduled on some host and
> >> > > fail to
> >> > > > > find these files.
> >> > > > >
> >> > > > > Ryanne
> >> > > > >
> >> > > > >
> >> > > > > On Wed, Mar 31, 2021, 8:22 PM Men Lim <zulu...@gmail.com> wrote:
> >> > > > >
> >> > > > > > Hello. I was wondering if someone can help answer my
> >> question. I'm
> >> > > > > trying
> >> > > > > > to run MirrorMaker 2 in distributed mode using SSL. I have the
> >> > > > > distributor
> >> > > > > > running in SSL but when I can't get the curl REST api to do so.
> >> I saw
> >> > > > > that
> >> > > > > > kif-208 fixed this but I can't seem to implement it.
> >> > > > > >
> >> > > > > > in my mm2-dist.prop file I have set:
> >> > > > > > ////
> >> > > > > > listeners=https://localhost:8443
> >> > > > > > security.protocol=SSL
> >> > > > > >
> >> > > > > >
> >> > > > >
> >> > >
> >> ssl.truststore.location=/home/ec2-user/kafka_2.13-2.7.0/cert/kafka.client.truststore.jks
> >> > > > > > ////
> >> > > > > > my connector.json file look like this:
> >> > > > > >
> >> > > > > > ////
> >> > > > > > {
> >> > > > > > "name": "mm2-connect-cluster",
> >> > > > > > "config":{
> >> > > > > > "connector.class":
> >> > > > > "org.apache.kafka.connect.mirror.MirrorSourceConnector",
> >> > > > > > "connector.client.config.override.policy": "All",
> >> > > > > > "name": "mm2-connect-cluster",
> >> > > > > > "topics": "test.*",
> >> > > > > > "tasks.max": "1",
> >> > > > > > "source.cluster.alias": "source",
> >> > > > > > "target.cluster.alias": "target",
> >> > > > > > "source.cluster.bootstrap.servers": "source:9094",
> >> > > > > > "target.cluster.bootstrap.servers": "target:9094",
> >> > > > > > "source->target.enabled": "true",
> >> > > > > > "target->source.enabled": "false",
> >> > > > > > "offset-syncs.topic.replication.factor": "4",
> >> > > > > > "topics.exclude": ".*[\\-\\.]internal, .*\\.replica,
> >> > > > > > __consumer_offsets",
> >> > > > > > "groups.blacklist": "console-consumer-.*, connect-.*,
> >> __.*",
> >> > > > > > "topic.creation.enabled": "true",
> >> > > > > > "topic.creation.default.replication.factor": "4",
> >> > > > > > "topic.creation.default.partitions": "1"
> >> > > > > > "key.converter":
> >> > > "org.apache.kafka.connect.json.JsonConverter",
> >> > > > > > "value.converter":
> >> > > "org.apache.kafka.connect.json.JsonConverter",
> >> > > > > > "security.protocol": "SSL",
> >> > > > > > "ssl.truststore.password":
> >> > > > > >
> >> "/home/ec2-user/kafka_2.13-2.7.0/cert/kafka.client.truststore.jks"
> >> > > > > > }
> >> > > > > > }
> >> > > > > > ////
> >> > > > > >
> >> > > > > > I would then start up the distributor and it launched fine. So
> >> I
> >> > > try to
> >> > > > > > run the CURl command
> >> > > > > >
> >> > > > > > ////
> >> > > > > > curl -s -X POST -H 'Content-Type: application/json' --data
> >> > > > > @connector.json
> >> > > > > > https://localhost:8443/connectors
> >> > > > > > ////
> >> > > > > > nada. nothing. no error. no reasons for not starting.
> >> > > > > >
> >> > > > > > Is it possible to run MM2 with SSL? If so, can someone point
> >> me to a
> >> > > > > > working example?
> >> > > > > >
> >> > > > > > thanks.
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
>
