I got this working late last week. I did the shotgun approach with all the
values in both the distributor and connector.json and once it worked, I
started to remove things to see when it stop working.
On Sun, Apr 18, 2021 at 10:00 PM Ning Zhang <ning2008w...@gmail.com> wrote:
> if the source kafka cluster is SSL-enabled, then the consumer of mm2
> should be configured to read from SSL-enabled cluster
> if the target kafka cluster is SSL-enabled, then the producer of mm2
> should be configured to write to SSL-enabled cluster.
>
> On 2021/04/16 03:23:21, Men Lim <zulu...@gmail.com> wrote:
> > well the 405 is due to a syntax error in the connector.json. after
> fixing
> > that, passing the -k switch, it started. but when looking at the
> > connect.log, mm2 is only talking in plaintext rather than SSL. After a
> > while it timed out because the port 9094 is ssl while mm2 is trying to
> use
> > plaintext. at least got past the curl problem.
> >
> > On Wed, Apr 14, 2021 at 12:11 PM Men Lim <zulu...@gmail.com> wrote:
> >
> > > I read thru the security_ssl page when I started this, it doesn't apply
> > > much to me because I'm running this in AWS MSK, where I can't access
> the
> > > broker. so my hands are tied there when it come to certificate.
> > >
> > > however, this morning, I decided to work on creating a self sign cert
> for
> > > the CURL command. I was able to get past the ssl handshake error and
> now
> > > running into another issue.
> > >
> > > * Trying 127.0.0.1...
> > > * TCP_NODELAY set
> > > * Connected to localhost (127.0.0.1) port 8443 (#0)
> > > * ALPN, offering h2
> > > * ALPN, offering http/1.1
> > > * Cipher selection:
> > > ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
> > > * successfully set certificate verify locations:
> > > * CAfile: /etc/pki/tls/certs/ca-bundle.crt
> > > CApath: none
> > > * TLSv1.2 (OUT), TLS header, Certificate Status (22):
> > > * TLSv1.2 (OUT), TLS handshake, Client hello (1):
> > > * TLSv1.2 (IN), TLS handshake, Server hello (2):
> > > * TLSv1.2 (IN), TLS handshake, Certificate (11):
> > > * TLSv1.2 (OUT), TLS alert, unknown CA (560):
> > >
> > > ** SSL certificate problem: self signed certificate* Closing
> connection 0*
> > >
> > > so I tried passing the -k flag and got the following message
> > >
> > > * Trying 127.0.0.1...
> > > * TCP_NODELAY set
> > > * Connected to localhost (127.0.0.1) port 8443 (#0)
> > > * ALPN, offering h2
> > > * ALPN, offering http/1.1
> > > * Cipher selection:
> > > ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
> > > * successfully set certificate verify locations:
> > > * CAfile: /etc/pki/tls/certs/ca-bundle.crt
> > > CApath: none
> > > * TLSv1.2 (OUT), TLS header, Certificate Status (22):
> > > * TLSv1.2 (OUT), TLS handshake, Client hello (1):
> > > * TLSv1.2 (IN), TLS handshake, Server hello (2):
> > > * TLSv1.2 (IN), TLS handshake, Certificate (11):
> > > * TLSv1.2 (IN), TLS handshake, Server key exchange (12):
> > > * TLSv1.2 (IN), TLS handshake, Server finished (14):
> > > * TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
> > > * TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
> > > * TLSv1.2 (OUT), TLS handshake, Finished (20):
> > > * TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
> > > * TLSv1.2 (IN), TLS handshake, Finished (20):
> > > * SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
> > > * ALPN, server did not agree to a protocol
> > > * Server certificate:
> > > * subject: C=us; ST=il; L=Unknown; O=usfoods; OU=Unknown; CN=Unknown
> > > * start date: Apr 14 16:52:07 2021 GMT
> > > * expire date: Apr 12 16:52:07 2031 GMT
> > > * issuer: C=US; ST=MI; L=Unknown; O=FUD; OU=Unknown; CN=Unknown
> > > * SSL certificate verify result: self signed certificate (18),
> continuing
> > > anyway.
> > > > PUT /connectors HTTP/1.1
> > > > Host: localhost:8443
> > > > User-Agent: curl/7.61.1
> > > > Accept: */*
> > > > Content-Type: application/json
> > > > Content-Length: 1251
> > > > Expect: 100-continue
> > > >
> > > < HTTP/1.1 100 Continue
> > > * We are completely uploaded and fine
> > > < HTTP/1.1 405 Method Not Allowed
> > > < Date: Wed, 14 Apr 2021 19:07:01 GMT
> > > < Content-Length: 58
> > > < Server: Jetty(9.4.33.v20201020)
> > > <
> > > * Connection #0 to host localhost left intact
> > > *{"error_code":405,"message":"HTTP 405 Method Not Allowed"}*
> > >
> > > with the error, the mm2 process still isn't running. i'm not sure how
> to
> > > go about troubleshooting this HTTP 405 msg. Maybe the cert can't be
> self
> > > signed and need an official one.
> > >
> > >
> > > On Tue, Apr 13, 2021 at 8:41 PM Ning Zhang <ning2008w...@gmail.com>
> wrote:
> > >
> > >> assume your target / destination kafka cluster is SSL enabled. If your
> > >> MM2 wants to write to such cluster, you may have the following config
> in
> > >> your MM2:
> > >>
> > >>
> > >>
> https://github.com/ning2008wisc/minikube-mm2-demo/blob/master/kafka-mm/values.yaml#L79-L80
> > >>
> > >> on the broker (even client) side, you may refer to:
> > >>
> > >> http://kafka.apache.org/090/documentation.html#security_ssl
> > >>
> > >> On 2021/04/09 15:01:26, Men Lim <zulu...@gmail.com> wrote:
> > >> > Hi Ning,
> > >> >
> > >> > thanks for the response. This self sign cert stays on the ec2
> instance,
> > >> > specifically for the curl command and I don't have to share it with
> the
> > >> > brokers correct?
> > >> >
> > >> > thanks,
> > >> >
> > >> >
> > >> >
> > >> > On Fri, Apr 9, 2021 at 7:55 AM Ning Zhang <ning2008w...@gmail.com>
> > >> wrote:
> > >> >
> > >> > > Hi Men,
> > >> > >
> > >> > > I used to deploy MM2 on EC2 with SSL and IIRC, probably give a
> try of
> > >> > > self-signing certs and key for testing purpose:
> > >> > > https://linuxize.com/post/creating-a-self-signed-ssl-certificate/
> > >> > >
> > >> > > On 2021/04/09 03:14:30, Men Lim <zulu...@gmail.com> wrote:
> > >> > > > Hi Ryanne,
> > >> > > >
> > >> > > > thanks for the reply. My kafka clusters are on AWS, their
> > >> serverless
> > >> > > > platform, MSK. I'm stuck with using the default java cacerts
> > >> unless I
> > >> > > use
> > >> > > > their AWS PCA which is pretty pricey.
> > >> > > >
> > >> > > > I ran the CURL command yesterday with the -v and --tlsv1.2 flag
> and
> > >> got
> > >> > > the
> > >> > > > following verbose message:
> > >> > > >
> > >> > > > curl -s -X POST -H 'Content-Type: application/json' --data
> > >> > > @connector.json
> > >> > > > https://localhost:8443/connectors -v --tlsv1.2
> > >> > > > * Trying 127.0.0.1...
> > >> > > > * TCP_NODELAY set
> > >> > > > * Connected to localhost (127.0.0.1) port 8443 (#0)
> > >> > > > * ALPN, offering h2
> > >> > > > * ALPN, offering http/1.1
> > >> > > > * Cipher selection:
> > >> > > > ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
> > >> > > > * successfully set certificate verify locations:
> > >> > > > * CAfile: /etc/pki/tls/certs/ca-bundle.crt
> > >> > > > CApath: none
> > >> > > > * TLSv1.2 (OUT), TLS header, Certificate Status (22):
> > >> > > > * TLSv1.2 (OUT), TLS handshake, Client hello (1):
> > >> > > > * TLSv1.2 (IN), TLS header, Unknown (21):
> > >> > > > * TLSv1.2 (IN), TLS alert, handshake failure (552):
> > >> > > > * error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert
> > >> > > handshake
> > >> > > > failure
> > >> > > >
> > >> > > > Thanks
> > >> > > >
> > >> > > > On Mon, Apr 5, 2021 at 7:26 AM Ryanne Dolan <
> ryannedo...@gmail.com>
> > >> > > wrote:
> > >> > > >
> > >> > > > > Yes it's possible. The most common issue in my experience is
> the
> > >> > > location
> > >> > > > > of the trust store and key store being different or absent on
> some
> > >> > > hosts.
> > >> > > > > You need to make sure that these locations are consistent
> across
> > >> all
> > >> > > hosts
> > >> > > > > in your Connect cluster, or use a ConfigProvider to provide
> the
> > >> > > location
> > >> > > > > dynamically. Otherwise, a task will get scheduled on some
> host and
> > >> > > fail to
> > >> > > > > find these files.
> > >> > > > >
> > >> > > > > Ryanne
> > >> > > > >
> > >> > > > >
> > >> > > > > On Wed, Mar 31, 2021, 8:22 PM Men Lim <zulu...@gmail.com>
> wrote:
> > >> > > > >
> > >> > > > > > Hello. I was wondering if someone can help answer my
> > >> question. I'm
> > >> > > > > trying
> > >> > > > > > to run MirrorMaker 2 in distributed mode using SSL. I have
> the
> > >> > > > > distributor
> > >> > > > > > running in SSL but when I can't get the curl REST api to do
> so.
> > >> I saw
> > >> > > > > that
> > >> > > > > > kif-208 fixed this but I can't seem to implement it.
> > >> > > > > >
> > >> > > > > > in my mm2-dist.prop file I have set:
> > >> > > > > > ////
> > >> > > > > > listeners=https://localhost:8443
> > >> > > > > > security.protocol=SSL
> > >> > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > >
> > >>
> ssl.truststore.location=/home/ec2-user/kafka_2.13-2.7.0/cert/kafka.client.truststore.jks
> > >> > > > > > ////
> > >> > > > > > my connector.json file look like this:
> > >> > > > > >
> > >> > > > > > ////
> > >> > > > > > {
> > >> > > > > > "name": "mm2-connect-cluster",
> > >> > > > > > "config":{
> > >> > > > > > "connector.class":
> > >> > > > > "org.apache.kafka.connect.mirror.MirrorSourceConnector",
> > >> > > > > > "connector.client.config.override.policy": "All",
> > >> > > > > > "name": "mm2-connect-cluster",
> > >> > > > > > "topics": "test.*",
> > >> > > > > > "tasks.max": "1",
> > >> > > > > > "source.cluster.alias": "source",
> > >> > > > > > "target.cluster.alias": "target",
> > >> > > > > > "source.cluster.bootstrap.servers": "source:9094",
> > >> > > > > > "target.cluster.bootstrap.servers": "target:9094",
> > >> > > > > > "source->target.enabled": "true",
> > >> > > > > > "target->source.enabled": "false",
> > >> > > > > > "offset-syncs.topic.replication.factor": "4",
> > >> > > > > > "topics.exclude": ".*[\\-\\.]internal, .*\\.replica,
> > >> > > > > > __consumer_offsets",
> > >> > > > > > "groups.blacklist": "console-consumer-.*,
> connect-.*,
> > >> __.*",
> > >> > > > > > "topic.creation.enabled": "true",
> > >> > > > > > "topic.creation.default.replication.factor": "4",
> > >> > > > > > "topic.creation.default.partitions": "1"
> > >> > > > > > "key.converter":
> > >> > > "org.apache.kafka.connect.json.JsonConverter",
> > >> > > > > > "value.converter":
> > >> > > "org.apache.kafka.connect.json.JsonConverter",
> > >> > > > > > "security.protocol": "SSL",
> > >> > > > > > "ssl.truststore.password":
> > >> > > > > >
> > >> "/home/ec2-user/kafka_2.13-2.7.0/cert/kafka.client.truststore.jks"
> > >> > > > > > }
> > >> > > > > > }
> > >> > > > > > ////
> > >> > > > > >
> > >> > > > > > I would then start up the distributor and it launched
> fine. So
> > >> I
> > >> > > try to
> > >> > > > > > run the CURl command
> > >> > > > > >
> > >> > > > > > ////
> > >> > > > > > curl -s -X POST -H 'Content-Type: application/json' --data
> > >> > > > > @connector.json
> > >> > > > > > https://localhost:8443/connectors
> > >> > > > > > ////
> > >> > > > > > nada. nothing. no error. no reasons for not starting.
> > >> > > > > >
> > >> > > > > > Is it possible to run MM2 with SSL? If so, can someone
> point
> > >> me to a
> > >> > > > > > working example?
> > >> > > > > >
> > >> > > > > > thanks.
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> >
>
