Do you get any output from *docker logs*? And does it work if you don't use authentication? How about if you try one of the dockerised Kafka Connect examples here? https://github.com/confluentinc/demo-scene/tree/master/kafka-connect-zero-to-hero
-- Robin Moffatt | Senior Developer Advocate | ro...@confluent.io | @rmoff On Fri, 19 Jul 2019 at 10:53, Aleksandar Irikov <airiko...@gmail.com> wrote: > Hey everyone, > > I am having some issues when running Kafka Connect inside of Docker, so > would really appreciate some feedback on what I'm doing wrong. > > I'm able to run locally (by executing `connect-distributed > config.properties`. However, when running in Docker and passing the same > configuration as environment variables, after Kafka Connect has started and > starts assigning tasks to workers, the container just exits without any > error. > > Local setup: > - I have Kafka installed locally, so I'm using the `connect-distributed` > script along with my properties file, which are listed below. This seems to > work without issues > > # Kafka broker IP addresses to connect to > bootstrap.servers=kafka:9092 > ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1 > ssl.truststore.location=truststore.jks > ssl.truststore.password=some-password > ssl.protocol=TLS > security.protocol=SASL_SSL > ssl.endpoint.identification.algorithm= > sasl.mechanism=SCRAM-SHA-256 > sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule > required \ > username="kafka_username" \ > password="kafka_password"; > consumer.ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1 > consumer.ssl.truststore.location=truststore.jks > consumer.ssl.truststore.password=some-password > consumer.ssl.protocol=TLS > consumer.security.protocol=SASL_SSL > consumer.ssl.endpoint.identification.algorithm= > consumer.sasl.mechanism=SCRAM-SHA-256 > # CONFIGURE USER AND PASSWORD! > > consumer.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule > required \ > > username="kafka_username" \ > password="kafka_password"; > > > group.id=my-group > config.storage.topic=kconnect.config > offset.storage.topic=kconnect.offsets > status.storage.topic=kconnect.status > > > # Path to directory containing the connect jar and dependencies > plugin.path=path/to/my/plugins/ > # Converters to use to convert keys and values > key.converter=org.apache.kafka.connect.converters.ByteArrayConverter > value.converter=org.apache.kafka.connect.converters.ByteArrayConverter > > # The internal converters Kafka Connect uses for storing offset and > configuration data > internal.key.converter=org.apache.kafka.connect.json.JsonConverter > internal.value.converter=org.apache.kafka.connect.json.JsonConverter > internal.key.converter.schemas.enable=false > internal.value.converter.schemas.enable=false > offset.storage.file.filename=/tmp/kconnect.offsets > rest.port=XXXX > > > Dockerized setup: > - I'm using the Confluent Kafka Connect Base Image (I've also tried with > the "fat" Confluent image), to which I'm passing my configuration as > environment variables when I run the container. > Dockerfile: > > FROM confluentinc/cp-kafka-connect-base > > COPY file-sink.txt /output/file-sink.txt > > COPY truststore.jks /trustores/ > > > Docker run with environs: > > docker run --rm --name kconnect -p 0.0.0.0:XXXX:XXXX \ > -e CONNECT_BOOTSTRAP_SERVERS=kafka:9092 \ > -e CONNECT_SSL_PROTOCOL=TLS \ > -e CONNECT_ENABLED_PROTOCOLS=TLSv1.2,TLSv1.1,TLSv1 \ > -e CONNECT_SSL_TRUSTSTORE_LOCATION=/truststores/truststore.jks \ > -e CONNECT_SSL_TRUSTSTORE_PASSWORD=instaclustr \ > -e CONNECT_SECURITY_PROTOCOL=SASL_SSL \ > -e > CONNECT_SASL_JAAS_CONFIG='org.apache.kafka.common.security.scram.ScramLoginModule > required username="KAFKA_USERNAME" password="KAFKA_PASSWORD";' \ > -e CONNECT_SASL_MECHANISM=SCRAM-SHA-256 \ > -e CONNECT_GROUP_ID=my-group \ > -e CONNECT_CONFIG_STORAGE_TOPIC="kconnect.config" \ > -e CONNECT_OFFSET_STORAGE_TOPIC="kconnect.offsets" \ > -e CONNECT_STATUS_STORAGE_TOPIC="kconnect.status" \ > -e > CONNECT_KEY_CONVERTER="org.apache.kafka.connect.converters.ByteArrayConverter" > \ > -e > CONNECT_VALUE_CONVERTER="org.apache.kafka.connect.converters.ByteArrayConverter" > \ > -e > CONNECT_INTERNAL_KEY_CONVERTER="org.apache.kafka.connect.json.JsonConverter" > \ > -e > CONNECT_INTERNAL_VALUE_CONVERTER="org.apache.kafka.connect.json.JsonConverter" > \ > -e CONNECT_REST_ADVERTISED_HOST_NAME=localhost \ > -e CONNECT_REST_PORT=XXXX:XXXX \ > -e CONNECT_PLUGIN_PATH='path/to/my/plugins/' \ > kconnect-image > > > In both cases, after the Kafka Connect Server has started, I send the same > request to both: > > http://localhost:XXXX/connectors \ > -H "Content-Type: application/json" -H "Accept: application/json" -d @- \ > << EOF > { > "name": "simple-file-sink-connector", > "config": { > "connector.class": > "org.apache.kafka.connect.file.FileStreamSinkConnector", > "file": "./output/file-sink.txt", > "key.converter": > "org.apache.kafka.connect.converters.ByteArrayConverter", > "tasks.max": 3, > "topics": "my-topic", > "value.converter": > "org.apache.kafka.connect.converters.ByteArrayConverter" > } > } > EOF > > > In this case I'm using the already built-in FileStreamSinkConnector to > eliminate another moving part. There are no issues regarding the existence > of the file-sink.txt file, if I have made a mistake somewhere here in my > example. > Without registering any connectors the Kconnect servers behave exactly the > same. However, when I send the above-mentioned request to local, everything > works as expected but when I send the request to the Kconnect in Docker, > after it has assigned the tasks it just dies without any logs. The exit > code from the docker container I got by running `echo $?` is 137. > > I'm really puzzled by this behaviour. I hope I have provided enough > information above, but if I haven't, please let me know. > > Thanks a lot for reading this wall of text and for your help. > > All the best! > Aleksandar Irikov >
