Koert, I think what you're seeing is that there are actually 3 different ways Connect can interact with Kafka. For both standalone and distributed mode, you have producers and consumers that are part of the source and sink connector implementations, respectively. Security for these are configured using the producer. and consumer. prefixed configurations in the worker config. In distributed mode, Connect also leverages Kafka's group membership protocol to coordinate the workers and distribute work between them. The security settings for these are picked up in the distributed worker config without any prefixes.
For more info on configuring security, you can see Confluent's docs on that here: http://docs.confluent.io/3.1.1/connect/security.html#security We realize having to specify this multiple times is annoying if you want to use the same set of credentials, but for other configurations it is important to keep the configs for worker/producer/consumer isolated (such as interceptors, which use the same config name but different interfaces for ProducerInterceptor vs ConsumerInterceptor). For configs we know might be shared, we'd like to find a way to make this configuration simpler. -Ewen On Fri, Nov 25, 2016 at 10:51 AM, Koert Kuipers <ko...@tresata.com> wrote: > well it seems if you run connect in distributed mode... its again > security.protocol=SASL_PLAINTEXT and not producer.security.protocol= > SASL_PLAINTEXT > > dont ask me why > > On Thu, Nov 24, 2016 at 10:40 PM, Koert Kuipers <ko...@tresata.com> wrote: > > > for anyone that runs into this. turns out i also had to set: > > producer.security.protocol=SASL_PLAINTEXT > > producer.sasl.kerberos.service.name=kafka > > > > > > On Thu, Nov 24, 2016 at 8:54 PM, Koert Kuipers <ko...@tresata.com> > wrote: > > > >> i have a secure kafka 0.10.1 cluster using SASL_PLAINTEXT > >> > >> the kafka servers seem fine, and i can start console-consumer and > >> console-producer and i see the message i type in the producer pop up in > the > >> consumer. no problems so far. > >> > >> for example to start console-producer: > >> $ kinit > >> $ export KAFKA_OPTS="-Djava.security.auth.login.config=config/jaas. > conf" > >> $ bin/kafka-console-producer.sh --producer.config > >> config/producer.properties --topic test --broker-list > >> SASL_PLAINTEXT://somenode:9092 > >> > >> but i am having no luck whatsoever with kafka-connect. i tried this: > >> $ kinit > >> $ export KAFKA_OPTS="-Djava.security.auth.login.config=config/jaas. > conf" > >> $ bin/connect-standalone.sh config/connect-standalone.properties > >> config/connect-console-source.properties > >> > >> my config/connect-console-source.properties is unchanged. my > >> config/connect-standalone has: > >> > >> bootstrap.servers=SASL_PLAINTEXT://somenode:9092 > >> security.protocol=SASL_PLAINTEXT > >> sasl.kerberos.service.name=kafka > >> key.converter=org.apache.kafka.connect.json.JsonConverter > >> value.converter=org.apache.kafka.connect.json.JsonConverter > >> internal.key.converter=org.apache.kafka.connect.json.JsonConverter > >> internal.value.converter=org.apache.kafka.connect.json.JsonConverter > >> internal.key.converter.schemas.enable=false > >> internal.value.converter.schemas.enable=false > >> offset.storage.file.filename=/tmp/connect.offsets > >> offset.flush.interval.ms=10000 > >> > >> i get these logs in an infinite loop: > >> [2016-11-24 20:47:18,528] DEBUG Node -1 disconnected. > >> (org.apache.kafka.clients.NetworkClient:463) > >> [2016-11-24 20:47:18,528] WARN Bootstrap broker somenode:9092 > >> disconnected (org.apache.kafka.clients.NetworkClient:568) > >> [2016-11-24 20:47:18,528] DEBUG Give up sending metadata request since > no > >> node is available (org.apache.kafka.clients.NetworkClient:625) > >> [2016-11-24 20:47:18,629] DEBUG Initialize connection to node -1 for > >> sending metadata request (org.apache.kafka.clients.NetworkClient:644) > >> [2016-11-24 20:47:18,629] DEBUG Initiating connection to node -1 at > >> somenode:9092. (org.apache.kafka.clients.NetworkClient:496) > >> [2016-11-24 20:47:18,631] DEBUG Created socket with SO_RCVBUF = 32768, > >> SO_SNDBUF = 124928, SO_TIMEOUT = 0 to node -1 > (org.apache.kafka.common.netwo > >> rk.Selector:327) > >> [2016-11-24 20:47:18,631] DEBUG Completed connection to node -1 > >> (org.apache.kafka.clients.NetworkClient:476) > >> [2016-11-24 20:47:18,730] DEBUG Sending metadata request > >> {topics=[connect-test]} to node -1 (org.apache.kafka.clients.Netw > >> orkClient:640) > >> [2016-11-24 20:47:18,730] DEBUG Connection with somenode/192.168.1.54 > >> disconnected (org.apache.kafka.common.network.Selector:365) > >> java.io.EOFException > >> at org.apache.kafka.common.network.NetworkReceive.readFromReada > >> bleChannel(NetworkReceive.java:83) > >> at org.apache.kafka.common.network.NetworkReceive.readFrom( > >> NetworkReceive.java:71) > >> at org.apache.kafka.common.network.KafkaChannel.receive(KafkaCh > >> annel.java:154) > >> at org.apache.kafka.common.network.KafkaChannel.read(KafkaChann > >> el.java:135) > >> at org.apache.kafka.common.network.Selector.pollSelectionKeys( > >> Selector.java:343) > >> at org.apache.kafka.common.network.Selector.poll(Selector.java: > >> 291) > >> at org.apache.kafka.clients.NetworkClient.poll(NetworkClient. > >> java:260) > >> at org.apache.kafka.clients.producer.internals.Sender.run(Sende > >> r.java:236) > >> at org.apache.kafka.clients.producer.internals.Sender.run(Sende > >> r.java:135) > >> at java.lang.Thread.run(Thread.java:745) > >> > >> i tried different kafka-connect connectors, same result. > >> > >> any ideas? thanks! > >> > > > > > -- Thanks, Ewen
