Hello everybody, I'm trying to run a simple Flink that both read from and writes to Kafka, however at the moment I'm having a hard time at running both the console consumer and producer to test the effectiveness of my test.
It is extremely important that I run these tests in a kerberized environment and right now I'm using Kafka 0.9.0.1. In the past I've used Hortonworks version without incurring in any problem but right now I can't run the "vanilla" open source version. I've followed the documentation to make Kafka work in a secure environment, created a couple of topics (flink-kafka-source and flink-kafka-sink) and assigned ACLs to them (all users have "All" permissions on the two topics I created). When I run > $KAFKA_HOME/bin/kafka-console-consumer.sh --zookeeper zk1:2181,zk2:2181,zk3:2181 --topic flink-kafka-source \ --consumer.config /home/user/kafka-test/kafka.properties I get this output: [2016-05-30 12:26:51,094] WARN Property sasl.kerberos.service.name is not valid (kafka.utils.VerifiableProperties) [2016-05-30 12:26:51,094] WARN Property sasl.mechanism is not valid (kafka.utils.VerifiableProperties) [2016-05-30 12:26:51,094] WARN Property security.protocol is not valid (kafka.utils.VerifiableProperties) [2016-05-30 12:26:51,383] WARN [console-consumer-77104_ip-172-31-18-184.eu-central-1.compute.internal-1464604011134-faf0daec], no brokers found when trying to rebalance. (kafka.consumer.ZookeeperConsumerConnector) And the output doesn't seem to be consumed. I can't seem to make the producer work either. When I run $ $KAFKA_HOME/bin/kafka-console-producer.sh --topic flink-kafka-source --broker-list kafka-broker:6667 \ --producer.config /home/user/kafka-test/kafka.properties I get this output [2016-05-30 12:47:21,718] WARN The configuration sasl.mechanism = GSSAPI was supplied but isn't a known config. (org.apache.kafka.clients.producer.ProducerConfig) And apparently stdin is not written on Kafka. The kafka.properties file contain the following line, as provided by the official documentation: security.protocol=SASL_PLAINTEXT sasl.mechanism=GSSAPI sasl.kerberos.service.name=kafka Does anybody have a hint on what I'm doing wrong? Thanks in advance. -- BR, Stefano Baghino Software Engineer @ Radicalbit
