Hi Ismael, thank you so much for helping out: both tips (using the proper configs for my Kafka version and appending the --new-consumer option) proved right and I've been able to run my simple Flink job reading from and writing to Kafka in a secure environment.
Best, Stefano On Mon, May 30, 2016 at 8:21 PM, Ismael Juma <isma...@gmail.com> wrote: > Hi Stefano, > > For the console consumer, you need to pass --new-consumer for it to work. > > For the producer, it seems like you are using configs for Kafka 0.10 with a > 0.9 producer, but the warning is benign. Are you sure the producer is not > working? > > Ismael > On 30 May 2016 11:50, "Stefano Baghino" <stefano.bagh...@radicalbit.io> > wrote: > > Hello everybody, > > I'm trying to run a simple Flink that both read from and writes to Kafka, > however at the moment I'm having a hard time at running both the console > consumer and producer to test the effectiveness of my test. > > It is extremely important that I run these tests in a kerberized > environment and right now I'm using Kafka 0.9.0.1. In the past I've used > Hortonworks version without incurring in any problem but right now I can't > run the "vanilla" open source version. > > I've followed the documentation to make Kafka work in a secure environment, > created a couple of topics (flink-kafka-source and flink-kafka-sink) and > assigned ACLs to them (all users have "All" permissions on the two topics I > created). > > When I run > > > $KAFKA_HOME/bin/kafka-console-consumer.sh --zookeeper > zk1:2181,zk2:2181,zk3:2181 --topic flink-kafka-source \ > --consumer.config /home/user/kafka-test/kafka.properties > > I get this output: > > [2016-05-30 12:26:51,094] WARN Property sasl.kerberos.service.name is not > valid (kafka.utils.VerifiableProperties) > [2016-05-30 12:26:51,094] WARN Property sasl.mechanism is not valid > (kafka.utils.VerifiableProperties) > [2016-05-30 12:26:51,094] WARN Property security.protocol is not valid > (kafka.utils.VerifiableProperties) > [2016-05-30 12:26:51,383] WARN > > [console-consumer-77104_ip-172-31-18-184.eu-central-1.compute.internal-1464604011134-faf0daec], > no brokers found when trying to rebalance. > (kafka.consumer.ZookeeperConsumerConnector) > And the output doesn't seem to be consumed. > > I can't seem to make the producer work either. When I run > > $ $KAFKA_HOME/bin/kafka-console-producer.sh --topic flink-kafka-source > --broker-list kafka-broker:6667 \ > --producer.config /home/user/kafka-test/kafka.properties > > I get this output > > [2016-05-30 12:47:21,718] WARN The configuration sasl.mechanism = GSSAPI > was supplied but isn't a known config. > (org.apache.kafka.clients.producer.ProducerConfig) > And apparently stdin is not written on Kafka. > > The kafka.properties file contain the following line, as provided by the > official documentation: > > security.protocol=SASL_PLAINTEXT > sasl.mechanism=GSSAPI > sasl.kerberos.service.name=kafka > Does anybody have a hint on what I'm doing wrong? Thanks in advance. > > -- > BR, > Stefano Baghino > > Software Engineer @ Radicalbit > -- BR, Stefano Baghino Software Engineer @ Radicalbit
