Great to hear Stefano, thanks for the update. Ismael
On Tue, May 31, 2016 at 10:13 AM, Stefano Baghino < stefano.bagh...@radicalbit.io> wrote: > Hi Ismael, > > thank you so much for helping out: both tips (using the proper configs for > my Kafka version and appending the --new-consumer option) proved right and > I've been able to run my simple Flink job reading from and writing to Kafka > in a secure environment. > > Best, > Stefano > > On Mon, May 30, 2016 at 8:21 PM, Ismael Juma <isma...@gmail.com> wrote: > > > Hi Stefano, > > > > For the console consumer, you need to pass --new-consumer for it to work. > > > > For the producer, it seems like you are using configs for Kafka 0.10 > with a > > 0.9 producer, but the warning is benign. Are you sure the producer is not > > working? > > > > Ismael > > On 30 May 2016 11:50, "Stefano Baghino" <stefano.bagh...@radicalbit.io> > > wrote: > > > > Hello everybody, > > > > I'm trying to run a simple Flink that both read from and writes to Kafka, > > however at the moment I'm having a hard time at running both the console > > consumer and producer to test the effectiveness of my test. > > > > It is extremely important that I run these tests in a kerberized > > environment and right now I'm using Kafka 0.9.0.1. In the past I've used > > Hortonworks version without incurring in any problem but right now I > can't > > run the "vanilla" open source version. > > > > I've followed the documentation to make Kafka work in a secure > environment, > > created a couple of topics (flink-kafka-source and flink-kafka-sink) and > > assigned ACLs to them (all users have "All" permissions on the two > topics I > > created). > > > > When I run > > > > > $KAFKA_HOME/bin/kafka-console-consumer.sh --zookeeper > > zk1:2181,zk2:2181,zk3:2181 --topic flink-kafka-source \ > > --consumer.config /home/user/kafka-test/kafka.properties > > > > I get this output: > > > > [2016-05-30 12:26:51,094] WARN Property sasl.kerberos.service.name is > not > > valid (kafka.utils.VerifiableProperties) > > [2016-05-30 12:26:51,094] WARN Property sasl.mechanism is not valid > > (kafka.utils.VerifiableProperties) > > [2016-05-30 12:26:51,094] WARN Property security.protocol is not valid > > (kafka.utils.VerifiableProperties) > > [2016-05-30 12:26:51,383] WARN > > > > > [console-consumer-77104_ip-172-31-18-184.eu-central-1.compute.internal-1464604011134-faf0daec], > > no brokers found when trying to rebalance. > > (kafka.consumer.ZookeeperConsumerConnector) > > And the output doesn't seem to be consumed. > > > > I can't seem to make the producer work either. When I run > > > > $ $KAFKA_HOME/bin/kafka-console-producer.sh --topic flink-kafka-source > > --broker-list kafka-broker:6667 \ > > --producer.config /home/user/kafka-test/kafka.properties > > > > I get this output > > > > [2016-05-30 12:47:21,718] WARN The configuration sasl.mechanism = GSSAPI > > was supplied but isn't a known config. > > (org.apache.kafka.clients.producer.ProducerConfig) > > And apparently stdin is not written on Kafka. > > > > The kafka.properties file contain the following line, as provided by the > > official documentation: > > > > security.protocol=SASL_PLAINTEXT > > sasl.mechanism=GSSAPI > > sasl.kerberos.service.name=kafka > > Does anybody have a hint on what I'm doing wrong? Thanks in advance. > > > > -- > > BR, > > Stefano Baghino > > > > Software Engineer @ Radicalbit > > > > > > -- > BR, > Stefano Baghino > > Software Engineer @ Radicalbit >
