Hi Yangze,
I tried the following: maybe I am missing something.
https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/cli.html
-yt,--yarnship <arg>
Run:
/usr/lib/flink/bin/flink run -m yarn-cluster
-yt ${app_install_path}/conf
my KRB5.conf is in ${app_install_path}/conf n master node (local build
path)
When this folder is shipped to yarn, how should i reference this KRB5.conf
now in run command?
I tried like: -yD java.security.krb5.conf=./krb5.conf \
Didn't work this way. Please suggest, can file be used as relative path
./krb5.conf or what is misinterpreted?
Note: When we manually updated KRB5.conf on all cluster nodes in
/etc/KRB5.conf it was working. But I am trying to make it available as JVM
property.
Regards,
Vijay
On Thu, Aug 13, 2020 at 9:21 PM Yangze Guo <karma...@gmail.com> wrote:
> Hi,
>
> When deploying Flink on Yarn, you could ship krb5.conf by "--ship"
> command. Notice that this command only supports to ship folders now.
>
> Best,
> Yangze Guo
>
> On Fri, Aug 14, 2020 at 11:22 AM Vijayendra Yadav <contact....@gmail.com>
> wrote:
> >
> > Any inputs ?
> >
> > On Tue, Aug 11, 2020 at 10:34 AM Vijayendra Yadav <contact....@gmail.com>
> wrote:
> >>
> >> Dawid, I was able to resolve the keytab issue by passing the service
> name, but now I am facing the KRB5 issue.
> >>
> >> Caused by: org.apache.kafka.common.errors.SaslAuthenticationException:
> Failed to create SaslClient with mechanism GSSAPI
> >> Caused by: javax.security.sasl.SaslException: Failure to initialize
> security context [Caused by GSSException: Invalid name provided (Mechanism
> level: KrbException: Cannot locate default realm)]
> >>
> >> I passed KRB5 from yaml conf file like:
> >>
> >> env.java.opts.jobmanager: -Djava.security.krb5.conf=/path/krb5.conf
> >> env.java.opts.taskmanager: -Djava.security.krb5.conf=/path/krb5.conf
> >>
> >> How can I resolve this? Is there another way to pass KRB5?
> >>
> >> I also tried via option#1 from flink run command -D parameter.
> >>
> >> Regards,
> >> Vijay
> >>
> >>
> >> On Tue, Aug 11, 2020 at 1:26 AM Dawid Wysakowicz <
> dwysakow...@apache.org> wrote:
> >>>
> >>> Hi,
> >>>
> >>> As far as I know the approach 2) is the supported way of setting up
> Kerberos authentication in Flink. In the second approach have you tried
> setting the `sasl.kerberos.service.name` in the configuration of your
> KafkaConsumer/Producer[1]? I think this might be the issue.
> >>>
> >>> Best,
> >>>
> >>> Dawid
> >>>
> >>> [1]
> https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/kafka.html#enabling-kerberos-authentication
> >>>
> >>>
> >>> On 09/08/2020 20:39, Vijayendra Yadav wrote:
> >>>
> >>> Hi Team,
> >>>
> >>> I am trying to stream data from kafkaconsumer using:
> https://ci.apache.org/projects/flink/flink-docs-stable/dev/connectors/kafka.html
> >>>
> >>> Here my KAFKA is Kerberos secured and SSL enabled.
> >>>
> >>> I am running my Flink streaming in yarn-cluster on EMR 5.31.
> >>>
> >>> I have tried to pass keytab/principal in following 2 Ways:
> >>>
> >>> 1) Passing as JVM property in Flink run Command.
> >>>
> >>> /usr/lib/flink/bin/flink run
> >>> -yt ${app_install_path}/conf/
> \
> >>>>
> >>>> -Dsecurity.kerberos.login.use-ticket-cache=false
> \
> >>>> -yDsecurity.kerberos.login.use-ticket-cache=false
> \
> >>>> -Dsecurity.kerberos.login.keytab=${app_install_path}/conf/keytab \
> >>>> -yDsecurity.kerberos.login.keytab=${app_install_path}/conf/.keytab \
> >>>> -Djava.security.krb5.conf=${app_install_path}/conf/krb5.conf
> \
> >>>> -yDjava.security.krb5.conf=${app_install_path}/conf/krb5.conf
> \
> >>>> -Dsecurity.kerberos.login.principal=x...@xx.net \
> >>>> -yDsecurity.kerberos.login.principal= x...@xx.net \
> >>>> -Dsecurity.kerberos.login.contexts=Client,KafkaClient
> \
> >>>> -yDsecurity.kerberos.login.contexts=Client,KafkaClient
> >>>
> >>>
> >>> Here, I am getting the following Error, it seems like KEYTAB Was not
> transported to the run environment and probably not found.
> >>>
> >>> org.apache.kafka.common.KafkaException: Failed to construct kafka
> consumer
> >>> Caused by: java.lang.IllegalArgumentException: Could not find a
> 'KafkaClient' entry in the JAAS configuration. System property
> 'java.security.auth.login.config'
> >>>
> >>> 2) Passing from flink config: /usr/lib/flink/conf/flink-conf.yaml
> >>>
> >>> security.kerberos.login.use-ticket-cache: false
> >>> security.kerberos.login.keytab: ${app_install_path}/conf/keytab
> >>> security.kerberos.login.principal: x...@xx.net
> >>> security.kerberos.login.contexts: Client,KafkaClient
> >>>
> >>> Here, I am getting the following Error,
> >>>
> >>> org.apache.kafka.common.KafkaException: Failed to construct kafka
> consumer
> >>> Caused by: org.apache.kafka.common.KafkaException:
> java.lang.IllegalArgumentException: No serviceName defined in either JAAS
> or Kafka config
> >>>
> >>>
> >>> Could you please help find, what are probable causes and resolution?
> >>>
> >>> Regards,
> >>> Vijay
> >>>
>
