Re: Reading from HDFS and publishing to Kafka

2020-09-29 Thread Aljoscha Krettek
Hi, I actually have no experience running a Flink job on K8s against a kerberized HDFS so please take what I'll say with a grain of salt. The only thing you should need to do is to configure the path of your keytab and possibly some other Kerberos settings. For that check out [1] and [2].

Re: Reading from HDFS and publishing to Kafka

2020-09-27 Thread Khachatryan Roman
Hi, 1. Yes, StreamingExecutionEnvironment.readFile can be used for files on HDFS 2. I think this is a valid concern. Besides that, there are plans to deprecate DataSet API [1] 4. Yes, the approach looks good I'm pulling in Aljoscha for your 3rd question (and probably some clarifications on others