Hi everybody,
I'm trying to set up reading from HDFS using docker-compose and Flink
1.11.3.
If I pass 'env.hadoop.conf.dir' and 'env.yarn.conf.dir'
using FLINK_PROPERTIES (under environment section of the docker-compose
service) I see in the logs the following line:
"Could not find Hadoop configuration via any of the supported method"
If I'm not wrong, this means that the HADOOP_CONF_DIR is actually not
generated by the run scripts.
Indeed, If I add HADOOP_CONF_DIR and YARN_CONF_DIR (always under
environment section of the docker-compose service) I don't see that line.
Is this the expected behavior?
Below the relevant docker-compose service I use (I've removed the content
of HADOOP_CLASSPATH content because is too long and I didn't report the
taskmanager that is similar):
flink-jobmanager:
container_name: flink-jobmanager
build:
context: .
dockerfile: Dockerfile.flink
args:
FLINK_VERSION: 1.11.3-scala_2.12-java11
image: 'flink-test:1.11.3-scala_2.12-java11'
ports:
- "8091:8081"
- "8092:8082"
command: jobmanager
environment:
- |
FLINK_PROPERTIES=
jobmanager.rpc.address: flink-jobmanager
rest.port: 8081
historyserver.web.port: 8082
web.upload.dir: /opt/flink
env.hadoop.conf.dir: /opt/hadoop/conf
env.yarn.conf.dir: /opt/hadoop/conf
- |
HADOOP_CLASSPATH=...
- HADOOP_CONF_DIR=/opt/hadoop/conf
- YARN_CONF_DIR=/opt/hadoop/conf
volumes:
- 'flink_shared_folder:/tmp/test'
- 'flink_uploads:/opt/flink/flink-web-upload'
- 'flink_hadoop_conf:/opt/hadoop/conf'
- 'flink_hadoop_libs:/opt/hadoop-3.2.1/share'
Thanks in advance for any support,
Flavio
