Great! Thanks for the support On Thu, Apr 22, 2021 at 2:57 PM Matthias Pohl <matth...@ververica.com> wrote:
> I think you're right, Flavio. I created FLINK-22414 to cover this. Thanks > for bringing it up. > > Matthias > > [1] https://issues.apache.org/jira/browse/FLINK-22414 > > On Fri, Apr 16, 2021 at 9:32 AM Flavio Pompermaier <pomperma...@okkam.it> > wrote: > >> Hi Yang, >> isn't this something to fix? If I look at the documentation at [1], in >> the "Passing configuration via environment variables" section, there is: >> >> "The environment variable FLINK_PROPERTIES should contain a list of Flink >> cluster configuration options separated by new line, >> the same way as in the flink-conf.yaml. FLINK_PROPERTIES takes precedence >> over configurations in flink-conf.yaml." >> >> To me this means that if I specify "env.hadoop.conf.dir" it should be >> handled as well. Am I wrong? >> >> [1] >> https://ci.apache.org/projects/flink/flink-docs-stable/deployment/resource-providers/standalone/docker.html >> >> Best, >> Flavio >> >> On Fri, Apr 16, 2021 at 4:52 AM Yang Wang <danrtsey...@gmail.com> wrote: >> >>> It seems that we do not export HADOOP_CONF_DIR as environment variables >>> in current implementation, even though we have set the env.xxx flink config >>> options. It is only used to construct the classpath for the JM/TM process. >>> However, in "HadoopUtils"[2] we do not support getting the hadoop >>> configuration from classpath. >>> >>> >>> [1]. >>> https://github.com/apache/flink/blob/release-1.11/flink-dist/src/main/flink-bin/bin/config.sh#L256 >>> [2]. >>> https://github.com/apache/flink/blob/release-1.11/flink-connectors/flink-hadoop-compatibility/src/main/java/org/apache/flink/api/java/hadoop/mapred/utils/HadoopUtils.java#L64 >>> >>> >>> Best, >>> Yang >>> >>> Best, >>> Yang >>> >>> Flavio Pompermaier <pomperma...@okkam.it> 于2021年4月16日周五 上午3:55写道： >>> >>>> Hi Robert, >>>> indeed my docker-compose does work only if I add also Hadoop and yarn >>>> home while I was expecting that those two variables were generated >>>> automatically just setting env.xxx variables in FLINK_PROPERTIES variable.. >>>> >>>> I just want to understand what to expect, if I really need to specify >>>> Hadoop and yarn home as env variables or not >>>> >>>> Il gio 15 apr 2021, 20:39 Robert Metzger <rmetz...@apache.org> ha >>>> scritto: >>>> >>>>> Hi, >>>>> >>>>> I'm not aware of any known issues with Hadoop and Flink on Docker. >>>>> >>>>> I also tried what you are doing locally, and it seems to work: >>>>> >>>>> flink-jobmanager | 2021-04-15 18:37:48,300 INFO >>>>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - >>>>> Starting >>>>> StandaloneSessionClusterEntrypoint. >>>>> flink-jobmanager | 2021-04-15 18:37:48,338 INFO >>>>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Install >>>>> default filesystem. >>>>> flink-jobmanager | 2021-04-15 18:37:48,375 INFO >>>>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Install >>>>> security context. >>>>> flink-jobmanager | 2021-04-15 18:37:48,404 INFO >>>>> org.apache.flink.runtime.security.modules.HadoopModule [] - Hadoop >>>>> user set to flink (auth:SIMPLE) >>>>> flink-jobmanager | 2021-04-15 18:37:48,408 INFO >>>>> org.apache.flink.runtime.security.modules.JaasModule [] - Jaas >>>>> file will be created as /tmp/jaas-811306162058602256.conf. >>>>> flink-jobmanager | 2021-04-15 18:37:48,415 INFO >>>>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - >>>>> Initializing cluster services. >>>>> >>>>> Here's my code: >>>>> >>>>> https://gist.github.com/rmetzger/0cf4ba081d685d26478525bf69c7bd39 >>>>> >>>>> Hope this helps! >>>>> >>>>> On Wed, Apr 14, 2021 at 5:37 PM Flavio Pompermaier < >>>>> pomperma...@okkam.it> wrote: >>>>> >>>>>> Hi everybody, >>>>>> I'm trying to set up reading from HDFS using docker-compose and Flink >>>>>> 1.11.3. >>>>>> If I pass 'env.hadoop.conf.dir' and 'env.yarn.conf.dir' >>>>>> using FLINK_PROPERTIES (under environment section of the docker-compose >>>>>> service) I see in the logs the following line: >>>>>> >>>>>> "Could not find Hadoop configuration via any of the supported method" >>>>>> >>>>>> If I'm not wrong, this means that the HADOOP_CONF_DIR is actually not >>>>>> generated by the run scripts. >>>>>> Indeed, If I add HADOOP_CONF_DIR and YARN_CONF_DIR (always under >>>>>> environment section of the docker-compose service) I don't see that line. >>>>>> >>>>>> Is this the expected behavior? >>>>>> >>>>>> Below the relevant docker-compose service I use (I've removed the >>>>>> content of HADOOP_CLASSPATH content because is too long and I didn't >>>>>> report >>>>>> the taskmanager that is similar): >>>>>> >>>>>> flink-jobmanager: >>>>>> container_name: flink-jobmanager >>>>>> build: >>>>>> context: . >>>>>> dockerfile: Dockerfile.flink >>>>>> args: >>>>>> FLINK_VERSION: 1.11.3-scala_2.12-java11 >>>>>> image: 'flink-test:1.11.3-scala_2.12-java11' >>>>>> ports: >>>>>> - "8091:8081" >>>>>> - "8092:8082" >>>>>> command: jobmanager >>>>>> environment: >>>>>> - | >>>>>> FLINK_PROPERTIES= >>>>>> jobmanager.rpc.address: flink-jobmanager >>>>>> rest.port: 8081 >>>>>> historyserver.web.port: 8082 >>>>>> web.upload.dir: /opt/flink >>>>>> env.hadoop.conf.dir: /opt/hadoop/conf >>>>>> env.yarn.conf.dir: /opt/hadoop/conf >>>>>> - | >>>>>> HADOOP_CLASSPATH=... >>>>>> - HADOOP_CONF_DIR=/opt/hadoop/conf >>>>>> - YARN_CONF_DIR=/opt/hadoop/conf >>>>>> volumes: >>>>>> - 'flink_shared_folder:/tmp/test' >>>>>> - 'flink_uploads:/opt/flink/flink-web-upload' >>>>>> - 'flink_hadoop_conf:/opt/hadoop/conf' >>>>>> - 'flink_hadoop_libs:/opt/hadoop-3.2.1/share' >>>>>> >>>>>> >>>>>> Thanks in advance for any support, >>>>>> Flavio >>>>>> >>>>> >
