Hi Martijn, Thanks for the link, it worked. However, I don't think I quite got that from the documentation alone. Basically, I downloaded the Hadoop binaries from the official website, unpacked, ran `hadoop classpath` from within the extracted bin directory, and added the output as the environment variable HADOOP_CLASSPATH to the jobmanager and taskmanager containers.
Thanks, Sunny On Mon, Jan 9, 2023 at 4:16 PM Martijn Visser <martijnvis...@apache.org> wrote: > Hi Raihan, > > This is documented at > https://nightlies.apache.org/flink/flink-docs-master/docs/dev/configuration/advanced/#hadoop-dependencies > > Best regards, > > Martijn > > On Mon, Jan 9, 2023 at 10:40 AM Raihan Sunny <raihan.su...@selise.ch> > wrote: > >> Hi, >> >> I'm trying to integrate HDFS in Flink for checkpoint and savepoint >> storage. I have an HDFS cluster running on Docker. I have made the >> following changes to the flink configuration: >> >> state.backend: filesystem >> state.savepoints.dir: hdfs://namenode:9000/user/root/savepoints >> state.checkpoints.dir: hdfs://namenode:9000/user/root/checkpoints >> >> The docker-compose.yml file contains the services for setting up the >> Hadoop cluster as well the Flink jobmanager and taskmanger containers. When >> I try to run the job, I get the errors: >> >> org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not >> find a file system implementation for scheme 'hdfs'. The scheme is not >> directly supported by Flink and no Hadoop file system to support this >> scheme could be loaded. >> >> org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Hadoop is >> not in the classpath/dependencies. >> >> >> How do I add Hadoop to the classpath? Note that I do not have Hadoop on >> the Flink jobmanager or taskmanager containers. Do I need to install it or >> does Flink have some built-in mechanisms? >> >> Version info: >> PyFlink on Flink 1.16 >> Hadoop 3.2.1 >> >> >> >> >> >> Secure Link Services Group >> Zürich: The Circle 37, 8058 Zürich-Airport, Switzerland >> Munich: Tal 44, 80331 München, Germany >> Dubai: Building 3, 3rd Floor, Dubai Design District, Dubai, United Arab >> Emirates >> Dhaka: Midas Center, Road 16, Dhanmondi, Dhaka 1209, Bangladesh >> Thimphu: Bhutan Innovation Tech Center, Babesa, P.O. Box 633, Thimphu, >> Bhutan >> >> Visit us: www.selise.ch >> >> *Important Note: This e-mail and any attachment are confidential and may >> contain trade secrets and may well also be legally privileged or otherwise >> protected from disclosure. If you have received it in error, you are on >> notice of its status. Please notify us immediately by reply e-mail and then >> delete this e-mail and any attachment from your system. If you are not the >> intended recipient please understand that you must not copy this e-mail or >> any attachment or disclose the contents to any other person. Thank you for >> your cooperation.* >> > -- Secure Link Services Group Zürich: The Circle 37, 8058 Zürich-Airport, Switzerland Munich: Tal 44, 80331 München, Germany Dubai: Building 3, 3rd Floor, Dubai Design District, Dubai, United Arab Emirates Dhaka: Midas Center, Road 16, Dhanmondi, Dhaka 1209, Bangladesh Thimphu: Bhutan Innovation Tech Center, Babesa, P.O. Box 633, Thimphu, Bhutan Visit us: www.selise.ch <http://www.selise.ch> -- *Important Note: This e-mail and any attachment are confidential and may contain trade secrets and may well also be legally privileged or otherwise protected from disclosure. If you have received it in error, you are on notice of its status. Please notify us immediately by reply e-mail and then delete this e-mail and any attachment from your system. If you are not the intended recipient please understand that you must not copy this e-mail or any attachment or disclose the contents to any other person. Thank you for your cooperation.*
