Re: Flink HDFS Integration

2023-01-09 Thread Raihan Sunny
Hi Martijn, Thanks for the link, it worked. However, I don't think I quite got that from the documentation alone. Basically, I downloaded the Hadoop binaries from the official website, unpacked, ran `hadoop classpath` from within the extracted bin directory, and added the output as the environment

Re: Flink HDFS Integration

2023-01-09 Thread Martijn Visser
Hi Raihan, This is documented at https://nightlies.apache.org/flink/flink-docs-master/docs/dev/configuration/advanced/#hadoop-dependencies Best regards, Martijn On Mon, Jan 9, 2023 at 10:40 AM Raihan Sunny wrote: > Hi, > > I'm trying to integrate HDFS in Flink for checkpoint and savepoint > s

Flink HDFS Integration

2023-01-09 Thread Raihan Sunny
Hi, I'm trying to integrate HDFS in Flink for checkpoint and savepoint storage. I have an HDFS cluster running on Docker. I have made the following changes to the flink configuration: state.backend: filesystem state.savepoints.dir: hdfs://namenode:9000/user/root/savepoints state.checkpoints.dir: