Re: About[jobmanager.memory.off-heap.size] not work on Flink(flink-session on yarn)

Hi, wenjiang It does seem a bit odd that it could theoretically work either way. I think you can check the Flink JobManager log to find more information. Best, Ron 傅文江 于2023年7月24日周一 17:37写道： > > When I use ESSink, I find that I need to set Flink’s > jobmanager.memory.off-heap.size to 256MB.

About[jobmanager.memory.off-heap.size] not work on Flink(flink-session on yarn)

When I use ESSink, I find that I need to set Flink’s jobmanager.memory.off-heap.size to 256MB. The default 128MB does not meet the running requirements. It can be successful by modifying the configuration file, but I want to dynamically adjust the jobmanager.memory.off-heap.size through the c

Re: Flink session on Yarn - ClassNotFoundException

Hi Albert, Your thread is really helpful for flink setup on HDInsight cluster. I followed the same and now I hit another error: *Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.conf.Configuration.reloadExistingConfigurations()V at org.apache.hadoop.fs.adl.AdlConf

Re: Flink session on yarn

oop.yarn.client.ClientRMProxy.createRMProxy(ClientRMProxy.java:72)* > > > I referred this mail thread also: http://apache-flink- > user-mailing-list-archive.2336050.n4.nabble.com/Flink-session-on-Yarn- > ClassNotFoundException-td15222.html > > I am using HDInsight Clu

Re: Flink session on yarn

RMProxy(RMProxy.java:94)* > *        at > org.apache.hadoop.yarn.client.ClientRMProxy.createRMProxy(ClientRMProxy.java:72)* > > > I referred this mail thread > also:  > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-session-on-Yarn-ClassNotFoundExcept

Flink session on yarn

flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-session-on-Yarn-ClassNotFoundException-td15222.html I am using HDInsight Cluster(HDP3.6). According to the documentation, set HADOOP_CONF_DIR and YARN_CONF_DIR as well. Any inputs will be really helpful. Thanks! -- Thanks & Regards, Nishu Tayal

Re: Flink session on Yarn - ClassNotFoundException

Thanks for the replies :) I managed to get it working following the instructions here , but I found a few issues that I guess were specific to HDInsight, or at least to the HDP version it u

Re: Flink session on Yarn - ClassNotFoundException

We had the same issue. Get the hdp version, from /usr/hdp/current/hadoop-client/hadoop-common-.jar for example. Then rebuild flink from src: mvn clean install -DskipTests -Pvendor-repos -Dhadoop.version= for example: mvn clean install -DskipTests -Pvendor-repos -Dhadoop.version=2.7.3.2.6.1.0-129

Re: Flink session on Yarn - ClassNotFoundException

Hi, What is your "hadoop version" output? I'm asking because you said your hadoop distribution is in /usr/hdp so it looks like you're using Hortonworks HDP, just like myself. So, this would be a third party distribution and you'd need to build Flink from source according to this: https://ci.apache.

Re: Flink session on Yarn - ClassNotFoundException

Hi Chesnay, Thanks for your reply. I did download the binaries matching my Hadoop version (2.7), that's why I was wondering if the issue had something to do with the exact hadoop version flink is compiled again, or if there might be things that are missing in my environment. -- Sent from: http:

Re: Flink session on Yarn - ClassNotFoundException

Hello, The ClassNotFoundException indicates that you are using a Flink version that wasn't compiled against hadoop 2.7. Replacing one part of the hadoop dependency will most likely not cut it (or fail mysteriously down the line), so i would suggest to check the downloads

Flink session on Yarn - ClassNotFoundException

Hi, I’m trying to start a flink (1.3.2) session as explained in the docs (https://ci.apache.org/projects/flink/flink-docs-release-1.3/setup/yarn_setup.html#start-a-session ), but I keep getting t