Here is a link to information on how to integrate Flink with Hadoop [1]. In the latest version you only need to point Flink to the Hadoop libraries via setting the HADOOP_CLASSPATH environment variable.
[1] https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/deployment/hadoop.html Cheers, Till On Thu, Aug 20, 2020 at 5:50 PM Till Rohrmann <trohrm...@apache.org> wrote: > I agree with Piotr's analysis. It should not matter whether you are using > RocksDBStateBackend or not. It seems as if you have a Hadoop dependency > clash. Could you check which dependencies are on the class path? > > Cheers, > Till > > On Thu, Aug 20, 2020 at 3:52 PM Piotr Nowojski <piotr.nowoj...@gmail.com> > wrote: > >> Hi, >> >> It looks more like a dependency convergence issue - you have two >> conflicting versions of >> `org.apache.hadoop.yarn.api.protocolrecords.AllocateRequest` >> on the class path. Or you built your jar with one version and trying to >> execute it with a different one. >> >> Till is it some kind of a known issue? >> >> Piotrek >> >> >> czw., 20 sie 2020 o 06:48 Vijayendra Yadav <contact....@gmail.com> >> napisał(a): >> >>> Hi Team, >>> >>> Getting the following error when using RocksDBStateBackend on yarn/EMR. >>> Am I missing any dependencies? >>> >>> >>> 2020-08-20 04:37:00,713 ERROR >>> org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl - >>> Exception on heartbeat >>> java.lang.NoSuchMethodError: >>> org.apache.hadoop.yarn.api.protocolrecords.AllocateRequest.newInstance(IFLjava/util/List;Ljava/util/List;Ljava/util/List;Lorg/apache/hadoop/yarn/api/records/ResourceBlacklistRequest;)Lorg/apache/hadoop/yarn/api/protocolrecords/AllocateRequest; >>> at >>> org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.allocate(AMRMClientImpl.java:280) >>> at >>> org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$HeartbeatThread.run(AMRMClientAsyncImpl.java:273) >>> 2020-08-20 04:37:00,714 INFO >>> org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl - >>> Interrupted while waiting for queue >>> java.lang.InterruptedException >>> at >>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014) >>> at >>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048) >>> at >>> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) >>> at >>> org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:323) >>> 2020-08-20 04:37:00,714 ERROR >>> org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl - >>> Stopping callback due to: >>> java.lang.NoSuchMethodError: >>> org.apache.hadoop.yarn.api.protocolrecords.AllocateRequest.newInstance(IFLjava/util/List;Ljava/util/List;Ljava/util/List;Lorg/apache/hadoop/yarn/api/records/ResourceBlacklistRequest;)Lorg/apache/hadoop/yarn/api/protocolrecords/AllocateRequest; >>> at >>> org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.allocate(AMRMClientImpl.java:280) >>> at >>> org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$HeartbeatThread.run(AMRMClientAsyncImpl.java:273) >>> 2020-08-20 04:37:00,714 ERROR org.apache.flink.yarn.YarnResourceManager >>> - Fatal error occurred in ResourceManager. >>> java.lang.NoSuchMethodError: >>> org.apache.hadoop.yarn.api.protocolrecords.AllocateRequest.newInstance(IFLjava/util/List;Ljava/util/List;Ljava/util/List;Lorg/apache/hadoop/yarn/api/records/ResourceBlacklistRequest;)Lorg/apache/hadoop/yarn/api/protocolrecords/AllocateRequest; >>> at >>> org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.allocate(AMRMClientImpl.java:280) >>> at >>> org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$HeartbeatThread.run(AMRMClientAsyncImpl.java:273) >>> 2020-08-20 04:37:00,714 ERROR >>> org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Fatal error >>> occurred in the cluster entrypoint. >>> java.lang.NoSuchMethodError: >>> org.apache.hadoop.yarn.api.protocolrecords.AllocateRequest.newInstance(IFLjava/util/List;Ljava/util/List;Ljava/util/List;Lorg/apache/hadoop/yarn/api/records/ResourceBlacklistRequest;)Lorg/apache/hadoop/yarn/api/protocolrecords/AllocateRequest; >>> at >>> org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.allocate(AMRMClientImpl.java:280) >>> at >>> org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$HeartbeatThread.run(AMRMClientAsyncImpl.java:273) >>> 2020-08-20 04:37:00,718 INFO org.apache.flink.runtime.blob.BlobServer >>> >>> - Stopped BLOB server at 0.0.0.0:45627 >>> >>> >>> Regards, >>> >>> Vijay >>> >>>
