Hello,
I am trying to deploy my application on Amazon EMR on EKS. My application is in
Python and from what I have read, I must create my own custom image. I tried to
run a simple test with a very simple Flink job. Here is the Flink application:
```python
# main.py
import logging
import sys
from pyflink.datastream import StreamExecutionEnvironment
from pyflink_proto.docker.aws.test.Person import Person
def main():
env = StreamExecutionEnvironment.get_execution_environment()
test = [
Person("Fred", 35),
Person("Wilma", 35),
Person("Pebbles", 2)
]
flintstones = env.from_collection(test)
adults = flintstones.filter(
lambda p: p.age >= 18
)
adults.print()
# submit for execution
env.execute()
if __name__ == '__main__':
logging.basicConfig(stream=sys.stdout, level=logging.INFO,
format="%(message)s")
main()
```
```python
# Person.py
class Person:
def __init__(self, name, age):
self.name = name
self.age = age
def __str__(self):
return '{}: age {}'.format(self.name, str(self.age))
```
Here is the Dockerfile used:
```dockerfile
FROM public.ecr.aws/emr-on-eks/flink/emr-7.0.0-flink:latest
USER root
RUN pip3 install numpy apache-flink==1.18.1
USER hadoop:hadoop
```
When I try to execute the Flink job in application mode using native Kubernetes:
```shell
$FLINK_HOME/bin/flink run-application \
--target kubernetes-application \
-Dkubernetes.namespace=$NAMESPACE \
-Dkubernetes.cluster-id=$EMR_CLUSTER \
-Dkubernetes.container.image.ref=$IMAGE \
-Dkubernetes.service-account=$FLINK_SERVICE_ACCOUNT \
-Djobmanager.heap.size=1024m \
-Dtaskmanager.memory.process.size=2048m \
-py local:///opt/flink_job/test.py
```
I get the following error:
```shell
ERROR org.apache.flink.client.python.PythonDriver [] - Run
python process failed
org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not find a
file system implementation for scheme 'local'. The scheme is not directly
supported by Flink and no Hadoop file system to support this scheme could be
loaded. For a full list of supported file systems, please see
https://nightlies.apache.org/flink/flink-docs-stable/ops/filesystems/.
at
org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:543)
~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0]
at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:409)
~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0]
at org.apache.flink.core.fs.Path.getFileSystem(Path.java:274)
~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0]
at
org.apache.flink.client.python.PythonEnvUtils.addToPythonPath(PythonEnvUtils.java:291)
~[flink-python-1.18.0-amzn-0.jar:1.18.0-amzn-0]
at
org.apache.flink.client.python.PythonEnvUtils.preparePythonEnvironment(PythonEnvUtils.java:226)
~[flink-python-1.18.0-amzn-0.jar:1.18.0-amzn-0]
at
org.apache.flink.client.python.PythonEnvUtils.launchPy4jPythonClient(PythonEnvUtils.java:487)
~[flink-python-1.18.0-amzn-0.jar:1.18.0-amzn-0]
at org.apache.flink.client.python.PythonDriver.main(PythonDriver.java:92)
~[flink-python-1.18.0-amzn-0.jar:1.18.0-amzn-0]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:?]
at
jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:?]
at
jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:?]
at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355)
~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0]
at
org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0]
at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:105)
~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0]
at
org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.runApplicationEntryPoint(ApplicationDispatcherBootstrap.java:301)
~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0]
at
org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.lambda$runApplicationAsync$2(ApplicationDispatcherBootstrap.java:254)
~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
~[?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
at
org.apache.flink.runtime.concurrent.pekko.ActorSystemScheduledExecutorAdapter$ScheduledFutureTask.run(ActorSystemScheduledExecutorAdapter.java:172)
~[?:?]
at
org.apache.flink.runtime.concurrent.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:68)
~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0]
at
org.apache.flink.runtime.concurrent.ClassLoadingUtils.lambda$withContextClassLoader$0(ClassLoadingUtils.java:41)
~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0]
at
org.apache.pekko.dispatch.TaskInvocation.run(AbstractDispatcher.scala:59)
[flink-rpc-akka73f86dfe-dc62-4aea-8884-6e4fccebb026.jar:1.18.0-amzn-0]
at
org.apache.pekko.dispatch.ForkJoinExecutorConfigurator$PekkoForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:57)
[flink-rpc-akka73f86dfe-dc62-4aea-8884-6e4fccebb026.jar:1.18.0-amzn-0]
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) [?:?]
at
java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
[?:?]
at java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656) [?:?]
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594) [?:?]
at
java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
[?:?]
Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException:
Hadoop is not in the classpath/dependencies.
at
org.apache.flink.core.fs.UnsupportedSchemeFactory.create(UnsupportedSchemeFactory.java:55)
~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0]
at
org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:526)
~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0]
... 27 more
2024-09-02 12:31:06,606 WARN
org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap
[] - Application failed unexpectedly:
java.util.concurrent.CompletionException:
org.apache.flink.client.deployment.application.ApplicationExecutionException:
Could not execute application.
at
java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331)
~[?:?]
at
java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346)
~[?:?]
at
java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1063)
~[?:?]
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
~[?:?]
at
java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2094)
~[?:?]
at
org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.runApplicationEntryPoint(ApplicationDispatcherBootstrap.java:337)
~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0]
at
org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.lambda$runApplicationAsync$2(ApplicationDispatcherBootstrap.java:254)
~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
~[?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
at
org.apache.flink.runtime.concurrent.pekko.ActorSystemScheduledExecutorAdapter$ScheduledFutureTask.run(ActorSystemScheduledExecutorAdapter.java:172)
~[?:?]
at
org.apache.flink.runtime.concurrent.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:68)
~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0]
at
org.apache.flink.runtime.concurrent.ClassLoadingUtils.lambda$withContextClassLoader$0(ClassLoadingUtils.java:41)
~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0]
at
org.apache.pekko.dispatch.TaskInvocation.run(AbstractDispatcher.scala:59)
[flink-rpc-akka73f86dfe-dc62-4aea-8884-6e4fccebb026.jar:1.18.0-amzn-0]
at
org.apache.pekko.dispatch.ForkJoinExecutorConfigurator$PekkoForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:57)
[flink-rpc-akka73f86dfe-dc62-4aea-8884-6e4fccebb026.jar:1.18.0-amzn-0]
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) [?:?]
at
java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
[?:?]
at java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656) [?:?]
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594) [?:?]
at
java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
[?:?]
Caused by:
org.apache.flink.client.deployment.application.ApplicationExecutionException:
Could not execute application.
... 14 more
Caused by: org.apache.flink.client.program.ProgramAbortException:
org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not find a
file system implementation for scheme 'local'. The scheme is not directly
supported by Flink and no Hadoop file system to support this scheme could be
loaded. For a full list of supported file systems, please see
https://nightlies.apache.org/flink/flink-docs-stable/ops/filesystems/.
at org.apache.flink.client.python.PythonDriver.main(PythonDriver.java:140)
~[?:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:?]
at
jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:?]
at
jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:?]
at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355)
~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0]
at
org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0]
at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:105)
~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0]
at
org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.runApplicationEntryPoint(ApplicationDispatcherBootstrap.java:301)
~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0]
... 13 more
Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could
not find a file system implementation for scheme 'local'. The scheme is not
directly supported by Flink and no Hadoop file system to support this scheme
could be loaded. For a full list of supported file systems, please see
https://nightlies.apache.org/flink/flink-docs-stable/ops/filesystems/.
at
org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:543)
~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0]
at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:409)
~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0]
at org.apache.flink.core.fs.Path.getFileSystem(Path.java:274)
~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0]
at
org.apache.flink.client.python.PythonEnvUtils.addToPythonPath(PythonEnvUtils.java:291)
~[?:?]
at
org.apache.flink.client.python.PythonEnvUtils.preparePythonEnvironment(PythonEnvUtils.java:226)
~[?:?]
at
org.apache.flink.client.python.PythonEnvUtils.launchPy4jPythonClient(PythonEnvUtils.java:487)
~[?:?]
at org.apache.flink.client.python.PythonDriver.main(PythonDriver.java:92)
~[?:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:?]
at
jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:?]
at
jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:?]
at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355)
~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0]
at
org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0]
at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:105)
~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0]
at
org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.runApplicationEntryPoint(ApplicationDispatcherBootstrap.java:301)
~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0]
... 13 more
Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException:
Hadoop is not in the classpath/dependencies.
at
org.apache.flink.core.fs.UnsupportedSchemeFactory.create(UnsupportedSchemeFactory.java:55)
~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0]
at
org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:526)
~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0]
at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:409)
~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0]
at org.apache.flink.core.fs.Path.getFileSystem(Path.java:274)
~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0]
at
org.apache.flink.client.python.PythonEnvUtils.addToPythonPath(PythonEnvUtils.java:291)
~[?:?]
at
org.apache.flink.client.python.PythonEnvUtils.preparePythonEnvironment(PythonEnvUtils.java:226)
~[?:?]
at
org.apache.flink.client.python.PythonEnvUtils.launchPy4jPythonClient(PythonEnvUtils.java:487)
~[?:?]
at org.apache.flink.client.python.PythonDriver.main(PythonDriver.java:92)
~[?:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:?]
at
jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:?]
at
jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:?]
at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355)
~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0]
at
org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0]
at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:105)
~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0]
at
org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.runApplicationEntryPoint(ApplicationDispatcherBootstrap.java:301)
~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0]
... 13 more
2024-09-02 12:31:06,615 ERROR
org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Fatal error
occurred in the cluster entrypoint.
java.util.concurrent.CompletionException:
org.apache.flink.client.deployment.application.ApplicationExecutionException:
Could not execute application.
at
java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331)
~[?:?]
at
java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346)
~[?:?]
at
java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1063)
~[?:?]
at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
~[?:?]
at
java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2094)
~[?:?]
at
org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.runApplicationEntryPoint(ApplicationDispatcherBootstrap.java:337)
~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0]
at
org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.lambda$runApplicationAsync$2(ApplicationDispatcherBootstrap.java:254)
~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
~[?:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
at
org.apache.flink.runtime.concurrent.pekko.ActorSystemScheduledExecutorAdapter$ScheduledFutureTask.run(ActorSystemScheduledExecutorAdapter.java:172)
~[?:?]
at
org.apache.flink.runtime.concurrent.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:68)
~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0]
at
org.apache.flink.runtime.concurrent.ClassLoadingUtils.lambda$withContextClassLoader$0(ClassLoadingUtils.java:41)
~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0]
at
org.apache.pekko.dispatch.TaskInvocation.run(AbstractDispatcher.scala:59)
[flink-rpc-akka73f86dfe-dc62-4aea-8884-6e4fccebb026.jar:1.18.0-amzn-0]
at
org.apache.pekko.dispatch.ForkJoinExecutorConfigurator$PekkoForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:57)
[flink-rpc-akka73f86dfe-dc62-4aea-8884-6e4fccebb026.jar:1.18.0-amzn-0]
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) [?:?]
at
java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020)
[?:?]
at java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656) [?:?]
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594) [?:?]
at
java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183)
[?:?]
Caused by:
org.apache.flink.client.deployment.application.ApplicationExecutionException:
Could not execute application.
... 14 more
Caused by: org.apache.flink.client.program.ProgramAbortException:
org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not find a
file system implementation for scheme 'local'. The scheme is not directly
supported by Flink and no Hadoop file system to support this scheme could be
loaded. For a full list of supported file systems, please see
https://nightlies.apache.org/flink/flink-docs-stable/ops/filesystems/.
at org.apache.flink.client.python.PythonDriver.main(PythonDriver.java:140)
~[?:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:?]
at
jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:?]
at
jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:?]
at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355)
~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0]
at
org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0]
at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:105)
~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0]
at
org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.runApplicationEntryPoint(ApplicationDispatcherBootstrap.java:301)
~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0]
... 13 more
Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could
not find a file system implementation for scheme 'local'. The scheme is not
directly supported by Flink and no Hadoop file system to support this scheme
could be loaded. For a full list of supported file systems, please see
https://nightlies.apache.org/flink/flink-docs-stable/ops/filesystems/.
at
org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:543)
~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0]
at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:409)
~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0]
at org.apache.flink.core.fs.Path.getFileSystem(Path.java:274)
~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0]
at
org.apache.flink.client.python.PythonEnvUtils.addToPythonPath(PythonEnvUtils.java:291)
~[?:?]
at
org.apache.flink.client.python.PythonEnvUtils.preparePythonEnvironment(PythonEnvUtils.java:226)
~[?:?]
at
org.apache.flink.client.python.PythonEnvUtils.launchPy4jPythonClient(PythonEnvUtils.java:487)
~[?:?]
at org.apache.flink.client.python.PythonDriver.main(PythonDriver.java:92)
~[?:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:?]
at
jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:?]
at
jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:?]
at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355)
~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0]
at
org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0]
at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:105)
~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0]
at
org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.runApplicationEntryPoint(ApplicationDispatcherBootstrap.java:301)
~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0]
... 13 more
Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException:
Hadoop is not in the classpath/dependencies.
at
org.apache.flink.core.fs.UnsupportedSchemeFactory.create(UnsupportedSchemeFactory.java:55)
~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0]
at
org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:526)
~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0]
at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:409)
~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0]
at org.apache.flink.core.fs.Path.getFileSystem(Path.java:274)
~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0]
at
org.apache.flink.client.python.PythonEnvUtils.addToPythonPath(PythonEnvUtils.java:291)
~[?:?]
at
org.apache.flink.client.python.PythonEnvUtils.preparePythonEnvironment(PythonEnvUtils.java:226)
~[?:?]
at
org.apache.flink.client.python.PythonEnvUtils.launchPy4jPythonClient(PythonEnvUtils.java:487)
~[?:?]
at org.apache.flink.client.python.PythonDriver.main(PythonDriver.java:92)
~[?:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:?]
at
jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:?]
at
jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:?]
at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
at
org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355)
~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0]
at
org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0]
at org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:105)
~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0]
at
org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.runApplicationEntryPoint(ApplicationDispatcherBootstrap.java:301)
~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0]
```
I tested the Flink job locally and it perfectly worker. I even tried to
translate it in Java and run using the AWS EMR_on_EKS_Flin container and it
also workerd. So I do not understand why it does not work in PyFlink. Moreover,
this error is related to the file system, but I never make any call to it. And
even if I did, why does it work in Java and not in Python ?
Best regards,
KY Alexandre
