Hi Alexandre, This seems to be complaining about the python script loading. It seems that the local file system is using `file` file prefix not `local`[1]. FYI inside your python script you can add more dependencies like connectors and so using python dependency management[2] which differs from Java's maven/gradle dependency management systems.
1- https://nightlies.apache.org/flink/flink-docs-release-1.20/docs/deployment/filesystems/overview/#local-file-system 2- https://nightlies.apache.org/flink/flink-docs-master/docs/dev/python/dependency_management Best Regards Ahmed Hamdy On Mon, 2 Sept 2024 at 15:42, Alexandre KY <alexandre...@magellium.fr> wrote: > Hello, > > I am trying to deploy my application on Amazon EMR on EKS. My application > is in Python and from what I have read, I must create my own custom image. > I tried to run a simple test with a very simple Flink job. Here is the > Flink application: > > ```python > # main.py > > import logging > import sys > > from pyflink.datastream import StreamExecutionEnvironment > > from pyflink_proto.docker.aws.test.Person import Person > > > def main(): > env = StreamExecutionEnvironment.get_execution_environment() > test = [ > Person("Fred", 35), > Person("Wilma", 35), > Person("Pebbles", 2) > ] > > flintstones = env.from_collection(test) > > adults = flintstones.filter( > lambda p: p.age >= 18 > ) > > adults.print() > > # submit for execution > env.execute() > > > if __name__ == '__main__': > logging.basicConfig(stream=sys.stdout, level=logging.INFO, > format="%(message)s") > > main() > ``` > > ```python > # Person.py > > class Person: > > def __init__(self, name, age): > self.name = name > self.age = age > > def __str__(self): > return '{}: age {}'.format(self.name, str(self.age)) > ``` > > Here is the Dockerfile used: > > ```dockerfile > FROM public.ecr.aws/emr-on-eks/flink/emr-7.0.0-flink:latest > USER root > RUN pip3 install numpy apache-flink==1.18.1 > USER hadoop:hadoop > ``` > > When I try to execute the Flink job in application mode using native > Kubernetes: > > ```shell > $FLINK_HOME/bin/flink run-application \ > --target kubernetes-application \ > -Dkubernetes.namespace=$NAMESPACE \ > -Dkubernetes.cluster-id=$EMR_CLUSTER \ > -Dkubernetes.container.image.ref=$IMAGE \ > -Dkubernetes.service-account=$FLINK_SERVICE_ACCOUNT \ > -Djobmanager.heap.size=1024m \ > -Dtaskmanager.memory.process.size=2048m \ > -py local:///opt/flink_job/test.py > ``` > > I get the following error: > > ```shell > ERROR org.apache.flink.client.python.PythonDriver [] - > Run python process failed > org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not > find a file system implementation for scheme 'local'. The scheme is not > directly supported by Flink and no Hadoop file system to support this > scheme could be loaded. For a full list of supported file systems, please > see https://nightlies.apache.org/flink/flink-docs-stable/ops/filesystems/. > at > org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:543) > ~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0] > at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:409) > ~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0] > at org.apache.flink.core.fs.Path.getFileSystem(Path.java:274) > ~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0] > at > org.apache.flink.client.python.PythonEnvUtils.addToPythonPath(PythonEnvUtils.java:291) > ~[flink-python-1.18.0-amzn-0.jar:1.18.0-amzn-0] > at > org.apache.flink.client.python.PythonEnvUtils.preparePythonEnvironment(PythonEnvUtils.java:226) > ~[flink-python-1.18.0-amzn-0.jar:1.18.0-amzn-0] > at > org.apache.flink.client.python.PythonEnvUtils.launchPy4jPythonClient(PythonEnvUtils.java:487) > ~[flink-python-1.18.0-amzn-0.jar:1.18.0-amzn-0] > at > org.apache.flink.client.python.PythonDriver.main(PythonDriver.java:92) > ~[flink-python-1.18.0-amzn-0.jar:1.18.0-amzn-0] > at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) ~[?:?] > at > jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > ~[?:?] > at > jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > ~[?:?] > at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?] > at > org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355) > ~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0] > at > org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222) > ~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0] > at > org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:105) > ~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0] > at > org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.runApplicationEntryPoint(ApplicationDispatcherBootstrap.java:301) > ~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0] > at > org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.lambda$runApplicationAsync$2(ApplicationDispatcherBootstrap.java:254) > ~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) > ~[?:?] > at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?] > at > org.apache.flink.runtime.concurrent.pekko.ActorSystemScheduledExecutorAdapter$ScheduledFutureTask.run(ActorSystemScheduledExecutorAdapter.java:172) > ~[?:?] > at > org.apache.flink.runtime.concurrent.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:68) > ~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0] > at > org.apache.flink.runtime.concurrent.ClassLoadingUtils.lambda$withContextClassLoader$0(ClassLoadingUtils.java:41) > ~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0] > at > org.apache.pekko.dispatch.TaskInvocation.run(AbstractDispatcher.scala:59) > [flink-rpc-akka73f86dfe-dc62-4aea-8884-6e4fccebb026.jar:1.18.0-amzn-0] > at > org.apache.pekko.dispatch.ForkJoinExecutorConfigurator$PekkoForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:57) > [flink-rpc-akka73f86dfe-dc62-4aea-8884-6e4fccebb026.jar:1.18.0-amzn-0] > at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) > [?:?] > at > java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020) > [?:?] > at java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656) [?:?] > at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594) > [?:?] > at > java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183) > [?:?] > Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: > Hadoop is not in the classpath/dependencies. > at > org.apache.flink.core.fs.UnsupportedSchemeFactory.create(UnsupportedSchemeFactory.java:55) > ~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0] > at > org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:526) > ~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0] > ... 27 more > 2024-09-02 12:31:06,606 WARN > org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap > [] - Application failed unexpectedly: > java.util.concurrent.CompletionException: > org.apache.flink.client.deployment.application.ApplicationExecutionException: > Could not execute application. > at > java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331) > ~[?:?] > at > java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346) > ~[?:?] > at > java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1063) > ~[?:?] > at > java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) > ~[?:?] > at > java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2094) > ~[?:?] > at > org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.runApplicationEntryPoint(ApplicationDispatcherBootstrap.java:337) > ~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0] > at > org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.lambda$runApplicationAsync$2(ApplicationDispatcherBootstrap.java:254) > ~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) > ~[?:?] > at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?] > at > org.apache.flink.runtime.concurrent.pekko.ActorSystemScheduledExecutorAdapter$ScheduledFutureTask.run(ActorSystemScheduledExecutorAdapter.java:172) > ~[?:?] > at > org.apache.flink.runtime.concurrent.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:68) > ~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0] > at > org.apache.flink.runtime.concurrent.ClassLoadingUtils.lambda$withContextClassLoader$0(ClassLoadingUtils.java:41) > ~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0] > at > org.apache.pekko.dispatch.TaskInvocation.run(AbstractDispatcher.scala:59) > [flink-rpc-akka73f86dfe-dc62-4aea-8884-6e4fccebb026.jar:1.18.0-amzn-0] > at > org.apache.pekko.dispatch.ForkJoinExecutorConfigurator$PekkoForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:57) > [flink-rpc-akka73f86dfe-dc62-4aea-8884-6e4fccebb026.jar:1.18.0-amzn-0] > at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) > [?:?] > at > java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020) > [?:?] > at java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656) [?:?] > at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594) > [?:?] > at > java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183) > [?:?] > Caused by: > org.apache.flink.client.deployment.application.ApplicationExecutionException: > Could not execute application. > ... 14 more > Caused by: org.apache.flink.client.program.ProgramAbortException: > org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not > find a file system implementation for scheme 'local'. The scheme is not > directly supported by Flink and no Hadoop file system to support this > scheme could be loaded. For a full list of supported file systems, please > see https://nightlies.apache.org/flink/flink-docs-stable/ops/filesystems/. > at > org.apache.flink.client.python.PythonDriver.main(PythonDriver.java:140) > ~[?:?] > at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) ~[?:?] > at > jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > ~[?:?] > at > jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > ~[?:?] > at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?] > at > org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355) > ~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0] > at > org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222) > ~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0] > at > org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:105) > ~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0] > at > org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.runApplicationEntryPoint(ApplicationDispatcherBootstrap.java:301) > ~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0] > ... 13 more > Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: > Could not find a file system implementation for scheme 'local'. The scheme > is not directly supported by Flink and no Hadoop file system to support > this scheme could be loaded. For a full list of supported file systems, > please see > https://nightlies.apache.org/flink/flink-docs-stable/ops/filesystems/. > at > org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:543) > ~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0] > at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:409) > ~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0] > at org.apache.flink.core.fs.Path.getFileSystem(Path.java:274) > ~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0] > at > org.apache.flink.client.python.PythonEnvUtils.addToPythonPath(PythonEnvUtils.java:291) > ~[?:?] > at > org.apache.flink.client.python.PythonEnvUtils.preparePythonEnvironment(PythonEnvUtils.java:226) > ~[?:?] > at > org.apache.flink.client.python.PythonEnvUtils.launchPy4jPythonClient(PythonEnvUtils.java:487) > ~[?:?] > at > org.apache.flink.client.python.PythonDriver.main(PythonDriver.java:92) > ~[?:?] > at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) ~[?:?] > at > jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > ~[?:?] > at > jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > ~[?:?] > at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?] > at > org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355) > ~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0] > at > org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222) > ~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0] > at > org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:105) > ~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0] > at > org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.runApplicationEntryPoint(ApplicationDispatcherBootstrap.java:301) > ~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0] > ... 13 more > Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: > Hadoop is not in the classpath/dependencies. > at > org.apache.flink.core.fs.UnsupportedSchemeFactory.create(UnsupportedSchemeFactory.java:55) > ~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0] > at > org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:526) > ~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0] > at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:409) > ~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0] > at org.apache.flink.core.fs.Path.getFileSystem(Path.java:274) > ~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0] > at > org.apache.flink.client.python.PythonEnvUtils.addToPythonPath(PythonEnvUtils.java:291) > ~[?:?] > at > org.apache.flink.client.python.PythonEnvUtils.preparePythonEnvironment(PythonEnvUtils.java:226) > ~[?:?] > at > org.apache.flink.client.python.PythonEnvUtils.launchPy4jPythonClient(PythonEnvUtils.java:487) > ~[?:?] > at > org.apache.flink.client.python.PythonDriver.main(PythonDriver.java:92) > ~[?:?] > at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) ~[?:?] > at > jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > ~[?:?] > at > jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > ~[?:?] > at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?] > at > org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355) > ~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0] > at > org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222) > ~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0] > at > org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:105) > ~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0] > at > org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.runApplicationEntryPoint(ApplicationDispatcherBootstrap.java:301) > ~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0] > ... 13 more > 2024-09-02 12:31:06,615 ERROR > org.apache.flink.runtime.entrypoint.ClusterEntrypoint [] - Fatal > error occurred in the cluster entrypoint. > java.util.concurrent.CompletionException: > org.apache.flink.client.deployment.application.ApplicationExecutionException: > Could not execute application. > at > java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331) > ~[?:?] > at > java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346) > ~[?:?] > at > java.util.concurrent.CompletableFuture$UniCompose.tryFire(CompletableFuture.java:1063) > ~[?:?] > at > java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) > ~[?:?] > at > java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2094) > ~[?:?] > at > org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.runApplicationEntryPoint(ApplicationDispatcherBootstrap.java:337) > ~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0] > at > org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.lambda$runApplicationAsync$2(ApplicationDispatcherBootstrap.java:254) > ~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) > ~[?:?] > at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?] > at > org.apache.flink.runtime.concurrent.pekko.ActorSystemScheduledExecutorAdapter$ScheduledFutureTask.run(ActorSystemScheduledExecutorAdapter.java:172) > ~[?:?] > at > org.apache.flink.runtime.concurrent.ClassLoadingUtils.runWithContextClassLoader(ClassLoadingUtils.java:68) > ~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0] > at > org.apache.flink.runtime.concurrent.ClassLoadingUtils.lambda$withContextClassLoader$0(ClassLoadingUtils.java:41) > ~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0] > at > org.apache.pekko.dispatch.TaskInvocation.run(AbstractDispatcher.scala:59) > [flink-rpc-akka73f86dfe-dc62-4aea-8884-6e4fccebb026.jar:1.18.0-amzn-0] > at > org.apache.pekko.dispatch.ForkJoinExecutorConfigurator$PekkoForkJoinTask.exec(ForkJoinExecutorConfigurator.scala:57) > [flink-rpc-akka73f86dfe-dc62-4aea-8884-6e4fccebb026.jar:1.18.0-amzn-0] > at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:290) > [?:?] > at > java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1020) > [?:?] > at java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:1656) [?:?] > at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1594) > [?:?] > at > java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:183) > [?:?] > Caused by: > org.apache.flink.client.deployment.application.ApplicationExecutionException: > Could not execute application. > ... 14 more > Caused by: org.apache.flink.client.program.ProgramAbortException: > org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not > find a file system implementation for scheme 'local'. The scheme is not > directly supported by Flink and no Hadoop file system to support this > scheme could be loaded. For a full list of supported file systems, please > see https://nightlies.apache.org/flink/flink-docs-stable/ops/filesystems/. > at > org.apache.flink.client.python.PythonDriver.main(PythonDriver.java:140) > ~[?:?] > at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) ~[?:?] > at > jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > ~[?:?] > at > jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > ~[?:?] > at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?] > at > org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355) > ~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0] > at > org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222) > ~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0] > at > org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:105) > ~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0] > at > org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.runApplicationEntryPoint(ApplicationDispatcherBootstrap.java:301) > ~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0] > ... 13 more > Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: > Could not find a file system implementation for scheme 'local'. The scheme > is not directly supported by Flink and no Hadoop file system to support > this scheme could be loaded. For a full list of supported file systems, > please see > https://nightlies.apache.org/flink/flink-docs-stable/ops/filesystems/. > at > org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:543) > ~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0] > at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:409) > ~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0] > at org.apache.flink.core.fs.Path.getFileSystem(Path.java:274) > ~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0] > at > org.apache.flink.client.python.PythonEnvUtils.addToPythonPath(PythonEnvUtils.java:291) > ~[?:?] > at > org.apache.flink.client.python.PythonEnvUtils.preparePythonEnvironment(PythonEnvUtils.java:226) > ~[?:?] > at > org.apache.flink.client.python.PythonEnvUtils.launchPy4jPythonClient(PythonEnvUtils.java:487) > ~[?:?] > at > org.apache.flink.client.python.PythonDriver.main(PythonDriver.java:92) > ~[?:?] > at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) ~[?:?] > at > jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > ~[?:?] > at > jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > ~[?:?] > at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?] > at > org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355) > ~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0] > at > org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222) > ~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0] > at > org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:105) > ~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0] > at > org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.runApplicationEntryPoint(ApplicationDispatcherBootstrap.java:301) > ~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0] > ... 13 more > Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: > Hadoop is not in the classpath/dependencies. > at > org.apache.flink.core.fs.UnsupportedSchemeFactory.create(UnsupportedSchemeFactory.java:55) > ~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0] > at > org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:526) > ~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0] > at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:409) > ~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0] > at org.apache.flink.core.fs.Path.getFileSystem(Path.java:274) > ~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0] > at > org.apache.flink.client.python.PythonEnvUtils.addToPythonPath(PythonEnvUtils.java:291) > ~[?:?] > at > org.apache.flink.client.python.PythonEnvUtils.preparePythonEnvironment(PythonEnvUtils.java:226) > ~[?:?] > at > org.apache.flink.client.python.PythonEnvUtils.launchPy4jPythonClient(PythonEnvUtils.java:487) > ~[?:?] > at > org.apache.flink.client.python.PythonDriver.main(PythonDriver.java:92) > ~[?:?] > at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native > Method) ~[?:?] > at > jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > ~[?:?] > at > jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > ~[?:?] > at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?] > at > org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355) > ~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0] > at > org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222) > ~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0] > at > org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:105) > ~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0] > at > org.apache.flink.client.deployment.application.ApplicationDispatcherBootstrap.runApplicationEntryPoint(ApplicationDispatcherBootstrap.java:301) > ~[flink-dist-1.18.0-amzn-0.jar:1.18.0-amzn-0] > > ``` > > I tested the Flink job locally and it perfectly worker. I even tried to > translate it in Java and run using the AWS EMR_on_EKS_Flin container and it > also workerd. So I do not understand why it does not work in PyFlink. > Moreover, this error is related to the file system, but I never make any > call to it. And even if I did, why does it work in Java and not in Python ? > > > Best regards, > > KY Alexandre >
