Hi Folks, I'm running Beam Python (2.31.0) on Flink on Kubernetes. I'm using the PortableRunner and a beam job server. I'm using S3 for the artifacts dir. The job server is throwing exceptions like the one below complaining that filesystem scheme S3 isn't registered.
I'm using the stock job server container apache/beam_flink1.13_job_server:2.31.0. It doesn't look like the jar that ships in that server includes the AWS Filesystem classes. So I tried adding the jar beam-sdks-java-io-amazon-web-services-2.31.0.jar. Now when the job server loads I get a ClassNotFound exception for AWSCredentialsProvider because I'm missing some of the dependencies for that. Does anyone have recommendations on the easiest path to obtaining the jar(s) needed to support S3 for the artifact service? Is there an uber jar published somewhere or will I have to build it myself? Thanks. J Aug 06, 2021 1:52:51 AM org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$2 finishStaging SEVERE: Error staging artifacts java.util.concurrent.ExecutionException: java.lang.IllegalArgumentException: No filesystem found for scheme s3 at java.util.concurrent.FutureTask.report(FutureTask.java:122) at java.util.concurrent.FutureTask.get(FutureTask.java:192) at org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$2.finishStaging(ArtifactStagingService.java:461) at org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$2.resolveNextEnvironment(ArtifactStagingService.java:439) at org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$2.onNext(ArtifactStagingService.java:417) at org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$2.onNext(ArtifactStagingService.java:303) at org.apache.beam.vendor.grpc.v1p36p0.io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onMessage(ServerCalls.java:255) at org.apache.beam.vendor.grpc.v1p36p0.io.grpc.ForwardingServerCallListener.onMessage(ForwardingServerCallListener.java:33) at org.apache.beam.vendor.grpc.v1p36p0.io.grpc.Contexts$ContextualizedServerCallListener.onMessage(Contexts.java:76) at org.apache.beam.vendor.grpc.v1p36p0.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailableInternal(ServerCallImpl.java:309) at org.apache.beam.vendor.grpc.v1p36p0.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailable(ServerCallImpl.java:292) at org.apache.beam.vendor.grpc.v1p36p0.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1MessagesAvailable.runInContext(ServerImpl.java:765) at org.apache.beam.vendor.grpc.v1p36p0.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) at org.apache.beam.vendor.grpc.v1p36p0.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.IllegalArgumentException: No filesystem found for scheme s3 at org.apache.beam.sdk.io.FileSystems.getFileSystemInternal(FileSystems.java:497) at org.apache.beam.sdk.io.FileSystems.matchNewResource(FileSystems.java:567) at org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$1.stagingDir(ArtifactStagingService.java:193) at org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$1.getDestination(ArtifactStagingService.java:169) at org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$StoreArtifact.call(ArtifactStagingService.java:271) at org.apache.beam.runners.fnexecution.artifact.ArtifactStagingService$StoreArtifact.call(ArtifactStagingService.java:247) at java.util.concurrent.FutureTask.run(FutureTask.java:266) ... 3 more --- Here's the exception after adding the amazon web services Jar Module could not be instantiated at java.util.ServiceLoader.fail(ServiceLoader.java:232) at java.util.ServiceLoader.access$100(ServiceLoader.java:185) at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:384) at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404) at java.util.ServiceLoader$1.next(ServiceLoader.java:480) at com.fasterxml.jackson.databind.ObjectMapper.findModules(ObjectMapper.java:1081) at org.apache.beam.sdk.options.PipelineOptionsFactory.<clinit>(PipelineOptionsFactory.java:478) at org.apache.beam.runners.flink.FlinkJobServerDriver.main(FlinkJobServerDriver.java:72) Caused by: java.lang.NoClassDefFoundError: com/amazonaws/auth/AWSCredentialsProvider at org.apache.beam.sdk.io.aws.options.AwsModule.<init>(AwsModule.java:86) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at java.lang.Class.newInstance(Class.java:442) at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:380) ... 5 more Caused by: java.lang.ClassNotFoundException: com.amazonaws.auth.AWSCredentialsProvider at java.net.URLClassLoader.findClass(URLClassLoader.java:382) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) ... 12 more
