But looking at the code of the exception <https://github.com/apache/beam/blob/master/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/DefaultJobBundleFactory.java#L252> it seems that it tries to use docker only because it thinks it's in a docker environment, no? Shouldn't it use the external environment (that's available there as well <https://github.com/apache/beam/blob/master/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/DefaultJobBundleFactory.java#L252>) if it got that in PipelineOptions?
Otherwise, how can I have docker on a k8s pod? I couldn't seem to find any examples for that and saw that it's highly unrecommended. Thanks! Nir On Tue, May 4, 2021 at 9:08 PM Chamikara Jayalath <chamik...@google.com> wrote: > Ah, I think you need the DOCKER environment to use cross-language > transforms not the EXTERNAL environment (agree that the terminology is > confusing). > > On Tue, May 4, 2021 at 11:04 AM Nir Gazit <nir....@gmail.com> wrote: > >> Yes that’s on purpose. I’m running in Kubernetes which makes it hard to >> install docker on the pods so I don’t want to use the docker environment. >> That’s why I specified EXTERNAL environment in PipelineOptions. However, it >> seems that it doesn’t get propagated. >> >> On Tue, 4 May 2021 at 20:59 Chamikara Jayalath <chamik...@google.com> >> wrote: >> >>> Is it possible that you don't have the "docker" command available in >>> your system ? >>> >>> On Tue, May 4, 2021 at 10:28 AM Nir Gazit <nir....@gmail.com> wrote: >>> >>>> Hey, >>>> I'm trying to run a pipeline with a Kafka Source, using an EXTERNAL >>>> environment. However, when the pipeline is run, the error below is thrown, >>>> which implies that for some reason the external environment pipeline >>>> options didn't get in. When replacing the Kafka Source with an S3 source >>>> (for example) I don't get this error so it implies that it's >>>> somewhere around the external transform / expansion service area. How can I >>>> debug this? >>>> >>>> Caused by: java.io.IOException: Cannot run program "docker": error=2, >>>> No such file or directory >>>> at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048) >>>> at >>>> org.apache.beam.runners.fnexecution.environment.DockerCommand.runShortCommand(DockerCommand.java:189) >>>> at >>>> org.apache.beam.runners.fnexecution.environment.DockerCommand.runShortCommand(DockerCommand.java:171) >>>> at >>>> org.apache.beam.runners.fnexecution.environment.DockerCommand.runImage(DockerCommand.java:95) >>>> at >>>> org.apache.beam.runners.fnexecution.environment.DockerEnvironmentFactory.createEnvironment(DockerEnvironmentFactory.java:131) >>>> at >>>> org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory$1.load(DefaultJobBundleFactory.java:252) >>>> at >>>> org.apache.beam.runners.fnexecution.control.DefaultJobBundleFactory$1.load(DefaultJobBundleFactory.java:231) >>>> >>>
