Hi Yilun! I made a quick proof of concept repo showcasing how to run a beam pipeline in flink on k8s. It may be useful for you as reference.
https://github.com/sambvfx/beam-flink-k8s On Wed, Feb 24, 2021, 8:13 AM yilun zhang <ilyak1...@gmail.com> wrote: > Hey, > > Our team is trying to use beam with connector Kafka and runner flink to > gather information and process data. We adopt python sdk and build in java > 11 in python 3.7 sdk image as java runtime for kafka expansion service. > so : > image: beam python 3.7 docker image + build in java 11 > connector: kafka > runner: flink > container: kubernetes > > We encounter an docker not found error when running: > python3 -m kafka_test --runner=FlinkRunner > --flink_master=flink-job-manager:8081 --flink_submit_uber_jar > --environment_type=EXTERNAL --environment_config=localhost:50000 > > We notice that in https://beam.apache.org/roadmap/portability/ it > mentioned the prerequisite also includes Docker. We wonder what is the > docker usage here? Is there any suggested way to build docker in > k8s container? (something maybe like sysbox for docker in docker?) > > Or maybe we should not use beam sdk+runner in k8s? > > Thanks, > Yilun >
