So sorry for subscribing errors on my side resulted in multiple duplicate email!
Thanks for reply and it does help! I am confused when submitting beam job with kafka connector to flink, I noticed that flink DAG diagram will included readFromKafka as part of flink workflow. while if we submit a pyflink job(connected with kafka) directly to flink, the flink workflow will exclude reading from kafka(which is the resource) but only has data processing parts. Is that how beam want flink to do? Thanks a lot and sincerely apologize again for silly duplicated emails! Yilun Sam Bourne <samb...@gmail.com>于2021年2月25日 周四上午11:58写道： > Hi Yilun! > > I made a quick proof of concept repo showcasing how to run a beam pipeline > in flink on k8s. It may be useful for you as reference. > > https://github.com/sambvfx/beam-flink-k8s > > > On Wed, Feb 24, 2021, 8:13 AM yilun zhang <ilyak1...@gmail.com> wrote: > >> Hey, >> >> Our team is trying to use beam with connector Kafka and runner flink to >> gather information and process data. We adopt python sdk and build in java >> 11 in python 3.7 sdk image as java runtime for kafka expansion service. >> so : >> image: beam python 3.7 docker image + build in java 11 >> connector: kafka >> runner: flink >> container: kubernetes >> >> We encounter an docker not found error when running: >> python3 -m kafka_test --runner=FlinkRunner >> --flink_master=flink-job-manager:8081 --flink_submit_uber_jar >> --environment_type=EXTERNAL --environment_config=localhost:50000 >> >> We notice that in https://beam.apache.org/roadmap/portability/ it >> mentioned the prerequisite also includes Docker. We wonder what is the >> docker usage here? Is there any suggested way to build docker in >> k8s container? (something maybe like sysbox for docker in docker?) >> >> Or maybe we should not use beam sdk+runner in k8s? >> >> Thanks, >> Yilun >> >
