Not exactly Apache Beam, but I notice if I run Apache Beam on Dataflow, using a flex template, I have import problems:
For example, the following code will fail because it can't find psycopg2 1 import psycopg2 class ReadDb(beam.DoFn): 50 51 def __init__(self, user, password, host): 52 self.user = user 53 self.password = password 54 self.host = host 55 56 def process(self, element): 58 conn = psycopg2.connect ( 59 host = self.host, 60 user = self.user, 61 password = self.password, 62 database = 'chassis_trusted_data', 63 port = 5432) 64 65 yield 'a' I actually need to import pyscopg2 in the process method (line 57) I know I can use pipeline_options.view_as(SetupOptions).save_main_session = save_main_session but this causes pickling problems, and defeats the purpose of building a Docker image
