RE: Transform Pattern Question

I would like to see an example. From: Joey Tran Sent: Tuesday, October 15, 2024 11:09 AM To: user@beam.apache.org Subject: Re: Transform Pattern Question You don't often get email from joey.t...@schrodinger.com. Learn why this is important

Where to import external dependencies when using Flex Template

Not exactly Apache Beam, but I notice if I run Apache Beam on Dataflow, using a flex template, I have import problems: For example, the following code will fail because it can't findpsycopg2 1 import psycopg2 class ReadDb(beam.DoFn): 50 51 def __init__(self, user, password, host): 52

How to write to a Kafka topic?

I am creating a simple pipeline to read and then write to Kafka. Here is my code: 156 with Pipeline(options=pipeline_options) as pipeline: 157 main = ( 158 pipeline 159 | ReadFromKafka( 160 consumer_config={'bootstrap.servers': configs['bootstra

RE: How to write to a Kafka topic?

elps - https://jaehyeon.me/blog/2024-04-18-beam-local-dev-3/ Cheers, Jaehyeon On Fri, 4 Oct 2024 at 08:21, Henry Tremblay via user mailto:user@beam.apache.org>> wrote: I am creating a simple pipeline to read and then write to Kafka. Here is my code: 156 with Pipeline(opti

RE: How to write to a Kafka topic?

https://aka.ms/LearnAboutSenderIdentification> Nice! Seems to be working. I will post a HowTo fo the next Python developer. | "temp convert" >> beam.Map(temp_convert).with_output_types(typing.Tuple[bytes, bytes]) On Fri, 4 Oct 2024 at 09:02, Henry Tremblay via user mailto:user@beam.apa

RE: Solution to import problem

You shouldn’t have to use an sdk_container_image. It is not in the docs, and I talked to Google, and they said a container image is not needed. Also, why does the requests library work, and the secret manager does not? From: XQ Hu via user Sent: Sunday, November 3, 2024 1:53 PM To: user@beam.ap

RE: Solution to import problem

My launcher image has: ENTRYPOINT ["/opt/google/dataflow/python_template_launcher"] If I use this as my sdk_harness_image, as I did before, the job will hang, probably because the worker is trying to launch a job. So I need a separate image with: ENTRYPOINT ["/opt/apache/beam/boot"] I think G

RE: Problem running pipeline in unit tests

You can’t have the same argument in the parse_know_args and XyzOptions (assuming you get these args in the same way). From: Sofia’s World Sent: Tuesday, December 3, 2024 11:32 PM To: user@beam.apache.org Subject: Re: Problem running pipeline in unit tests Hello thanks for getting back.I track