Fwd: [YAML] New YAML examples with Kafka, Iceberg and ML use cases

+user -- Forwarded message - From: Chamikara Jayalath Date: Tue, Jul 15, 2025 at 3:09 PM Subject: Re: [YAML] New YAML examples with Kafka, Iceberg and ML use cases To: Thanks for adding these Charles. These should serve as production ready examples (tested on Direct runner and

Managed API for Beam sources and sinks

Hi all, Beam's Managed API for Java [1] and Python [2] offers a unified and simplified way to use existing Beam sources and sinks in a pipeline. Currently this API supports Beam I/O connectors Iceberg, Kafka, and BigQuery. There's also additional documentation and examples in the Dataflow Website

Re: Expansion service for SqlTranform fails with a local flink cluster using Python SDK

> When I check the expansion service docker container, normally it downloads a JAR file and starts SDK Fn Harness To clarify the terminology here, I think you meant the Java SDK harness container not the expansion service. Expansion service is only needed during job submission and your failure is

Re: [Question] ReadFromKafka can't get messages.

Which runner are you using ? There's a known issue with SDFs not triggering for portable runners: https://github.com/apache/beam/issues/20979 This should not occur for Dataflow. For Flink, you could use the option "--experiments=use_deprecated_read" to make it work. Thanks, Cham On Fri, Mar 8,

Re: Downloading and executing addition jar file when using Python API

The expansion service jar is needed since sql.py includes cross-language transforms that use the Java implementation behind the hood. Once downloaded, the jar is cached, and subsequent jobs should use the jar from that location. If you want to use a locally available jar, you can manually startup

Re: [Question] - How can i listen to multiple Pub/Sub input topics using SideInput?

rDo' is >> only option, but I would also suggest rethinking Your infrastructure setup. >> >> Best >> Wiśniowski Piotr >> >> śr., 1 lis 2023, 19:06 użytkownik Chamikara Jayalath via user < >> user@beam.apache.org> napisał: >> >>> Current

Re: [Question] - How can i listen to multiple Pub/Sub input topics using SideInput?

Currently only some Beam sources are able to consume a configuration (set of topics here) that is dynamically generated and I don't think PubSubIO is one of them. So probably you'll have to implement a custom DoFn that reads from Cloud Pub/Sub to support this. Also, probably you'll have to constrai

Re: [Request for Feedback] Swift SDK Prototype

On Wed, Sep 20, 2023 at 10:48 AM Danny McCormick wrote: > > I think the process should be similar to other code/design reviews for > large contributions. I don't think you need a PMC involvement here. > > I think it does require PMC involvement to create the actual repo once we > have public cons

Re: [Request for Feedback] Swift SDK Prototype

On Wed, Sep 20, 2023 at 9:54 AM Byron Ellis wrote: > Hi all, > > I've chatted with a couple of people offline about this and my impression > is that folks are generally amenable to a separate repo to match the target > community? I have no idea what the next steps would be though other than > gue

Re: [Question] Multiple images per pipeline

It's possible in theory but currently we don't have a good API for replacing the environment of a given transform when defining a pipeline. Environments are configured during transform expansion and if transforms use expansion services (with different dependencies) they will get unique environments

Re: [Request for Feedback] Swift SDK Prototype

On Thu, Aug 24, 2023 at 12:27 PM Robert Bradshaw wrote: > I would like to figure out a way to get the stream-y interface to work, as > I think it's more natural overall. > > One hypothesis is that if any elements are carried over loop iterations, > there will likely be some that are carried over

Re: [Request for Feedback] Swift SDK Prototype

Thanks Byron. This sounds great. I wonder if there is interest in Swift SDK from folks currently subscribed to the +user list. On Wed, Aug 16, 2023 at 6:53 PM Byron Ellis via dev wrote: > Hello everyone, > > A couple of months ago I decided that I wanted to really understand how > the Beam FnAp

[ANNOUNCE] Transform Service

Hi All, We recently added a Docker Compose based service named Transform Service to Beam. Transform service includes a number of transforms released with Beam and provides a single endpoint for accessing them via the Beam's multi-language pipelines framework. I've updated Beam Java/Python SDKs t

Re: Create IO connector for HTTP or ParDO

Connectors are written using ParDos. A connector (source) may use a source framework (Splittable DoFn is the recommended framework currently) or may be written using regular ParDos. The main advantages of a source framework are various features provided by such frameworks (progress reporting, dynam

Re: Getting Started With Implementing a Runner

Another advantage of a portable runner would be that it will be using well defined and backwards compatible Beam portable APIs to communicate with SDKs. I think this is specially important for runners that do not live in the Beam repo since otherwise future SDK releases could break your runner in s

Re: BatchUpdateException while trying to use WriteToJdbc

ail > when we try with a bigger source data payload. > Does it look like multiple threads trying to acquire a write lock to the > DB table(Oracle table)? > > *Thanks and Regards,* > > *Varun Rauthan* > > > > > On Wed, Feb 22, 2023 at 1:23 AM Chamikara Jay

Re: BatchUpdateException while trying to use WriteToJdbc

Which runner are you using ? Also, do you have the bottom of the StackTrace here ? It's possibly due to Docker containers running the Java SDK not having access to your database, but I'm not sure based on the information provided. Thanks, Cham On Tue, Feb 21, 2023 at 11:32 AM Somnath Chouwdhury

Re: ElasticsearchIO write to trigger a percolator

On Thu, Feb 2, 2023 at 1:56 PM Kaggal, Vinod C. (Vinod C.), M.S. via user < user@beam.apache.org> wrote: > Hello! Thank you for all the hard work on implementing these useful > libraries. > > > > *Background:* We have been using Apache Storm in production for some time > (over 8 years) and have re

[ANNOUNCE] Apache Beam 2.43.0 Released

The Apache Beam team is pleased to announce the release of version 2.43.0. Apache Beam is an open source unified programming model to define and execute data processing pipelines, including ETL, batch and stream (continuous) processing. See https://beam.apache.org You can download the release her

Fwd: Cross Language

entation/sdks/java-multi-language-pipelines/ Thank, Cham > Will the above scenario work if it does how can implement python transform > in Java. > > Regards, > Phani Geeth > > On Tue, 11 Oct, 2022, 7:19 pm Chamikara Jayalath via user, < > user@beam.apache.org> wrote: > &

Re: Cross Language

Is there a specific I/O connector you are hoping to use ? Thanks, Cham On Tue, Oct 11, 2022 at 4:31 AM Alexey Romanenko wrote: > Yes, it’s possible though Java IO connector should support being used via > X-language. > > For more details regarding which connector supports this, you may want to

Re: [question] Good Course to learn beam

Not necessarily a course but I would also highly recommend reading the Beam Programming Guide to learn about Beam. https://beam.apache.org/documentation/programming-guide/ I think it's well written and it comes with example code for all supported SDKs. Thanks, Cham On Wed, Aug 31, 2022 at 12:31

Re: PubSub Lite IO & Python?

ns) as p: >>>>>> >>>>>> p | 'Read From PubSubLite' >> ReadFromPubSubLite( >>>>>> >>>>>> subscription_path='projects/starwatch/locations/us-west1-a/subscriptions/sw-test-sky-events-ti

Re: PubSub Lite IO & Python?

ring.Element() | beam.Map(print >>>> ) | 'Writing to DB' >> relational_db.Write( >>>> source_config=source_config, >>>> table_config=table_config >>>> ) >>>> >>> >>> On Fri

Re: How to register as external cross language transform ?

On Wed, Aug 17, 2022 at 3:05 AM Yu Watanabe wrote: > Hello. > > I am trying to write code for cross language transform for > ElasticsearchIO but having trouble with it. > I would appreciate it if I could get help. > > As describe in doc and also referencing KafkaIO , > > > https://beam.apache.org

Re: PubSub Lite IO & Python?

On Thu, Aug 4, 2022 at 5:29 PM Daniel Collins wrote: > Hello Drew, > > > I upgraded to apache-beam 2.40.0 and tried to access > apache_beam.io.gcp.pubsublite.ReadFromPubSubLite > > You should ensure to import `apache_beam.io.gcp.pubsublite.*`. I have no > idea why the specific import isn't workin

Re: PubSub Lite IO & Python?

I believe this should be fully working. I'm not familiar with PyBeam though. Is the execution mechanism the same as running a regular Beam pipeline ? Also, note that for multi-language, you need to use a portable Beam runner. +Daniel Collins who implemented this. Thanks, Cham On Thu, Aug 4, 202

Re: How to configure external service for Kafka IO to run the flink job in k8s

Command for starting a customer expansion service is available here: https://beam.apache.org/documentation/sdks/python-multi-language-pipelines/#choose-an-expansion-service Also you can customize the environment set by this expansion service in the expanded transforms using the environementType an

Re: [Dataflow][Python] Guidance on HTTP ingestion on Dataflow

On Wed, Jul 20, 2022 at 12:57 PM Chamikara Jayalath wrote: > I don't think it's an antipattern per se. You can implement arbitrary > operations in a DoFn or an SDF to read data. > > But if a single resource ID maps to a large amount of data, Beam runners > (including Dataflow) will be able to par

Re: [Dataflow][Python] Guidance on HTTP ingestion on Dataflow

I don't think it's an antipattern per se. You can implement arbitrary operations in a DoFn or an SDF to read data. But if a single resource ID maps to a large amount of data, Beam runners (including Dataflow) will be able to parallelize reading, hence your solution may have suboptimal performance

Re: Implementing a custom I/O Connector

n' object has no attribute 'windowing' > > > NB > - I did some editing on the notebook so the original revision is here > <https://colab.research.google.com/drive/1ljtoEtyG0gwbq6SPTY1EHHmpdh6EHuxu#scrollTo=vaXnHuVOtEdG> > > On Thu, Jul 14, 2022 at 10:15 P

Re: Implementing a custom I/O Connector

Do you have the full stacktrace ? Also, what does the Read() transform in the example entail ? Thanks, Cham On Thu, Jul 14, 2022 at 7:39 AM Damian Akpan wrote: > Hi Everyone, > > I've been working on implementing a Google Sheets IO source for my > pipeline. I've tried this example >

Re: Any guideline for building golang connector ?

Strong +1 for using x-lang instead if re-implementing the ElasticSearch connector in Go. Thanks, Cham On Fri, Jul 8, 2022 at 5:16 AM Yu Watanabe wrote: > Hello Danny. > > Thank you for the details. I appreciate your message. > > I am a newbie around building io . So I will look into the links a