Re: Any guideline for building golang connector ?

Hey Yu, The guidance on that page should generally apply for Go as well, though we are missing an example transform; I filed https://github.com/apache/beam/issues/22194 to fix this, but a couple examples are our textio implementation

Re: Benchmark tests for the Beam RunInference API

I left a few comments, but overall this sounds like a good plan to me - thanks for the writeup! On Tue, Aug 16, 2022 at 9:36 AM Anand Inguva via dev wrote: > Hi, > > I created a doc > [1] > which outlines the

Re: How to run expansion service using go sdk in local development environment ?

Hey Yu, as the error you posted suggests, the Go direct runner which you're using in your local development environment doesn't support external transforms using an expansion service. If you're going to do a x-lang transform using an expansion service you should use a different runner like Dataflow

Re: [Question] [Go SDK] Generic Register DoFn having iter(*interface{})

Hey Rener, you should be able to register that function with `register.DoFn3x1[context.Context, int, func(*interface{}) bool, error]`. You would use DoFn3x1 because you have 3 inputs and 1 output in your ProcessElement, and then the constraints are just the input types to your ProcessElement functi

Re: [Question] [Go SDK] Generic Register DoFn having iter(*interface{})

Ah, I see - I didn't realize the problem was happening post-registration. This error originates from https://github.com/apache/beam/blob/8b2676782a62f8bdf912395267056c9f37251338/sdks/go/pkg/beam/core/runtime/graphx/serialize.go#L502 and basically means that we are not able to infer a default coder

Re: An AI/ML landing page for Beam is available now

Thank you to everyone who worked on this and especially to Aizhamal for coordinating the work! These resources make it much easier to build out your Beam ML workloads. Thanks, Danny On Thu, Nov 10, 2022 at 12:53 PM Aizhamal Nurmamat kyzy wrote: > Hi Beam community! > > We are happy to announce

Re: [Question] 2.47.0 Release

Hey Ayoyinka, That PR with protobuf>=4.21.1,<4.23.0 support should be included in the 2.47 release, that is correct. The release schedule is tracked here - https://calendar.google.com/calendar/u/0/embed?src=0p73sl034k80oob7seouani...@group.calendar.google.com&ctz=America/Los_Angeles. The next rel

Re: How to handle errors in GO SDK in a custom PTransform

The Go SDK doesn't use tagged outputs, instead it uses positional ordering for emitting multiple outputs. So you can do something like: func processElements(element string, goodEmit, errEmit func(string)) { if element.isGood { goodEmit(element) } else { errEmit(element) } } w

Re: Options for visualizing the pipeline DAG

Hey Joey, Dataflow and Beam playground are 2 options as you mentioned, locally many SDKs have local runner options with a visual component. For example, in Python you can use the interactive runner with the apache-beam-jupyterlab-sidepanel extension

Re: [Request for Feedback] Swift SDK Prototype

> I think the process should be similar to other code/design reviews for large contributions. I don't think you need a PMC involvement here. I think it does require PMC involvement to create the actual repo once we have public consensus. I tried the flow at https://infra.apache.org/version-control

Re: [DISCUSSION] Apache Diversity & Apache Beam Good First Issues

Hey Gláucia, we do use the "good first issue" label. In my experience I've seen mixed results with it - sometimes it is a good starting point, other times folks self-assign the issues and then never make progress on them. It is also challenging to enforce a consistent quality of "good first issue"

Beam 2.52.0 Release

I am happy to announce that the 2.52.0 release of Beam has been finalized. This release includes both improvements and new functionality. For more information on changes in 2.52.0, check out the detailed release notes - https://github.com/apache/beam/milestone/16. Here is an overview of the change

Re: Does withkeys transform enforce a reshuffle?

For runners that support Reshuffle, it should be safe to use. Its been "deprecated" for 7 years, but is still heavily used/often the recommended way to do things like this. I actually just added a PR to undeprecate it earlier today. Looks like you're usin

Re: [Dataflow][Java][2.52.0] Upgrading to 2.52.0 Surfaces Pubsub Coder Error

I imagine this is no longer helpful to you, Evan, but I ran into this issue this week and tracked down the underlying problem. Basically, a snappy-java version upgrade [1] seems to have changed how the SnappyCoder [2] is serialized. Since this is used by PubSub read, it caused upgrades to start fai

Beam 2.61.0 Release

Hi, I am happy to announce that Beam 2.61.0 has been fully released. For more information about the release, check out the release notes - https://github.com/apache/beam/releases/tag/v2.61.0 Thanks, Danny

Re: Inquiry about Beam Summit 2025

Hey Marcin, We don't have firm information yet, but we should have an update before the end of this year. At this point, I can definitely say that the Summit will not happen early in 2025, it will happen sometime in the summer or later. Thanks, Danny On Wed, Nov 20, 2024 at 7:36 AM Marcin Stańcz

Re: Regarding Updates, Slack and Contribution

Hey, welcome to the Beam community! > Can anyone please tell me how I can join slack channel of ASF (Apache Software Foundation) as I don’t have apache.org email address. Also, it would help me to know the community as well as know about the current workings on the project. I just sent you an inv

Removing Deprecated v1 AWS IOs

Hey everyone, this is a heads up that during the 2.63.0 Beam release (planned to be cut Feb 5, 2025), we will be removing the AWS v1 IOs [1]. These IOs have been deprecated since Beam 2.41.0 (July 2022), with the message "You are using a deprecated IO for DynamoDB. Please migrate to module 'org.apa

Re: Regarding Updates, Slack and Contribution

as reading in the Apache Beam >website it said that we need to fill up the agreement before contributing. >2. And wanted to ask any plans for GSOC 2025? > > > > Regards, > > Thanking You > > Siddharth Salian > > > > > > *From: *Danny McCormick

Re: Regarding the GSOC 2025 Project

overall structure of >proposal, contents to be added and overview of how the proposal should be >like, which would provide better clarity sir. > > > > Best Regards, > > Thanking you > > Siddharth Salian > > > > *From: *Danny McCormick via user > *Da

Re: Regarding the GSOC 2025 Project

Hey Siddharth, thanks for reaching out. I'm glad you're interested in the project. In general, I would expect there to be more details about projects once we know which ones have been accepted. > Sir, if you could tell me the pre-required knowledge (such as major programming languages used, etc.,

Re: Regarding the GSOC 2025 Project

t; *To: *user@beam.apache.org > *Subject: *Re: Regarding the GSOC 2025 Project > > Hello Sir, > > Thank you for the email. I have understood. > > > > Thanks, > > Siddharth Salian > > > > *From: *Danny McCormick via user > *Date: *Thursday, 20 February 20

Re: Regarding the GSOC 2025 Project

e I/O page >in the documentation page as well as vector DB’s, features. > > > >2. Sir, as you have mentioned in the mail, Python is must for this >project, I just wanted to ask, what about Java and Golang SDK applications, >I mean I know it’s an AI/ML pipeline bas

Re: Regarding the GSOC 2025 Project

unication >through mailing lists. > > > >3. Sir, with reference to the point about python, I meant to ask that > sir, like apart from learning the main coding language of python, anything >more important topic has to be learnt (such as python with ML pipelines,

Re: Regarding the GSOC 2025 Project

n coding language of python, anything > more important topic has to be learnt (such as python with ML pipelines, >etc.) as a part of project prerequisites? > > > > Best Regards, > > Thanking you, > > Siddharth Salian > > > > *From: *Danny McCormick via

Re: Regarding the GSOC 2025 Project

gt; > > *From: *SIDDHARTH SALIAN > *Date: *Tuesday, 4 March 2025 at 1:53 AM > *To: *Danny McCormick , Danny McCormick via > user > *Subject: *Re: Regarding the GSOC 2025 Project > > Respected Sir, > > Thank you for the email. I have understood. I’ll continue the conver

Re: Regarding the GSOC 2025 Project

roject, don’t you think RAG is still >limited to capturing historical data, or it has capability of capturing >latest/modern data’s too? > > > > Best regards, > > Thanking you, > > Siddharth Salian > > > > *From: *Danny McCormick via user > *

Re: Langchain/Agents on Beam

Yeah, this shouldn't really be particularly different than any other sort of inference. I'm curious about your use case - would you be willing to share more about what you are using this for? For context, I'm interested in seeing if we can add broader agentic support to Beam, though I'm having a ha

Breaking change to disable argument abbreviation in Beam Python

Today, you can abbreviate arguments in Beam Python. This is generally convenient since you can do things like specify `--r` instead of `--runner`, and Beam will infer your intent. Unfortunately, it also has unintended side effects. For example, specifying `--u` will impact not just `--update`, but