Re: [RESULT] [vendor-calcite] Vendored Dependencies Release

Updated (after chatting offline about correct format). Thanks! https://dist.apache.org/repos/dist/release/beam/vendor/beam-vendor-calcite-1_40_0/0.1/ On Mon, Jul 28, 2025 at 4:17 PM Yi Hu wrote: > Sorry for not being clear, one needs to create a nested folder > calcite-1_40_0/0.1/ under vendor/

Re: [RESULT] [vendor-calcite] Vendored Dependencies Release

I can help! I noticed that https://dist.apache.org/repos/dist/release/beam/vendor/ has artifacts formatted as beam-vendor-calcite-1_26_0/0.1 though instead of calcite-1_40_0/0.1/ like you describe: is that change intentional? Thanks, Danny On Mon, Jul 28, 2025 at 4:03 PM Yi Hu via dev wrote: >

Re: [VOTE][vendor-calcite] Vendored Dependencies Release

+1 (binding) Thanks for driving this! Danny On Thu, Jul 24, 2025 at 9:55 PM Chamikara Jayalath via dev < dev@beam.apache.org> wrote: > +1 > > Thanks, > Cham > > On Wed, Jul 23, 2025 at 4:05 PM Yi Hu via dev wrote: > >> Hi everyone, >> >> Please review and vote on the release candidate #1 for >>

Re: [doc] Evaluating Third-Party Runtime Type Checking Libraries for Beam Python

Thanks! This is helpful and I like the proposed direction moving towards BearType. On Wed, Jul 23, 2025 at 10:03 AM Jack McCluskey via dev wrote: > Hey everyone, > > Hot on the heels of the type hinting overview doc and my talk on the same > subject at Beam Summit, I've done a little bit of work

Re: [Proposal][GSoC 2025] Milvus Vector Sink I/O Connector for Beam

Thanks for putting this together! I left a suggestion, but overall it looks like a great proposal! On Wed, Jul 16, 2025 at 4:27 PM Mohamed Awnallah wrote: > It looks like the intro stripped somehow. Here is the full email :) > > [Proposal][GSoC 2025] Milvus Vector Sink I/O Connector for Beam > >

Re: [Proposal] Beam ML containers

ry docker images.) (One could > possibly get away with the "AnyOf" environment as the base environment > as well, if we define (and enforce) a preference order.) > > This being the messy world of ML, would these images be > mahine/accelerator agnostic? > > > K

[Proposal] Beam ML containers

Hey everyone, I'd like to propose publishing some ML-specific Beam containers alongside our normal base containers. The end result would be allowing users to specify `--sdk_container_image=ml` or `--sdk_container_image=gpu` so that their jobs run in containers which work well with ML/GPU jobs. I p

Re: [VOTE] Release 2.66.0, release candidate #2

+1 (binding) Ran through some ML examples locally and on Dataflow. Thanks, Danny On Thu, Jun 26, 2025 at 2:13 AM Chamikara Jayalath via dev < dev@beam.apache.org> wrote: > +1 > > Thanks, > Cham > > On Tue, Jun 24, 2025 at 8:24 AM Vitalii Terentev > wrote: > >> +1 (non-binding) >> >> Tested Pyt

Re: [GSoC 2025] Git based Privilege Management System

Hey Enrique, thanks for looking into this - I left some feedback in the PR. Thanks, Danny On Wed, Jun 18, 2025 at 3:06 PM Enrique Calderon wrote: > Hi Beam community! > > Once again talking about this migration. Some problems related to some > permissions where fixed, now I am asking for your f

Re: Introducing Catalogs to Beam SQL

Thanks, the doc generally looks good to me. My suggestion here would be: 1) Move forward with the PR mostly as is (it seems like it has gotten meaningful feedback and has been merged, so I guess this is done). 2) Cut the release branch so we can start to make progress on releasing 3) In parallel m

Re: [ANNOUNCE] New Committer: Shunping Huang

Congratulations Shunping! This is well deserved! On Sat, Jun 7, 2025 at 12:42 PM Robert Burke wrote: > Congratulations Shunping! > > On Sat, Jun 7, 2025, 7:02 AM XQ Hu via dev wrote: > >> Congratulations Shunping! Thanks a lot for your contributions! >> >> On Sat, Jun 7, 2025 at 9:29 AM LDesire

Re: [python] subprocess call of "pip freeze" per pipeline

don't feel strongly either way FWIW) > > [1] > https://github.com/apache/beam/blob/b7f2e1611556cf2dab7e9a901d3477023cd71294/sdks/python/apache_beam/runners/trivial_runner.py#L47 > > On Thu, Jun 5, 2025 at 9:59 AM Danny McCormick via dev < > dev@beam.apache.org> wrote: > &g

Re: [python] subprocess call of "pip freeze" per pipeline

Thanks for calling this out. I generally agree with you. I've found this feature to be generally quite useful for production jobs running in distributed environments. I have seen several issues which have been solved because of it (and similarly I have seen issues which would have benefited from it

Re: [Proposal][GSoC 2025] Milvus Vector Enrichment Handler for Beam

Thanks! I left a few comments, but I like the idea/approach! On Thu, May 29, 2025 at 1:55 PM Mohamed Awnallah wrote: > Hello Beam Dev Community, > > I'm excited to share the design document for the Milvus Vector Enrichment > Handler for Apache Beam as part of my GSoC 2025 project. > > This enric

Re: Proposal: Implementing automated stale issue management (173 days inactivity + 7 days warning)

+1 to the proposal. > +1 generally, this seems to be the approach many other projects follow, so it seems reasonable. One note - the 7 day deadline feels a little too strict. I'd propose to change this to 150 days + 30 days, the total would be the same, but people can have more time to react. Thi

Re: [Beam ML] GSoC 2025 Acceptance - Vector DB/Feature Store Integrations Project

Woohoo, congratulations! Looking forward to working with you this summer! On Mon, May 19, 2025 at 2:50 PM XQ Hu via dev wrote: > Welcome to the community! and congratulations! > > On Mon, May 19, 2025 at 2:39 PM Mohamed Awnallah < > mohamedmohey2...@gmail.com> wrote: > >> Hi Beam Devs, >> >> I'm

Re: [Discuss] Breaking change to disable argument abbreviation in Beam Python

eviated parameter >>> names are for saving typing during interactive usage. >>> >>> Let's not innovate in the command line interface arena, and stick to >>> data processing :-) >>> >>> Kenn >>> >>> On Tue, May 13, 20

[Discuss] Breaking change to disable argument abbreviation in Beam Python

Today, you can abbreviate arguments in Beam Python. This is generally convenient since you can do things like specify `--r` instead of `--runner`, and Beam will infer your intent. Unfortunately, it also has unintended side effects. For example, specifying `--u` will impact not just `--update`, but

Re: [RESULT] [VOTE] Release 2.65.0, release candidate #2

Done - thanks! I'll wait to do the post on linkedin step until the website is published with the overview of the release, but it is drafted. Thanks, Danny On Mon, May 12, 2025 at 2:10 PM Yi Hu via dev wrote: > Hi, > > Could a PMC member help finalize the release? That is follow > https://github

Re: [VOTE] Release 2.65.0, release candidate #2

+1 (binding) Tested with a few ML pipelines on local and Dataflow runners Thanks, Danny On Thu, May 8, 2025 at 9:45 AM Yi Hu via dev wrote: > +1 (non-binding) > > Tested Dataflow Templates integration test [1], also tested YAML > validation [2] and Go Wordcount [3] > > [1] https://github.com/G

Secret Managers in Core Beam + Beam Yaml

Hey everyone, in the context of Beam Yaml I've noticed a greater need for supporting secret managers in more of a first class way to avoid plaintext secrets as part of the yaml. To handle this, I put together a mini doc on how we could support secret managers natively. Please take a look - https://

Re: Inquiry About GSoC Project - Beam ML Vector DB/Feature Store Integrations

Yeah, that would be best - thanks! On Sun, Apr 6, 2025 at 6:34 AM Aditya wrote: > *Subject:* Question Regarding OpenAI Embedding Implementation > > Hi Danny, > > I have a quick question: > > Should the OpenAI embedding implementation handle Chunk objects from the > RAG framework similarly to how

Re: GSoC Proposal - Infrastructure Automation for Apache Beam's Test Environment

Noting that the submission deadline is today in 4.5 hours - https://developers.google.com/open-source/gsoc/timeline - given that, I would recommend submitting the proposal as is. You can still follow up with a Google doc to get more feedback and resubmit if time permits. Thanks, Danny On Tue, Apr

Re: GSoC 2025 Proposal - Pinecone & Tecton Connectors (Wesam Abed)

Thanks for the proposal! I took a look and left a few comments; overall the content looks good and comprehensive to me though. Thanks, Danny On Thu, Apr 3, 2025 at 4:43 PM Wesam Abed wrote: > Hi all, > > I’m Wesam Abed, and I’ve submitted a GSoC 2025 proposal focused on > building I/O connector

Re: GSOC-278 : Beam YAML ML, Iceberg, and Kafka User Accessibility - Interest

> On Mon, Mar 10, 2025 at 9:56 AM Danny McCormick via dev < > dev@beam.apache.org> wrote: > >> Hi Jose, >> >> I hope you're doing well. I'd recommend that you start by familiarizing >> yourself with the following links: >> >> - Beam contri

Re: [VOTE] Release 2.64.0 release candidate #2

n-binding) > >> > >> Validated with GCP-IO load tests ( > https://github.com/apache/beam/tree/master/it/google-cloud-platform) on > Dataflow runner (legacy, v2) > >> > >> Thanks, > >> Yi > >> > >> > >> On Fri, Mar

Re: [VOTE] Release 2.64.0 release candidate #2

+1 (binding) Validated with some ML examples on the local runner and on Dataflow. Thanks, Danny On Thu, Mar 27, 2025 at 1:47 AM Rohit Sinha via dev wrote: > Updated vote for RC2: +1 (non-binding) > > All tests are passing now. > > > > > > On Wed, Mar 26, 2025 at 5:08 PM Chamikara Jayalath > w

Re: Inquiry About GSoC Project - Beam ML Vector DB/Feature Store Integrations

Thanks - I left a few comments, overall it seems like a good proposal though. Thanks, Danny On Mon, Mar 24, 2025 at 3:20 PM Aditya wrote: > *Subject:* Additional Submission for GSoC > > Hi Danny, > > Please take a look at this as well. > > I will be submitting this to GSoC as they require a PDF

Re: [Design Doc] Generic Remote Model Handlers for RunInference

Thanks - left a UX suggestion, but overall I think this looks good and will be quite useful. On Mon, Mar 24, 2025 at 10:12 AM Jack McCluskey via dev wrote: > Hey everyone, > > I've put together a design doc for a generic remote model handler base > class >

Re: [ANNOUNCE] New Committer: Vitaly Terentev

Congratulations Vitaly! Thanks for all the work you've done on Beam infrastructure in particular! On Mon, Mar 24, 2025 at 12:10 PM Ahmet Altay via dev wrote: > Congratulations Vitaly! > > On Mon, Mar 24, 2025 at 8:36 AM Kenneth Knowles wrote: > >> Hi all, >> >> Please join me and the rest of th

Re: Inquiry About GSoC Project - Beam ML Vector DB/Feature Store Integrations

> Should I submit the proposal as a GitHub link or a PDF? Either is fine for submitting the proposal. > I’d also love any tips you have on improving my proposal. Have you shared it as a google doc like mentioned earlier? Thanks, Danny On Sun, Mar 23, 2025 at 8:37 AM Aditya wrote: > Subject:

Re: Inquiry About GSoC Project - Beam ML Vector DB/Feature Store Integrations

Hey Aditya, I would recommend sharing this as a google doc or something which allows people to leave comments on it if you'd like feedback. At a high level, the proposal generally looks reasonable and well written to me, though! Thanks, Danny On Wed, Mar 19, 2025 at 5:10 PM Aditya wrote: > *Sub

Re: Contributing to Beam

Hey Suvrat, welcome to Beam! I just sent you an invitation to join the slack community. On Thu, Mar 20, 2025 at 7:56 AM Suvrat Acharya wrote: > Hello Beam Community. > > Hope you all are doing well. > I am Suvrat, an engineering student, I have been a regular contributor to > various apache repo

Re: Inquiry About GSoC Project - Beam ML Vector DB/Feature Store Integrations

this? > > Alex > > On Mon, 17 Mar 2025 at 13:18, Danny McCormick via dev > wrote: > >> Hey Aditya, there is not necessarily a single set of benchmarks which we >> can use to evaluate an IO, and defining exactly what/how we should be >> measuring completeness a

Re: Inquiry About GSoC Project - Beam ML Vector DB/Feature Store Integrations

Hey Aditya, there is not necessarily a single set of benchmarks which we can use to evaluate an IO, and defining exactly what/how we should be measuring completeness and performance is part of the work to be done here. I think this is a good thing for you to try to initially define in your project

Re: Joining Apache Beam Slack

Sure, I added you On Thu, Mar 13, 2025 at 12:08 AM José Ortiz wrote: > Hi all, > > My name is Jose, and I am an applicant for the GSoC program. I'm keen to > stay updated on the *latest* Beam developments. > > I would like to join the ASF Slack #beam channel. Could you please add me? > My email

Re: Joining Apache Beam Slack

Yes, it would probably make sense to just direct people to email the dev list to be added. Feel free to add a PR, I'm happy to review/merge. On Wed, Mar 12, 2025 at 7:26 PM Rakesh Kumar wrote: > We should update this page (https://beam.apache.org/community/join-beam/) > with the correct steps fo

Re: Inquiry About GSoC Project - Beam ML Vector DB/Feature Store Integrations

Yeah, that probably makes the most sense for most vector DBs and feature stores. Thanks, Danny On Wed, Mar 12, 2025 at 4:25 PM Aditya wrote: > *Subject:* Clarification on Sink and Source Handler Implementation > > Hi Danny, > > I need one more clarification—am I required to implement the sink a

Re: Joining Apache Beam Slack

Hey Rakesh, I just sent you an invite. Thanks, Danny On Wed, Mar 12, 2025 at 2:32 PM Rakesh Kumar via dev wrote: > Hi Ahmet, > > Can please you also include my email address in slack: > rakeshcu...@gmail.com > > Thank you, > Rakesh > > On Mon, Nov 11, 2024 at 4:11 PM Ahmet Altay via dev > wrot

Re: GSOC-278 : Beam YAML ML, Iceberg, and Kafka User Accessibility - Interest

Hi Jose, I hope you're doing well. I'd recommend that you start by familiarizing yourself with the following links: - Beam contribution guide - https://github.com/apache/beam/blob/master/CONTRIBUTING.md - Beam yaml overview - https://beam.apache.org/documentation/sdks/yaml/ - Beam yaml docs - htt

Re: Aspiring Gsoc ‘25 Contributor

Hi Brijesh, I hope you're doing well. I'd recommend that you start by familiarizing yourself with the following links: - Beam contribution guide - https://github.com/apache/beam/blob/master/CONTRIBUTING.md - Beam yaml overview - https://beam.apache.org/documentation/sdks/yaml/ - Beam yaml docs -

Re: Inquiry About GSoC Project - Beam ML Vector DB/Feature Store Integrations

Sure, you're welcome to try working on it, it would just be outside of the scope of GSOC. Regardless, you are always welcome to make contributions to Beam :) Thanks, Danny On Wed, Mar 5, 2025 at 2:44 PM Aditya wrote: > *Subject:* Implementation of OpenAI Embeddings Before GSoC > > Dear Sir, > >

Re: Inquiry About GSoC Project - Beam ML Vector DB/Feature Store Integrations

I generally agree that this would be good to add (along with something for Anthropic and maybe others). I think it is not necessarily within the scope of this project, though, so I would not recommend including it as an early item in a project proposal (it could be a nice to have if there's time at

Re: Beam Infrastructure: Health Status Report for Feb 2025

Thanks Vitaly and team! Its great to see the progress; anecdotally, it has also been nice to not have a bunch of flakes on most PRs :) On Tue, Mar 4, 2025 at 11:03 AM Vitaly Terentyev via dev < dev@beam.apache.org> wrote: > Dear Community, > > Our team has been actively monitoring and improving B

Re: Inquiry About GSoC Project - Beam ML Vector DB/Feature Store Integrations

Hey Aditya, I don't think there is a very well defined priority order. I;ll note that we already have enrichment handlers for Feast/Vertex AI for reading/enriching data with lookups to those systems, so I'd probably say the following prioritization makes sense: - Sink for Vertex/Feast (finish wha

Re: Innaproppriate cache usage on PR validation suites?

There are a few options for turning off the cache. 1. The remote cache is dependent on having a cache username defined [1]. So you could just remove that from the workflow [2]. 2. You can explicitly disable all caching with --no-build-cache [3]. 3. You can allow a specific task to skip the cache c

Re: [Feature Proposal] Expose Kafka Client Metrics in Beam

Thanks for putting this together! I am generally in favor of a DIY PTransform option for now to get the value of these metrics in the shorter term. I am interested in the alternative of restructuring metrics, but I agree that we shouldn't block all progress on this. Thanks, Danny On Wed, Feb 26,

Re: Inquiry About GSoC Project - Beam ML Vector DB/Feature Store Integrations

Where available it is usually simpler to use the client libraries. Thanks, Danny On Wed, Feb 26, 2025 at 6:06 AM Aditya wrote: > Sir one more thing > >> Should we use only client library or api or both >

Re: Inquiry About GSoC Project - Beam ML Vector DB/Feature Store Integrations

Sure, I have added you. Thanks, Danny On Mon, Feb 24, 2025 at 12:21 PM Aditya wrote: > Thanks for the reply. > > Can i ask something > Can I join slack communication channel of beam > > On Mon, 24 Feb, 2025, 22:44 Danny McCormick, > wrote: > >> Hey Aditya, glad to hear that you are interested

Re: Best way to normalize TFRecordIO

Thanks for looking into this! I think I like option (2) for the base transform since it allows us to normalize across languages and get this added with the lowest amount of effort, plus it doesn't stop us from adding (1), or (3) in the future (though this may eventually require some more complex fo

Re: Inquiry About GSoC Project - Beam ML Vector DB/Feature Store Integrations

Hey Aditya, glad to hear that you are interested in this project. I've tried to answer your questions below: > What are the key technical challenges in integrating Beam with Pinecone and Tecton? The main challenges will be around understanding how those systems (and other similar systems) work, h

Re: Best way to expose windowing information in Beam YAML

+1 to `ReifyWindowingInfo` (or maybe `ExtractWindowingInfo` or `GetWindowing` is a little more understandable to the average user). I definitely prefer something which doesn't require extending the set of concepts/advanced usages we're exposing through Yaml, especially for a feature that I think wi

Re: [VOTE] Release 2.63.0, release candidate #2

+1 (binding). Ran some ML pipelines on the local and Dataflow runners. Thanks, Danny On Wed, Feb 12, 2025 at 1:47 PM XQ Hu via dev wrote: > +1 (non-binding). Tested the Python SDK with a simple Dataflow ML > pipeline: > https://github.com/google/dataflow-ml-starter/actions/runs/13291770412/job/

Re: Regarding Slack, Contributions

Hey Siddharth, I think I answered your questions on the user list - https://lists.apache.org/thread/bbfvd6h6lmw8q2od6tdnotf634okhqbg - if you have any more questions, let me know in that thread! Thanks, Danny On Tue, Feb 11, 2025 at 2:50 PM SIDDHARTH SALIAN < siddharthsalia...@gmail.com> wrote:

Re: [VOTE] Release 2.63.0, release candidate #1

+1 (binding) - I validated this with a few ML pipelines on the interactive and Dataflow runners. Thanks, Danny On Fri, Feb 7, 2025 at 4:00 PM XQ Hu via dev wrote: > +1 (non-binding). Tested it with a simple Dataflow ML pipeline: > https://github.com/google/dataflow-ml-starter/actions/runs/13205

Re: Beam High Priority Issue Report (31)

PM Ahmet Altay wrote: > > > On Thu, Feb 6, 2025 at 10:00 AM Danny McCormick via dev < > dev@beam.apache.org> wrote: > >> I do look at this most of the time and sometimes take action on it (maybe >> once every week or 2). I agree that I mostly care about (a) the ne

Re: Beam High Priority Issue Report (31)

I do look at this most of the time and sometimes take action on it (maybe once every week or 2). I agree that I mostly care about (a) the new issues and (b) the ones which aren't just flaky tests. I'd probably vote we keep it, but reduce the frequency from daily to weekly (or even monthly). I put

How vLLM Model Handler Works (Plus a Summary of Model Memory Management in Beam ML)

Late last year, I added support for vLLM in RunInference. I ended up being able to go from prototyping to checked in code quickly enough that I didn't put together/share a full design, but in retrospect I thought it might be helpful to have a record of what I did since others might want to do simil

[Design] Beam Python Dependency Extras

Hey everyone, I put together a mini-doc on bundling some more Beam Python/ML extras so that we have a better strategy for making sure that users can use ML (or other Python) dependencies which are well tested with their Beam version. It is mostly in line with how we handle our other dependencies an

Re: [VOTE] Vendored Grpc Dependency Release

, 2025 at 3:24 PM Chamikara Jayalath via dev < dev@beam.apache.org> wrote: > +1 > > Thanks, > Cham > > On Thu, Jan 23, 2025 at 10:31 AM Kenneth Knowles wrote: > >> +1 (binding) >> >> On Thu, Jan 23, 2025 at 11:50 AM Danny McCormick via dev < >>

Re: Viewer permission on the GCP

Done - thanks! On Fri, Jan 24, 2025 at 11:37 AM Enrique Calderon wrote: > I have just associated this email to my gmail account, could you please > try again with ksobrena...@ks32.dev? > Thank you, > - Quique (@ksobrenat32) > > On 1/24/25 10:15, Danny McCormick via dev wrote:

Re: Viewer permission on the GCP

Moving dev@ to bcc, I can follow up and help make this happen. Hey Quique, I went to add you, but it looks like you need to have an account associated with Google to do so: "Email addresses and domains must be associated with an active Google Account, Google Workspace account, or Cloud Identity ac

Re: [VOTE] Vendored Grpc Dependency Release

ersion number beam-vendor-grpc-1_69_0/0.3. Maybe there is >>> something missing. Need to change >>> https://github.com/apache/beam/blob/b82bde87572b5e2b8f5cebe09aec6373de22b818/vendor/grpc-1_60_1/build.gradle#L26 >>> >>> On Mon, Jan 13, 2025 at 3:01 PM Danny McCormick via dev < >>>

Re: [VOTE] Release 2.62.0, release candidate #1

+1 (binding) - tested some example ML pipelines on the local (interactive) and Dataflow runners. Thanks, Danny On Mon, Jan 13, 2025 at 12:53 PM XQ Hu via dev wrote: > +1 (non-binding) - tested this with a simple Dataflow ML pipeline: > https://github.com/google/dataflow-ml-starter/actions/runs/

Re: Using resource hints or annotations for transform expansion

In my opinion, what you are describing fits the intention/current behavior of resource hints. Resource hints are just hints which allow the runner to optimize the execution environment where possible, so it should be legal for any runner to ignore any hints; as long as we're maintaining that behavi

Re: [VOTE] Vendored Grpc Dependency Release

https://github.com/apache/beam/blob/b82bde87572b5e2b8f5cebe09aec6373de22b818/vendor/grpc-1_60_1/build.gradle#L26 > > On Mon, Jan 13, 2025 at 3:01 PM Danny McCormick via dev < > dev@beam.apache.org> wrote: > >> Hi everyone! I've been working on the release of Beam's

[VOTE] Vendored Grpc Dependency Release

Hi everyone! I've been working on the release of Beam's vendored grpc artifact, following the process [6]: Please review and vote on the release candidate #1 for the version 1.2.3, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) The co

[PROPOSAL] Upgrade vendor grpc

Hi everyone, I would like to volunteer to upgrade the Beam vendored grpc, following the process described in our docs [1]. This will let us get up to date on some of its dependencies, including upgrading to latest protobuf 3 (and will just help us stay on top of grpc patches). My plan is to follo

Re: [ANNOUNCE] New PMC Member: Danny McCormick

Thanks everyone! I'm excited and honored to join! On Fri, Dec 20, 2024 at 3:08 PM Ravi Magham wrote: > Congrats Danny ! > > On Fri, Dec 20, 2024 at 12:03 PM Valentyn Tymofieiev via dev < > dev@beam.apache.org> wrote: > >> So well deserved!! >> >> Congratulations, Danny! >> >> >> >> On Fri, Dec 2

Re: Remove Deprecated v1 AWS IOs

wrote: > +1 > Yes, long waiting thing! > > Makes sense for me since 2+ two years should be quite enough to move to > AWS v2 Io connectors. Though, I'd recommend to announce it on user@ as > well in advance. > > --- > Alexey > > On 2024/12/12 20:25:20

Re: [PROPOSAL] Implement Kerberos support for Python and Java SDK

Upleveling my high level feedback from the doc in case others have thoughts: I'm a little skeptical about baking specific auth logic mostly needed for Kafka into the core worker logic, I wonder if we could make this easier without going this far - one option would be to provide a templated dockerf

Re: [PROPOSAL] Implement Beam SDK harness initialization capability for Python

Thanks - I actually was thinking about this today and was annoyed that we don't have this ability. I'm +1 to the proposed approach. I dropped a comment, but also upleveling in case there is broader interest; it would be nice to have a similar capability for expansion service containers as well. O

Re: [Design] Anomaly Detection with Beam

Thanks - this is exciting! I left a couple comments, but I am a big +1 to this effort! On Fri, Dec 13, 2024 at 3:07 PM XQ Hu via dev wrote: > Thanks for sharing. Great doc! > > On Fri, Dec 13, 2024 at 1:27 PM Shunping Huang > wrote: > >> Hi all, >> >> Recently, I have been working on a design f

Remove Deprecated v1 AWS IOs

Hey everyone, I've been working on upgrading our Java version of protobuf to protobuf 4 (also needed to keep many other dependencies up to date). As part of this, I've found that the AWS v1 KinesisIO [1] is incompatible with protobuf 4 (on upgrade, tests now hang [2]). Other v1 libraries likely are

Re: Automatic spotlessApply

+1, I think this is a good idea. The only downside I can think of is a little bit of extra time per local build, but it seems like a worthwhile tradeoff. 2 additional suggestions: - we could consider doing this across languages as well (e.g. running the python lint and format precommits as part of

Re: Distroless container image naming convention

please) > > Kenn > > On Tue, Nov 26, 2024 at 9:26 AM Danny McCormick via dev < > dev@beam.apache.org> wrote: > >> Thanks - I'm +1 to both doing this work and the naming convention. The >> main naming alternative I can think of is using tags fo

Re: Distroless container image naming convention

Thanks - I'm +1 to both doing this work and the naming convention. The main naming alternative I can think of is using tags for distroless, aka apache/beam_python3.9_sdk:2.61.0-distroless (and probably also apache/beam_python3.9_sdk:latest-distroless), but I think that having separate repos is prob

Beam 2.61.0 Release

Hi, I am happy to announce that Beam 2.61.0 has been fully released. For more information about the release, check out the release notes - https://github.com/apache/beam/releases/tag/v2.61.0 Thanks, Danny

[RESULT] [VOTE] Release 2.61.0, release candidate #3

I'm happy to announce that we have unanimously approved this release. There are 6 approving votes, 3 of which are binding: * Chamikara Jayalath (binding) * Jan Lukavský (binding) * Kenneth Knowles (binding) * XQ Hu * Damon Douglas * Yi Hu There are no disapproving votes. I will now proceed to fin

[VOTE] Release 2.61.0, release candidate #3

Hi everyone, Please review and vote on the release candidate #3 for the version 2.61.0, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) Reviewers are encouraged to test their own use cases with the release candidate, and vote +1 if no i

Re: [VOTE] Release 2.61.0, release candidate #2

ing). Tested this with a simple Dataflow ML job ( > https://github.com/google/dataflow-ml-starter/actions/runs/11915579645/job/33206160169 > ). > > On Mon, Nov 18, 2024 at 9:01 PM Danny McCormick via dev < > dev@beam.apache.org> wrote: > >> Hi everyone, >> Please

[VOTE] Release 2.61.0, release candidate #2

Hi everyone, Please review and vote on the release candidate #1 for the version 2.61.0, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) Reviewers are encouraged to test their own use cases with the release candidate, and vote +1 if no

Re: [VOTE] Release 2.61.0, release candidate #1

> > On Thu, Nov 14, 2024 at 2:11 PM Danny McCormick via dev < > dev@beam.apache.org> wrote: > >> Hi everyone, >> Please review and vote on the release candidate #1 for the version >> 2.61.0, as follows: >> [ ] +1, Approve the release >> [ ] -1, Do not

[VOTE] Release 2.61.0, release candidate #1

Hi everyone, Please review and vote on the release candidate #1 for the version 2.61.0, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) Reviewers are encouraged to test their own use cases with the release candidate, and vote +1 if no

Re: 2.61.0 release

Whoops, forgot the links: release branch - https://github.com/apache/beam/tree/release-2.61.0 milestone for blocking issues - https://github.com/apache/beam/milestone/25 On Wed, Nov 13, 2024 at 1:15 PM Danny McCormick wrote: > I just cut the 2.61.0 release branch [1]. There are currently no rel

Re: 2.61.0 release

I just cut the 2.61.0 release branch [1]. There are currently no release blocking issues. I will now work on making sure the release branch is stable and then will work on creating the first release candidate. Thanks, Danny On Wed, Oct 30, 2024 at 8:41 AM Danny McCormick wrote: > Hi everyone, >

Re: RAG with Apache Beam design proposal

I left a few comments, but overall it looks like a great proposal! Hopefully we can keep building off of the RAG momentum from Beam summit :) Thanks, Danny On Fri, Nov 8, 2024 at 4:38 PM Claudius van der Merwe wrote: > Hi all, > > As Large Language Models (LLMs) continue to transform the ML lan

Re: Plan for upgrading Debezium library for Apache DebeziumIO Connector

I don't know of anyone with plans to do this upgrade; with that said, if you're running into issues with the older version and want to contribute a patch, I at least would generally be in favor of trying to do this upgrade. We would need to be careful to keep it backwards compatible, though, or at

Re: ML for Beam YAML design proposal

Thanks, this LGTM and I think will be a nice addition here. On Mon, Oct 28, 2024 at 12:25 PM Robert Bradshaw via dev < dev@beam.apache.org> wrote: > Thanks. It will be a great feature to be able to do (basic) ML in a > low/no-code setting. > > On Wed, Oct 16, 2024 at 10:29 AM Jeff Kinard wrote:

2.61.0 release

Hi everyone, The next release (2.61.0) branch cut is scheduled for Nov 13, 2024, according to the release calendar [1]. I'd like to perform this release. The plan is to cut the branch on that date, and cherry-pick release-blocking fixes afterwards, if any. Please help with the release by: - Makin

Re: [YAML] Reprocessing failed records

> (1a) Provide a special operation "Unnest" that takes a single field > and emits it as the top-level element. This can of course result in > unschema'd PCollections (which are supported, but generally don't play > as well with the other operations, including xlang ones). I like this the most out

Re: [VOTE] Release 2.60.0, release candidate #2

+1 (non-binding). Ran some ML examples against the interactive and Dataflow runners. Thanks, Danny On Mon, Oct 14, 2024 at 3:45 PM XQ Hu via dev wrote: > +1 (non-binding). Tested the Python SDK with a simple Dataflow ML > pipeline: > https://github.com/google/dataflow-ml-starter/actions/runs/11

Re: [Dataflow][Java][2.52.0] Upgrading to 2.52.0 Surfaces Pubsub Coder Error

I imagine this is no longer helpful to you, Evan, but I ran into this issue this week and tracked down the underlying problem. Basically, a snappy-java version upgrade [1] seems to have changed how the SnappyCoder [2] is serialized. Since this is used by PubSub read, it caused upgrades to start fai

Re: Query Regarding Customizing Apache Beam for Sequence-Based Workload Processing

I'm not sure if I fully understand the use case. When you require ordering, do you need a set of transforms completed on all data before moving to the next set of transforms? Or do you need transforms to complete on a subset of the data before moving to the next subset of the data for the same tran

Re: Question abount Spark Runner's Filter in parDo

This seems like a reasonable optimization to me, I think moving it to a pull request is a good idea - thanks! - Danny On Sun, Sep 22, 2024 at 11:58 PM LDesire wrote: > Hello Beam community. > > I'm currently trying out Spark Runner and while going through the code, > I noticed that when evaluat

Re: [VOTE] Release 2.59.0, release candidate #1

+1 (non-binding). Tested with some ML pipelines against the local and Dataflow runners On Sun, Aug 25, 2024 at 2:32 PM XQ Hu via dev wrote: > +1 (non-binding). Tested this with the simple Dataflow ML pipeline ( > https://github.com/google/dataflow-ml-starter/actions/runs/10540551699/job/29205343

Re: Beam Patch Releases

reed - thanks! Thanks, Danny On Mon, Aug 26, 2024 at 5:11 PM Robert Burke wrote: > I've been burned several times recently through implicit assumptions, so i > felt it was worth mentioning. :) > > On Mon, Aug 26, 2024, 9:09 AM Danny McCormick via dev > wrote: > >> &

Re: Sunsetting Beam Python 3.8 Support

Was about to respond, Rebo you beat me to it! I agree DockerHub is the right thing to look at since Pypi reporting isn't awesome, I think we should only look at the most recent versions though, since 3.8 will work for old versions forever. For 2.58.0 last month (partial month results), I see: "Re

Re: Beam Patch Releases

enneth Knowles wrote: >> >>> This looks great to me. >>> >>> On Fri, Aug 23, 2024 at 4:52 AM Danny McCormick via dev < >>> dev@beam.apache.org> wrote: >>> >>>> Hey folks, we've now run 2 emergency patch releases in the last y

Re: [DISCUSS] Beam 3.0: Paving the Path to the Next Generation Data Processing Framework

I'm generally +1 on doing this as well. Things I'm interested in are: - Expanded turnkey transform support (especially ML). I think moving Beam beyond just being a core "here's some pieces, build it yourself" SDK to a tool that can solve business problems is useful. --- Corollary - if we're increa

  1   2   3   >