Re: Extending 2.41.0 Java snapshot TTL

I think you could change the TTL on the Jenkins side (That sound right to you Danny?) but I'm not sure we could preserve a specific snapshot without keeping all of them... On Thu, Jul 21, 2022 at 4:16 PM Ahmet Altay wrote: > Thank you for the email Daniel. > > Adding people who could help: @Kenn

Re: Extending 2.41.0 Java snapshot TTL

14.0-SNAPSHOT/ >>> >>> On Thu, Jul 21, 2022 at 8:54 PM Evan Galpin wrote: >>> >>>> Admittedly this is potentially self-serving, but I feel there could be >>>> mutual benefit. >>>> >>>> I have a similar situation where I wan

Re: [ANNOUNCE] New committer: John Casey

Congratulations John! On Fri, Jul 29, 2022 at 1:09 PM Danny McCormick via dev wrote: > Congrats John and welcome! This is well deserved! > > On Fri, Jul 29, 2022 at 4:07 PM Kenneth Knowles wrote: > >> Hi all, >> >> Please join me and the rest of the Beam PMC in welcoming a new committer: >> Joh

Re: Easy Multi-language via a SchemaTransform-aware Expansion Service

I think there are some practical advantages to having the ability to support a dynamic version---at previous places where I've worked having Kafka's Schema Service was incredibly useful for data processing (it was a Java/Scala shop and we mostly used a "decode to POJO" approach rather than codegen.

Re: Easy Multi-language via a SchemaTransform-aware Expansion Service

Indeed, there's nothing stopping you from doing codegen where it's useful but I think it's probably easier to implement codegen from dynamic than it is to go the other way around (Avro vs Proto) On Fri, Aug 5, 2022 at 1:15 PM Chamikara Jayalath wrote: > > > On Fri, Aug 5, 2022 at 12:00 PM Byron

Re: Easy Multi-language via a SchemaTransform-aware Expansion Service

I like that idea, sort of like Kafka’s Schema Service but for transforms? On Mon, Aug 8, 2022 at 2:45 PM Robert Bradshaw via dev wrote: > This is a great idea. I would like to approach this from the > perspective of making it easy to provide a catalog of well-defined > transforms for use in expa

Re: [ANNOUNCE] Apache Beam 2.41.0 Released

Thanks Kiley! On Fri, Aug 26, 2022 at 9:37 AM Ahmet Altay via dev wrote: > Thank you Kiley! > > On Fri, Aug 26, 2022 at 6:38 AM P Singh > wrote: > >> Hi Team, >> >> Looking forward to trying and testing the new version, It's always >> fascinating. >> >> On Fri, 26 Aug 2022 at 03:45, Pablo Estra

Updated invitation: Beam Community Meetup @ Thu Sep 22, 2022 9:30am - 10:30am (PDT) (dev@beam.apache.org)

BEGIN:VCALENDAR PRODID:-//Google Inc//Google Calendar 70.9054//EN VERSION:2.0 CALSCALE:GREGORIAN METHOD:REQUEST BEGIN:VEVENT DTSTART:20220922T163000Z DTEND:20220922T173000Z DTSTAMP:20220922T151507Z ORGANIZER;CN=Clowder Events:mailto:c_m8k910vjcu6b4b7udo0kngo4d4@group.calen dar.google.com UID:1c9cv

Re: [ANNOUNCE] New committer: Ritesh Ghorse

Congratulations! On Thu, Nov 3, 2022 at 4:21 PM Austin Bennett wrote: > Congratulations, and Thanks @riteshgho...@apache.org! > > On Thu, Nov 3, 2022 at 4:17 PM Sachin Agarwal via dev > wrote: > >> Congrats Ritesh! >> >> On Thu, Nov 3, 2022 at 4:16 PM Kenneth Knowles wrote: >> >>> Hi all, >>>

Re: [ANNOUNCE] New committer: Yi Hu

Congratulations! On Wed, Nov 9, 2022 at 11:00 AM Pablo Estrada via dev wrote: > +1 thanks Yi : D > > On Wed, Nov 9, 2022 at 10:47 AM Danny McCormick via dev < > dev@beam.apache.org> wrote: > >> Congrats Yi! I've really appreciated the ways you've consistently taken >> responsibility for improvin

Re: Credentials Rotation Failure on IO-Datastores cluster

Is there something we need to do here? On Thu, Dec 1, 2022 at 10:10 AM Apache Jenkins Server < jenk...@builds.apache.org> wrote: > Something went wrong during the automatic credentials rotation for > IO-Datastores Cluster, performed at Thu Dec 01 15:00:47 UTC 2022. It may be > necessary to check

Re: [Proposal] | Move FileIO and TextIO from :sdks:java:core to :sdks:java:io:file

Talk it with a grain of salt since I'm not even a committer, but is perhaps the reorganization of Beam into smaller components the real work of a 3.0 effort? Splitting of Beam into smaller more independently managed components would be a pretty huge breaking change from a dependency management pers

Re: A Declarative API for Apache Beam

Since Robert opened that can of worms (and we happened to talk about it yesterday)... :-) I figured I'd also share my start on a "port" of dbt to the Beam SDK. This would be complementary as it doesn't really provide a way of specifying a pipeline, more orchestrating and packaging a complex pipeli

Re: A Declarative API for Apache Beam

And I guess also a PR for completeness to make it easier to find going forward instead of my random repo: https://github.com/apache/beam/pull/24670 On Wed, Dec 14, 2022 at 2:37 PM Byron Ellis wrote: > Since Robert opened that can of worms (and we happened to talk about it > yesterday)... :-) > >

Re: A Declarative API for Apache Beam

gt; On Thu, Dec 15, 2022 at 12:48 AM Chamikara Jayalath via dev < >>>> dev@beam.apache.org> wrote: >>>> >> >>>> >> +1 for these proposals and agree that these will simplify and >>>> demystify Beam for many new users. I think whe

Testing Multilanguage Pipelines?

Hi all, I spent some more time adding things to my dbt-for-Beam clone ( https://github.com/apache/beam/pull/24670) and actually made a fair amount of progress, including starting to add in the profile support so I can start to run it against real workloads (though at the moment only the "test" con

Re: Testing Multilanguage Pipelines?

al via dev < > dev@beam.apache.org> wrote: > >> > >> Given the increasing importance of multi language pipelines, it does > seem that we should expand the capabilities of the DirectRunner or just go > all in on FlinkRunner for testing and local / small scale development

Re: Testing Multilanguage Pipelines?

t/src/apache_beam/runners/universal.ts#L34 > > https://github.com/apache/beam/blob/master/sdks/typescript/src/apache_beam/transforms/python.ts#L60 > > https://github.com/apache/beam/blob/release-2.43.0/sdks/java/extensions/python/src/main/java/org/apache/beam/sdk/extensions/python/Python

Thoughts on extensions/datasketches vs adding to the existing sketching library?

Hi everyone, I was looking at adding at least a couple of the sketches from the Apache Datasketches library to the Beam Java SDK and I was wondering if folks had a preference for adding to the existing "sketching" extension vs splitting it out into its own extension? The reason I ask is that ther

Re: Thoughts on extensions/datasketches vs adding to the existing sketching library?

;t know whether it still has an advantage or not. >> >> On Wed, Jan 18, 2023 at 10:41 AM Byron Ellis via dev >> wrote: >> >>> Hi everyone, >>> >>> I was looking at adding at least a couple of the sketches from the >>> Apache Datasketches

Re: Thoughts on extensions/datasketches vs adding to the existing sketching library?

ketch implementations. However this was a number of >>> years ago, and I don't know whether it still has an advantage or not. >>> >>> On Wed, Jan 18, 2023 at 10:41 AM Byron Ellis via dev < >>> dev@beam.apache.org> wrote: >>> >>>> Hi e

Re: Thoughts on extensions/datasketches vs adding to the existing sketching library?

o so at this point. And like Byron brought up, a key >> aspects of sketches is their serialized form being compatible so the user >> really needs to know exactly what implementation they are using. >> >> Kenn >> >> On Wed, Jan 18, 2023 at 12:22 PM Byron Ellis via dev

Convenience methods for Schema definition in Java

Hi all, I've been spending a lot of time recently working with Schemas in Java on Beam and I noticed that there aren't convenience methods for creating nullable versions of a field like there are for the NOT NULL versions. If folks are interested I made a small quality-of-life PR

Re: OpenJDK8 / OpenJDK11 container deprecation

FWIW I am Team Upgrade Docker :-) On Tue, Feb 14, 2023 at 2:53 PM Luke Cwik via user wrote: > I made some progress in testing the container and did hit an issue where > Ubuntu 22.04 "Jammy" is dependent on the version of Docker installed. It > turns out that our boot.go crashes with "runtime/cgo

Re: [DISCUSS] @Experimental, @Internal, @Stable, etc annotations

Honestly, I think APIs could be pretty simply defined if you think of it in terms of the user: @Deprecated = this was either stable or evolve but the functionality/interface will go away at a future date @Stable = the user of this API opting out of changes to functionality and interface. For exam

Re: [ANNOUNCE] New committer: Anand Inguva

Congrats Anand! On Mon, Apr 24, 2023 at 9:54 AM Ahmet Altay via dev wrote: > Congratulations Anand! > > On Mon, Apr 24, 2023 at 8:05 AM Kerry Donny-Clark via dev < > dev@beam.apache.org> wrote: > >> Great work Anand, this is well deserved. >> >> >> On Mon, Apr 24, 2023 at 10:35 AM Yi Hu via dev

Re: [ANNOUNCE] New committer: Damon Douglas

Congrats Damon! On Mon, Apr 24, 2023 at 8:57 PM Austin Bennett wrote: > thanks for all you do @Damon Douglas ! > > On Mon, Apr 24, 2023 at 1:00 PM Robert Burke wrote: > >> Congratulations Damon!!! >> >> On Mon, Apr 24, 2023, 12:52 PM Kenneth Knowles wrote: >> >>> Hi all, >>> >>> Please join m

Re: Thoughts on coder evolution

I think I'm not understanding the use case here? Are we talking about encoding of data in motion (e.g. between stages of a streaming pipeline) or data at rest? (e.g. input formats and output formats) Or maybe something else? On Wed, May 3, 2023 at 6:58 AM Jan Lukavský wrote: > Hi, > > I'd like t

Re: Proposal to reduce the steps to make a Java transform portable

Or perhaps the other way around? If you have a Schema we can auto-generate the associated builder on the PTransform? Either way, more DRY. On Tue, May 30, 2023 at 10:59 AM Robert Bradshaw via dev < dev@beam.apache.org> wrote: > +1 to this simplification, it's a historical artifact that provides n

Re: Proposal to reduce the steps to make a Java transform portable

Is it actually necessary for a PTransform that is configured via the Schema mechanism to also be one that uses RowCoder? Those strike me as two separate concerns and unnecessarily limiting. On Tue, May 30, 2023 at 1:29 PM Chamikara Jayalath wrote: > +1 for the simplification. > > On Tue, May 30,

Re: Proposal to reduce the steps to make a Java transform portable

Sure, I get that… though perhaps we should consider just going to something Avro for portable coding rather than something custom. On Tue, May 30, 2023 at 2:22 PM Chamikara Jayalath wrote: > Input/output PCollection types at least have to be portable Beam types [1] > for cross-language to work.

Re: Calcite to Beam mapping, possible?

Any particular reason you're not using the SqlTransform to do that? That'd probably be a good place to start code-wise (perhaps with something a bit newer than 2.29) to see how Beam uses Calcite today to produce a PTransform from a query plan. On Wed, Jun 14, 2023 at 5:32 AM Soumyadeep Mukhopadhya

Re: Calcite to Beam mapping, possible?

r your help! > > With regards, > Soumyadeep Mukhopadhyay. > > > > > On Wed, Jun 14, 2023 at 7:47 PM deepak kumar wrote: > >> I would be really interested to contribute as well on calcite to >> PTransform kind of conversions >> >> On Wed, Jun 14, 20

Re: [DISCUSS] Enable Github Discussions?

-1. This just leads to needless fragmentation not to mention being at the mercy of a specific technology provider. On Mon, Jul 3, 2023 at 11:39 AM XQ Hu via dev wrote: > +1 with GH discussion. > If Airflow can do this https://github.com/apache/airflow/discussions, I > think we can do this as wel

Re: [DISCUSS] Upgrade vended guava version

It's not designed to be merged, but there is a PR with this change in it--- https://github.com/apache/beam/pull/27695/files#diff-0435a83a413ec063bf7e682cadcd56776cd18fc878f197cc99a65fc231ef2047 On Mon, Jul 31, 2023 at 10:07 AM Ahmet Altay via dev wrote: > Hi Hong, > > Thank you for reaching o

[Request for Feedback] Swift SDK Prototype

Hello everyone, A couple of months ago I decided that I wanted to really understand how the Beam FnApi works and how it interacts with the Portable Runner. For me at least that usually means I need to write some code so I can see things happening in a debugger and to really prove to myself I under

The Swift SDK now works on Linux

Hello everyone, I got a chance to test and fix a small issue that prevented the Swift SDK from working on Linux boxes due to differences in the way Data is handled in non-macOS Foundation implementations (which hopefully will stop being an issue with with the native-Swift Foundation revamp). I als

Re: [Request for Feedback] Swift SDK Prototype

ft > SDK from folks currently subscribed to the +user > list. > > On Wed, Aug 16, 2023 at 6:53 PM Byron Ellis via dev > wrote: > >> Hello everyone, >> >> A couple of months ago I decided that I wanted to really understand how >> the Beam FnApi works and

Re: [Request for Feedback] Swift SDK Prototype

raph is a separate operation than defining a > composite. > > On Thu, Aug 17, 2023 at 4:28 PM Chamikara Jayalath > wrote: > >> Thanks Byron. This sounds great. I wonder if there is interest in Swift >> SDK from folks currently subscribed to the +user >> list. >

Re: [Request for Feedback] Swift SDK Prototype

before adding the ability to make >>> composites in the language itself. With the way the SDK is laid out adding >>> composites to the pipeline graph is a separate operation than defining a >>> composite. >>> >>> On Thu, Aug 17, 2023 at 4:28 PM Chamikara Ja

Re: [Request for Feedback] Swift SDK Prototype

gt;>>> Thanks Cham, >>>> >>>> Definitely happy to open a draft PR so folks can comment---there's not >>>> as much code as it looks like since most of the LOC is just generated >>>> protobuf. As for the support, I definitely want to add

Re: [Request for Feedback] Swift SDK Prototype

here's the draft PR: >>>>> >>>>> https://github.com/apache/beam/pull/28062 >>>>> >>>>> I haven't had a chance to test it on my M1 machine yet though (there's >>>>> a good chance there are a few places that need to

Re: [Request for Feedback] Swift SDK Prototype

ut it would allow for >>>>>>> SQL >>>>>>> statements on individual files with projection pushdown supported for >>>>>>> things like Parquet which could have some cool and performant data lake >>>>>>> applications.

Re: [ANNOUNCE] New committer: Ahmed Abualsaud

Congratulations! On Thu, Aug 24, 2023 at 5:34 PM Robert Burke wrote: > Congratulations Ahmed!! > > On Thu, Aug 24, 2023, 4:08 PM Chamikara Jayalath via dev < > dev@beam.apache.org> wrote: > >> Congrats Ahmed!! >> >> On Thu, Aug 24, 2023 at 4:06 PM Bruno Volpato via dev < >> dev@beam.apache.org>

Re: [Request for Feedback] Swift SDK Prototype

ne { pipeline in ... } and just instead have things attach >>>>>>>> themselves >>>>>>>> to the pipeline implicitly. >>>>>>>> >>>>>>>> That said, there are some interesting IO possibilities that would >>>&g

Re: [Request for Feedback] Swift SDK Prototype

al transforms for the reasons you suggest. :-) >>>>>>>>> I've got >>>>>>>>> the basics of a composite transform (there's an equivalent wordcount >>>>>>>>> example) and am hooking it into the

Re: [Request for Feedback] Swift SDK Prototype

>>>>>>>>>> default >>>>>>>>>> stream implementation call the equivalent of processElement. To make >>>>>>>>>> that >>>>>>>>>> performant I need to implement an @DoFn macro and I just haven&

Re: [Request for Feedback] Swift SDK Prototype

;>> That said it would also be natural to provide elementwise >>>>>>>>>>> pardos--- that would probably mean having explicit type signatures >>>>>>>>>>> in the >>>>>>>>>>> closure. I had that a

Re: [Request for Feedback] Swift SDK Prototype

typechecked at compile time BTW. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> the (filename,_,_) is a tuple spreading construct like you have >>>>>>>>>>>> in ES6

Re: [Request for Feedback] Swift SDK Prototype

... >>>>>>>>>>>>>> output.emit(data) >>>>>>>>>>>>>> >>>>>>>>>>>>>> } >>>>>>>>>>>>>> &g

Re: Runner Bundling Strategies

I've actually wondered about this specifically for streaming... if you're writing a pipeline there it seems like you're often going to want to put high fixed cost things like database connections even outside of the bundle setup. You really only want to do that once in the lifetime of the worker it

Re: User-facing website vs. contributor-facing website

I feel like that's actually pretty easy with Github actions? I think maybe there's even one that exists Github Pages and probably any other static site generator thingy we could care to name. Related, I stumbled across this the other day: https://github.com/apache/beam-site which appears to be unus

Re: [ANNOUNCE] New PMC Member: Valentyn Tymofieiev

Congrats! On Tue, Oct 3, 2023 at 12:40 PM Danielle Syse via dev wrote: > Congrats Valentyn!! > > On Tue, Oct 3, 2023 at 2:59 PM Ahmet Altay via dev > wrote: > >> Congratulations Valentyn! Well deserved! >> >> On Tue, Oct 3, 2023 at 11:54 AM Ritesh Ghorse via dev < >> dev@beam.apache.org> wrote:

Re: [ANNOUNCE] New PMC Member: Robert Burke

Congrats! On Tue, Oct 3, 2023 at 12:40 PM Danielle Syse via dev wrote: > Congrats Rebo!!! Can't wait to work more closely with you. > > On Tue, Oct 3, 2023 at 2:56 PM Ahmet Altay via dev > wrote: > >> Congratulations Robert! Well deserved! >> >> On Tue, Oct 3, 2023 at 11:54 AM Ritesh Ghorse via

Re: [ANNOUNCE] New PMC Member: Alex Van Boxel

Congrats! On Tue, Oct 3, 2023 at 12:40 PM Danielle Syse via dev wrote: > Congratulations Alex!! Definitely well deserved! > > On Tue, Oct 3, 2023 at 2:57 PM Ahmet Altay via dev > wrote: > >> Congratulations Alex! Well deserved! >> >> On Tue, Oct 3, 2023 at 11:54 AM Ritesh Ghorse via dev < >> de

Re: [QUESTION] Why no auto labels?

This feels like something that maybe should be more explicit? Overloading the transform name to provide a unique stable id feels like perhaps too much magic... also maybe feels like this is leaking specific runner behavior? I get that it's convenient On Wed, Oct 4, 2023 at 9:16 AM Robert Bradshaw

Re: [PROPOSAL] [Nice-to-have] CI job names and commands that match

I'm +1 on standardizing the names and while I don't have a strong opinion on which standard (so long as it's only one) using the Gradle name seems like a perfectly good choice... I don't know the GHA setup well enough, but would that help maintain those? Presumably the various actions all eventuall

Re: [YAML] Fileio sink parameterization (streaming, sharding, and naming)

FWIW dbt (which is also YAML and has this problem for other reasons) does something like this. It also chooses to assume that everything is a string but allows users to use the Jinja templating language to make those strings dynamic where needed. Syntactically I think that's a bit nicer to look at

Re: [YAML] Fileio sink parameterization (streaming, sharding, and naming)

e: > On Tue, Oct 10, 2023 at 7:22 AM Byron Ellis via dev > wrote: > >> FWIW dbt (which is also YAML and has this problem for other reasons) does >> something like this. It also chooses to assume that everything is a string >> but allows users to use the Jinja templating

Re: [ANNOUNCE] New Committer: Sam Whittle

Congrats Sam! On Mon, Oct 16, 2023 at 10:32 AM Chamikara Jayalath via dev < dev@beam.apache.org> wrote: > Congrats Sam! > > On Mon, Oct 16, 2023 at 9:32 AM Kenneth Knowles wrote: > >> Hi all, >> >> Please join me and the rest of the Beam PMC in welcoming a new >> committer: Sam Whittle (scwhit..

Re: [NOTICE] Deprecation Avro classes in "core" and use "extensions/avro" instead for Java SDK

Awesome! On Wed, Oct 18, 2023 at 1:14 PM Alexey Romanenko wrote: > Heads up! > > Finally, all Avro-related code and Avro dependency, that was deprecated > before (see a message above), has been removed from Beam Java SDK “core” > module [1]. We believe that it was a sufficient number of Beam rel

Re: [YAML] Aggregations

Is it worth taking a look at similar prior art in the space? The first one that comes to mind is Transform, but with the dbt labs acquisition that spec is a lot harder to find. Rill is pretty similar though. On Wed, Oct 18, 2023 at 1:12 PM Robert Bradsh

Re: [YAML] Aggregations

Rill is definitely SQL-oriented but I think that's going to be the most common. Dataframes are explicitly modeled on the relational approach so that's going to look a lot like SQL, which leaves us with S-style formulas (which I like but are pretty niche) and I guess pivot tables coming from the spr

Re: [YAML] Aggregations

On Thu, Oct 19, 2023 at 9:28 AM Robert Bradshaw wrote: > On Thu, Oct 19, 2023 at 9:00 AM Byron Ellis wrote: > > > > Rill is definitely SQL-oriented but I think that's going to be the most > common. Dataframes are explicitly modeled on the relational approach so > that's going to look a lot like

Re: Implementing tuple type support in for ClickHouse connector

I'm not all that familiar with Clickhouse, but from the docs it looks like Tuples are basically some sort of anonymous structure so I'm guessing you'll want to encode it as a Row itself... that's a lossy conversion though so probably some sort of Logical Type that wraps the Row object is the "right

Re: Implementing tuple type support in for ClickHouse connector

Indeed, hence me cheaping out and just tagging the base type with the one I want (in my case things like "UUID" which are just byte arrays). :-) In any case, the (very very very early) draft PR is here: https://github.com/apache/beam/pull/29569 On Mon, Dec 4, 2023 at 9:26 AM Robert Bradshaw via d

Re: [ANNOUNCE] New Committer: Svetak Sundhar

Congrats Svetak! On Mon, Feb 12, 2024 at 6:57 PM Shunping Huang via dev wrote: > Congratulations, Svetak! > > On Mon, Feb 12, 2024 at 9:50 PM XQ Hu via dev wrote: > >> Great job, Svetak! Thanks for all your contributions to Beam!!! >> >> On Mon, Feb 12, 2024 at 4:44 PM Valentyn Tymofieiev via d

Re: [ANNOUNCE] New Committer: XQ Hu

Congrats XQ! On Mon, Jun 24, 2024 at 1:40 PM Kenneth Knowles wrote: > Hi all, > > Please join me and the rest of the Beam PMC in welcoming a new committer: > XQ Hu (x...@apache.org). > > XQ has been with Beam since mid-2022. In that time he's contributed many > fixes throughout the project, incl

Re: [ANNOUNCE] New PMC Member: Danny McCormick

Congrats Danny! On Fri, Dec 20, 2024 at 11:02 AM Svetak Sundhar via dev wrote: > Congrats Danny! Really well deserved, and excited to see all that you do > to continue to take the project forward. > > > Svetak Sundhar > > Data Engineer > s vetaksund...@google.com > > > > On Fri, Dec 20, 2024 a