Re: Publishing release notes to the Github releases page

Sure thing! Just submitted a PR here: https://github.com/apache/beam/pull/11918 On Wed, Jun 3, 2020 at 2:00 PM Kyle Weaver wrote: > Thanks Julien! Do you mind making a PR for this? > > Also, I filed https://issues.apache.org/jira/browse/BEAM-10188 so we have > a JIRA to track further improvement

Re: Running Beam python pipeline on Spark

Thanks for the pointers, Thomas. Let me give it a shot tomorrow. Thanks, Xinyu On Wed, Jun 3, 2020 at 7:13 PM Thomas Weise wrote: > If all Python dependencies are pre-installed on the yarn container hosts, > then you can use the process environment to spawn processes, like so: > > > https://git

Re: [VOTE] Release 2.22.0, release candidate #1

+1, I validated the python quickstarts. On Wed, Jun 3, 2020 at 2:01 PM Brian Hulette wrote: > A few more useful links: I verified the release-2.22.0 branch with jenkins > jobs in a PR [1]. There are some VR test failures that I don't believe > represent regressions, summarized in a comment [2],

Re: Running Beam python pipeline on Spark

If all Python dependencies are pre-installed on the yarn container hosts, then you can use the process environment to spawn processes, like so: https://github.com/lyft/flinkk8soperator/blob/bb8834d69e8621d636ef2085fdc167a9d2c2bfa3/examples/beam-python/src/beam_example/pipeline.py#L16-L17 Thomas

Running Beam python pipeline on Spark

Hi, folks, I am trying to do some experiment to run a simple "hello world" python pipeline on a remote Spark cluster on Hadoop. So far I ran the SparkJobServerDriver on the Yarn application master and managed to submit a python pipeline to it. SparkPipelineRunner was able to run the portable pipel

[PROPOSAL] Abstractions for Schema-Aware IOs (Was: Leveraging SQL TableProviders for Cross-Language IOs)

Hi everyone, You may remember the proposal [1] I sent out early this year aiming to unify Beam SQL Tables and Row-oriented cross-language IOs. I received a lot of feedback that it was a good idea, but the specific implementation was a little odd since it relied on existing Beam SQL concepts. I've

Re: [VOTE] Release 2.22.0, release candidate #1

A few more useful links: I verified the release-2.22.0 branch with jenkins jobs in a PR [1]. There are some VR test failures that I don't believe represent regressions, summarized in a comment [2], also discussed in another dev@ thread [3]. [1] https://github.com/apache/beam/pull/11770 [2] https:/

Re: Publishing release notes to the Github releases page

Thanks Julien! Do you mind making a PR for this? Also, I filed https://issues.apache.org/jira/browse/BEAM-10188 so we have a JIRA to track further improvements. On Wed, Jun 3, 2020 at 4:39 PM Julien Phalip wrote: > I see that Kyle added >

Re: [VOTE] Release 2.22.0, release candidate #1

The contents, hashes, and signatures look good to me. I also tried running a simple pipeline with one of the wheels. +1 (binding) On Wed, Jun 3, 2020 at 1:15 PM Brian Hulette wrote: > Hi everyone, > Please review and vote on the release candidate #1 for the version 2.22.0, > as follows: > [ ]

Re: Publishing release notes to the Github releases page

Thanks Julien! I can try to integrate this into the release scripts when finalizing 2.22.0. The only other nice to have would be some kind of script to generate RELEASE_NOTES from the blog post markdown [1]. It shouldn't be too tricky to just trim off the front matter and license. Brian [1] http

Re: Publishing release notes to the Github releases page

I see that Kyle added some steps in the release guide, and the release notes now appear in Github for the recent 2.21.0 releas

[VOTE] Release 2.22.0, release candidate #1

Hi everyone, Please review and vote on the release candidate #1 for the version 2.22.0, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) The complete staging area is available for your review, which includes: * JIRA release notes [1], *

Beam Summit Status Report - 6/3

Hi folks, I wanted to provide you with the Beam Summit Status report from today's meeting. If you would like to join the next public meeting on Wednesday, June 8th at 11:30 AM PST please let me know and I will send a calendar invite over to you! Also don't forget to submit your CFP

Re: [RESULT][VOTE] Accept the Firefly design donation as Beam Mascot - Deadline Mon April 6

Thank you Ismael for reviewing and merging the pull request. Now the mascot files can be found here https://beam.apache.org/community/mascot/ on the website and here https://github.com/apache/beam/tree/master/website/www/site/static/images/mascot in the repo. This project is complete now, therefo

Re: Snowflake account donation for ongoing testing

Hi Pablo, I was wondering why there was no response :) >From our side, we need a signed partner agreement which covers usage of our demo accounts. If that is signed, we can then provide an account with $400 free usage per month which should be more than enough to cover any testing. We can put som

Re: Snowflake account donation for ongoing testing

Thanks, Pablo! +Harsha Kapre , who appears to have been accidentally dropped from the original thread. -Tyler On Mon, May 18, 2020, 15:21 Pablo Estrada wrote: > Hello Harsha! > Thanks for contributing the connector, and the support for testing it. > This is great. > I am a PMC member, and would

Re: Snowflake account donation for ongoing testing

Sounds good. I'll move this discussion to the PMC list to continue. Best -P. On Wed, Jun 3, 2020 at 9:08 AM Harsha Kapre wrote: > Hi Pablo, > > I was wondering why there was no response :) > > From our side, we need a signed partner agreement which covers usage of > our demo accounts. If that is

Re: Snowflake account donation for ongoing testing

Oops. I'm sorry to have dropped you Harsha : / Let me know what are next steps! Best -P. On Wed, Jun 3, 2020 at 9:03 AM Tyler Akidau wrote: > Thanks, Pablo! +Harsha Kapre , who appears to > have been accidentally dropped from the original thread. > > -Tyler > > On Mon, May 18, 2020, 15:21 Pablo