Re: [VOTE] Community Examples Repository

Hi guys, For this kind of discussion, I would prefer to avoid Google Doc and directly put the point/proposal on the mailing list. It's easier for the community to follow. The statement is more for 3 because it's more convenient for users to easily find the examples and include in the distributio

[Discussion] Clarify the support story for released Beam versions

Hi all, I would like us to clarify the life cycle of Beam releases a little bit more for our users. I think we increased the predictability significantly by agreeing to a release cadence and kudos to everyone on that. As a follow up to that I would like to address the following problem: It is unc

Jenkins build is back to normal : beam_SeedJob #2404

See

[HELP] Blog post for 2.6.0 Release

Hello all, During my work on the release, I missed that we have started doing blog posts for every release. I decided to announce the release, and start the blog post afterwards - the blog post will only be late for a few days. Please add all your release notes and comments in this doc: https://do

Build failed in Jenkins: beam_SeedJob #2403

See -- GitHub pull request #6188 of commit 3f3e784b136f12ae6ffca29ff4766522ae409b92, no merge conflicts. Setting status of 3f3e784b136f12ae6ffca29ff4766522ae409b92 to PENDING with url https

Build failed in Jenkins: beam_SeedJob #2402

See -- GitHub pull request #6188 of commit 4353d9295466500fb2b069265e045ff298dd28de, no merge conflicts. Setting status of 4353d9295466500fb2b069265e045ff298dd28de to PENDING with url https

[ANNOUNCE] Apache Beam 2.6.0 released!

The Apache Beam team is pleased to announce the release of 2.6.0 version! Apache Beam is an open source unified programming model to define and execute data processing pipelines, including ETL, batch and stream (continuous) processing. See https://beam.apache.org You can download the release here

Re: Renewing the Slack channel join link?

Thanks Luke! On Wed, Aug 8, 2018 at 3:04 PM Lukasz Cwik wrote: > Ask on the #general channel for a slack admin to regenerate a slack self > invite link. > Go to http://s.apache.org and paste the URL in the URL field and set the > ID field to "slack-invite" and click "Get short-URL" > > On Wed, A

Re: Renewing the Slack channel join link?

Ask on the #general channel for a slack admin to regenerate a slack self invite link. Go to http://s.apache.org and paste the URL in the URL field and set the ID field to "slack-invite" and click "Get short-URL" On Wed, Aug 8, 2018 at 2:17 PM Pablo Estrada wrote: > Hello all, > Can we renew the

Re: [VOTE] Community Examples Repository

Charles, I agree with your comments and questions. I want to add one more benefit that was mentioned earlier: A place for people to contribute example that is not tied to the beam release cycle. Such a repository could be a place for casual contributors to add examples over time. On Wed, Aug 8, 2

Re: [VOTE] Community Examples Repository

It looks like the main claim is that 1 and 2 have the benefit of increasing visibility for examples on the Beam site. I agree with Robert's comments on the doc which claim that this is orthogonal to whether a separate repository is created (the comments are unresolved: https://docs.google.com/a/go

Renewing the Slack channel join link?

Hello all, Can we renew the link please? Also, if y{all teach me how, I'll gladly set reminders to do it every month : ) Best -P. -- Got feedback? go/pabloem-feedback

Re: [VOTE] Community Examples Repository

I'd vote for 2. Giving independence to an example repository and creating the right infrastructure to maintain them will give visibility to the efforts our users are creating to solve their uses cases with Beam. I also want to make the process of sharing common work more easily. Re:The examples t

Re: [VOTE] Community Examples Repository

I would go for 1 or 3 to be consistent. Regards JB On 08/08/2018 18:38, David Cavazos wrote: > Hi everyone! > > We discussed several options as well as some of the implications of each > option. Please vote for your favorite option, feel free to back it up > with any reasons that make you feel t

Re: Schema Aware PCollections

Yes, this should be possible eventually. In fact, limited version of this functionality is already supported for Beans (e.g. see this test )

Re: [VOTE] Community Examples Repository

I would vote for 3. My reasoning is that Java has a good mechanism to get a starter/example project going by using the the maven archetypes already. Our quickstart guide for Apache Beam for the Java SDK already covers generating the examples archetype. We could point users to the starter project a

Re: [VOTE] Community Examples Repository

I guess I'm voting for 2. Tests obviously belong in the code repo, both as a sample usage (not "how-to", more like "man") and, well, for testing. These pipelines might not be annotated as completely and include hitting edge cases and other non-standard situations. Any example where the primary pu

Re: [VOTE] Community Examples Repository

2 - examples that rely on experimental API can still stay in where they are because such examples could be changed. -Rui On Wed, Aug 8, 2018 at 10:52 AM Charles Chen wrote: > 3 - We benefit from increased test coverage by having examples together > with the rest of the code. As Robert mentions

Re: [VOTE] Community Examples Repository

3 - We benefit from increased test coverage by having examples together with the rest of the code. As Robert mentions in the doc, hosting the Beam examples in the main repository is the best way to keep the examples visible, tested and maintained. Given that we recently moved to a single reposito

Re: Looking for a re-review of 6101

Hi, I gonna take a look. Regards JB On 08/08/2018 19:38, John Rudolf Lewis wrote: > I'm looking for a re-review of my SqsIO contribution. > > https://github.com/apache/beam/pull/6101 > > I would like to see this accepted into master so that we can start to > use the daily builds in one of our

Looking for a re-review of 6101

I'm looking for a re-review of my SqsIO contribution. https://github.com/apache/beam/pull/6101 I would like to see this accepted into master so that we can start to use the daily builds in one of our projects.

Re: Portable streaming side inputs

I have been using this inside Dataflow with the PushbackSideInputDoFnRunner as the window mapping fn: https://github.com/lukecwik/incubator-beam/commit/4324185192b27026270ee342b01720ff40e71df8 The input is a KV with the key being a nonce and the value being a window, the output must be a KV with t

Re: [VOTE] Community Examples Repository

2 - Similar to Huygaa, I see value in keeping a core set of examples tested and maintained against head. At the same time I understand the value of a growing set of community grown examples that are targeted against a pre-defined versions of Beam and not necessarily updated at every release. On We

Re: [VOTE] Community Examples Repository

2 - I like the idea of having a separate repo where we can have more freedom to check in examples. However, we benefit from having immediate core examples in Beam for testing purposes. On Wed, Aug 8, 2018 at 9:38 AM David Cavazos wrote: > Hi everyone! > > We discussed several options as well as

[VOTE] Community Examples Repository

Hi everyone! We discussed several options as well as some of the implications of each option. Please vote for your favorite option, feel free to back it up with any reasons that make you feel that way. 1) Move *all* samples to a *new *examples* repository* 2) Move *some* samples to a *new *exampl

Portable streaming side inputs

I'm working on support for side inputs in streaming mode in the portable Flink runner [1]. The runner would be responsible for holding main inputs (durably) when side inputs are not available. To check if side inputs are available, the window mapping function is required (see SimplePushbackSideIn

Build failed in Jenkins: beam_Release_Gradle_NightlySnapshot #135

See Changes: [apilloud] Pin Dataflow Nexmark Autoscale to 4 [relax] Make sure that SchemaCoder overrides consistentWithEquals. [relax] Address code-review comments. [apilloud] Make event coun