Re: Error in FlinkRunnerTest.test_external_transforms

I think I've seen this sometimes as a flake. Best -P. On Tue, Jun 30, 2020 at 11:23 AM Maximilian Michels wrote: > Is this a flake? I don't see this in master atm. > > -Max > > On 27.06.20 02:59, Alex Amato wrote: > > Hi, > > > > I was wondering if this is something wrong with my PR > >

Re: [PROPOSAL] Preparing for Beam 2.23.0 release

Valentyn, +tobiasz.kedzier...@polidea.com added a github action for building python wheel files ( https://github.com/apache/beam/pull/11877). You should be able to build the wheel files using this github action instead of using the beam-wheels repo and Travis. Please give it a try during the rele

Re: [ANNOUNCE] New committer: Aizhamal Nurmamat kyzy

Congrats! On Tue, Jun 30, 2020 at 5:05 PM Aizhamal Nurmamat kyzy wrote: > Thanks everyone! It's been really nice to work with all of you :) > > On Tue, Jun 30, 2020, 1:45 PM Ismaël Mejía wrote: > >> Congratulations Aizhamal! >> And thanks for all the dedication to the project. >> >> On Tue, Ju

Re: [ANNOUNCE] New committer: Aizhamal Nurmamat kyzy

Thanks everyone! It's been really nice to work with all of you :) On Tue, Jun 30, 2020, 1:45 PM Ismaël Mejía wrote: > Congratulations Aizhamal! > And thanks for all the dedication to the project. > > On Tue, Jun 30, 2020 at 8:08 PM Maximilian Michels wrote: > > > > Congrats Aizhamal! > > > > On

Re: [DISCUSS] ReadAll pattern and consistent use in IO connectors

I don't think an enum of most common closures will work. The input types are typically generics that are made concrete by the caller who also provides the closures. I think Luke's (2) is the same idea as my "Java still assembles it [using opaque Python closures/transforms]". It seems like an approa

Re: Composable DoFn IOs Connection Reuse

I agree at the level of GroupIntoBatches. The part that is similar is if you implement GroupIntoBatches with a new primitive supporting runner-scoped state where the state is per-shard and the runner is able to split state shards. Kenn On Tue, Jun 30, 2020 at 9:22 AM Luke Cwik wrote: > I'm not

Re: Exploding windows and FnApiDoFnRunner

Just to close this thread. This one is addressed now via https://issues.apache.org/jira/browse/BEAM-10303 Thanks Luke for taking care. On Mon, May 4, 2020 at 9:31 PM Luke Cwik wrote: > > Kenn, the optimization is not complex, just never done. > > The FnApiDoFnRunner was rewritten to be designed

Re: [ANNOUNCE] New committer: Aizhamal Nurmamat kyzy

Congratulations Aizhamal! And thanks for all the dedication to the project. On Tue, Jun 30, 2020 at 8:08 PM Maximilian Michels wrote: > > Congrats Aizhamal! > > On 30.06.20 17:34, Jan Lukavský wrote: > > Congratulations Aizhamal! > > > > On 6/30/20 1:35 PM, Alexey Romanenko wrote: > >> Congratula

Re: Is there an easy way to figure out why my build failed?

This rabbitmq:test stuckness has been a common pain point for me recently. I did some investigation and found one issue with the test and also added a class level timeout for the test. Opened up https://github.com/apache/beam/pull/12137 On Tue, Jun 30, 2020 at 11:22 AM Maximilian Michels wrote:

Re: Error in FlinkRunnerTest.test_external_transforms

Is this a flake? I don't see this in master atm. -Max On 27.06.20 02:59, Alex Amato wrote: Hi, I was wondering if this is something wrong with my PR or an issue in master. Thanks for your help. Seeing this in my PR's presubmit https://ci-beam.apac

Re: Is there an easy way to figure out why my build failed?

Hi Alex, Fully agree with you that it can be hard to find the cause for a failing build. You basically need to know the exact keyword to grep for. The reason is that Jenkins does not understand all build logs to display the error directly in the UI. I often do the following for large logs:

Re: [ANNOUNCE] New committer: Aizhamal Nurmamat kyzy

Congrats Aizhamal! On 30.06.20 17:34, Jan Lukavský wrote: Congratulations Aizhamal! On 6/30/20 1:35 PM, Alexey Romanenko wrote: Congratulations! Well deserved and thank you for your hard work, Aizhamal! On 30 Jun 2020, at 13:31, Reza Rokni > wrote: Congratulations !

Is there an easy way to figure out why my build failed?

Often I see the build failing, but on the next page there are no warnings and no errors. Then when you dive into the full log, it slows down the browser and there is no obvious ctrl-f keyword to find the error ("error" yields over 100 results, and the error isn't always at the bottom). Is there a

Re: [DISCUSS] ReadAll pattern and consistent use in IO connectors

Yeah, mainly I just feel like dynamic reads and dynamic writes (and perhaps not-yet-invented similar transforms of other kinds) are tightly related - they are either very similar, or are duals of each other - so they should use the same approach. If they are using different approaches, it is a sign

Re: [DISCUSS] ReadAll pattern and consistent use in IO connectors

Kenn, I'm not too worried about closures since: 1) the expansion service for a transform could have a well set of defined closures by name that are returned as serialized objects that don't need to be interpretable by the caller 2) the language could store serialized functions of another language a

Re: Composable DoFn IOs Connection Reuse

I'm not sure that runner determined sharding/GroupIntoBatches applies to splittable DoFns. Splittable DoFns is about taking one element and having a high fan-out / high cost function broken down to smaller pieces while runner determined sharding/GroupIntoBatches is about taking a lot of small eleme

Re: [ANNOUNCE] New committer: Aizhamal Nurmamat kyzy

Congratulations Aizhamal! On 6/30/20 1:35 PM, Alexey Romanenko wrote: Congratulations! Well deserved and thank you for your hard work, Aizhamal! On 30 Jun 2020, at 13:31, Reza Rokni > wrote: Congratulations ! On Tue, Jun 30, 2020 at 2:46 PM Michał Walenia mailto:mich

Re: [ANNOUNCE] New committer: Aizhamal Nurmamat kyzy

Congratulations! Well deserved and thank you for your hard work, Aizhamal! > On 30 Jun 2020, at 13:31, Reza Rokni wrote: > > Congratulations ! > > On Tue, Jun 30, 2020 at 2:46 PM Michał Walenia > wrote: > Congratulations, Aizhamal! :) > > On Tue, Jun 30, 202

Re: [ANNOUNCE] New committer: Aizhamal Nurmamat kyzy

Congratulations ! On Tue, Jun 30, 2020 at 2:46 PM Michał Walenia wrote: > Congratulations, Aizhamal! :) > > On Tue, Jun 30, 2020 at 8:41 AM Tobiasz Kędzierski < > tobiasz.kedzier...@polidea.com> wrote: > >> Congratulations Aizhamal! :) >> >> On Mon, Jun 29, 2020 at 11:50 PM Austin Bennett < >> w