Re: Providing default values for ValueProvider

2018-12-20 Thread Kenneth Knowles
It sounds like it may simply be an oversight that @Default does not work with ValueProvider. Copying dev@ to see if there's some pitfall associated with adding this. Kenn On Thu, Dec 20, 2018 at 5:17 PM Jeff Klukas wrote: > I am also now realizing that runtime params show up in --help with type

Re: Recordings and presentations from Beam Summit London 2018

2018-12-20 Thread Kenneth Knowles
Thanks! Really happy I can watch these since I was unable to attend in person. Kenn On Thu, Dec 20, 2018 at 6:49 PM Manu Zhang wrote: > Thanks for sharing. The YouTube channel link is > https://www.youtube.com/channel/UChNnb_YO_7B0HlW6FhAXZZQ > > Thanks, > Manu Zhang > On Nov 2, 2018, 11:06 PM

Re: Recordings and presentations from Beam Summit London 2018

2018-12-20 Thread Manu Zhang
Thanks for sharing. The YouTube channel link is  https://www.youtube.com/channel/UChNnb_YO_7B0HlW6FhAXZZQ Thanks, Manu Zhang On Nov 2, 2018, 11:06 PM +0800, Matthias Baetens , wrote: > Hi everyone, > > Very happy to be sharing the great materials from the speakers and > contributors at the Beam

Re: [VOTE] Release Vendored gRPC 1.13.1 v0.2, release candidate #1

2018-12-20 Thread Tyler Akidau
+1, Approve the release. -Tyler On Thu, Dec 20, 2018 at 9:49 AM Ahmet Altay wrote: > I meant BEAM-6249 in my last sentence. It should read: "BEAM-6249 has a > comment about user building the libraries themselves, I am not sure if they > are using the release 2.9 version directly or not." > > On

Re: [VOTE] Release Vendored gRPC 1.13.1 v0.2, release candidate #1

2018-12-20 Thread Ahmet Altay
I meant BEAM-6249 in my last sentence. It should read: "BEAM-6249 has a comment about user building the libraries themselves, I am not sure if they are using the release 2.9 version directly or not." On Thu, Dec 20, 2018 at 9:48 AM Ahmet Altay wrote: > +1 > > I don't think there is a need for a

Re: [VOTE] Release Vendored gRPC 1.13.1 v0.2, release candidate #1

2018-12-20 Thread Ahmet Altay
+1 I don't think there is a need for a hotfix release. The reason the initial vendoring PR (https://github.com/apache/beam/pull/7024) that started the issue was not cherry picked to the release branch. BEAM-6056 has a comment about user building the libraries themselves, I am not sure if they are

Re: [VOTE] Release Vendored gRPC 1.13.1 v0.2, release candidate #1

2018-12-20 Thread Kenneth Knowles
I don't know yet about 2.9.1. There's a bit more context on BEAM-6249. Kenn [1] https://issues.apache.org/jira/browse/BEAM-6249 On Thu, Dec 20, 2018 at 12:02 PM Scott Wegner wrote: > Releasing new vendored artifacts won't generally imply a full Beam > release. The plan is to pick up the new ar

Re: [VOTE] Release Vendored gRPC 1.13.1 v0.2, release candidate #1

2018-12-20 Thread Scott Wegner
Releasing new vendored artifacts won't generally imply a full Beam release. The plan is to pick up the new artifact version at HEAD which will roll into the next release. For this particularly case, the question is if the Dataflow issue that this fixes (BEAM-6056) warrants a hotfix release (2.9.1)

Re: (fyi) PR#7324 updates how projects consume vendored guava

2018-12-20 Thread Kenneth Knowles
On Thu, Dec 20, 2018 at 5:29 AM Ismaël Mejía wrote: > I may be misreading or probably missed part of the previous > discussion, but why don't we have the vendoring artifacts in a > different repo? > This will do the release really independent and avoid the artificial > staging step, no? > Other

Re: excessive java precommit logging

2018-12-20 Thread Kenneth Knowles
I support lowering the log level. The default is `lifecycle`. For reference, here's where it was increased to `info`: https://github.com/apache/beam/pull/4644 The reason was to see more details about Gradle's dependency management. We were seeing dependency download flakes on things that should n

Re: excessive java precommit logging

2018-12-20 Thread Robert Bradshaw
Interestingly, I was thinking exactly the same thing the other day. If we could drop the info logs for passing tests, that would be ideal. Regardless, tests should fail (when possible) with actionable messages. I think the rare case of not being able to reproduce the error locally if info logs are

Re: Spark-optimized Shuffle (SOS) any update?

2018-12-20 Thread David Morávek
I've just quickly went trough the design doc and it seems that it has nothing to do with the paper Marek has mentioned. The paper is trying to solving the problem of io ops required for shuffle are growing quadratically with number of tasks (shuffle files), therefore we need to keep number of tasks

Re: Spark-optimized Shuffle (SOS) any update?

2018-12-20 Thread David Morávek
This is an awesome news! Is there anything we can do to help? We are currently facing huge performance penalties due this issue. Thanks, David On Wed, Dec 19, 2018 at 5:43 PM Ilan Filonenko wrote: > Recently, the community has actively been working on this. The JIRA to > follow is: > https://is

Re: (fyi) PR#7324 updates how projects consume vendored guava

2018-12-20 Thread Ismaël Mejía
I may be misreading or probably missed part of the previous discussion, but why don't we have the vendoring artifacts in a different repo? This will do the release really independent and avoid the artificial staging step, no? Thomas what do you mean with: > It is unfortunate that we have to change

Re: excessive java precommit logging

2018-12-20 Thread Maximilian Michels
Thanks Udi for bringing this up. I'm also for dropping INFO. It's just incredible verbose. More importantly, from my experience the INFO level doesn't help debugging problems, but it makes finding errors messages or warnings harder. That said, here's what I do to search through the log: 1) cur

Re: [RFC] I made a new tabbed Beam view in Jenkins

2018-12-20 Thread Łukasz Gajowy
Looks great. +1 for making this a top level view. > After that, maybe we could clean the categories so they fit into the tabs > more easily with fewer regexes (to make sure things don't get missed). I > have read also that if you use / instead of _ as a separator in a name then > Jenkins will dis

Re: [VOTE] Release Vendored gRPC 1.13.1 v0.2, release candidate #1

2018-12-20 Thread Ismaël Mejía
Does this imply that we need a subsequent full release afterwards? I am assuming this new release is related to the reported issues with the dataflow worker or is this something different? On Thu, Dec 20, 2018 at 2:51 AM Kenneth Knowles wrote: > > +1 > > - sigs good > - `jar tf` looks good > >

Re: [Fwd: [Apache Beam] Custom DataSourceV2 instanciation: parameters passing and Encoders]

2018-12-20 Thread Etienne Chauchot
@Manu, anyway spark DataSource (V1) instanciation mechanism is the same than DataSource V2, so question remains even if we target v1 for stability reasons. Etienne Le mercredi 19 décembre 2018 à 17:07 +0100, Etienne Chauchot a écrit : > Yes, this is thanks to these spark community meetings that I

Re: Beam Summits!

2018-12-20 Thread Matthias Baetens
Great stuff, thanks for the overview, Austin. For EU, there are things to say for both Stockholm and Berlin, but I think it makes sense to do it on the back of another conference (larger chance of people being in town with the same interest). I like Thomas comment - we will attract more people fro