Re: [DISCUSS] Moving FakeBigQueryServices to main/ rather than test/

2019-07-31 Thread Kenneth Knowles
Publishing the "-tests" jar does not work for this purpose. The "test" classifier means "these are tests". The classifier does not mean "this is test related stuff". This is because "test" scope does not have transitive dependencies resolved in the same way and does not work for shipping a library

FYI - vacation until Monday, August 12

2019-07-31 Thread Kenneth Knowles
Hi all, Just so no one is waiting on code review or votes from me. I will be almost totally offline from now until Monday, August 12. Kenn

[RESULT] [VOTE] Release 2.14.0, release candidate #1

2019-07-31 Thread Anton Kedin
I'm happy to announce that we have unanimously approved this release. There are 7 approving votes, 4 of which are binding (in order): * Ahmet (al...@google.com); * Robert (rober...@google.com); * Pablo (pabl...@google.com); * Ismaël (ieme...@gmail.com); There are no disapproving votes. Thanks ev

Re: [Update] Beam 2.15 Release Progress

2019-07-31 Thread Yifan Zou
Snapshots are published http://repository.apache.org/content/groups/snapshots/org/apache/beam/. On Wed, Jul 31, 2019 at 1:28 PM Yifan Zou wrote: > Hi, > > The release branch is cut > https://github.com/apache/beam/tree/release-2.15.0. > The next step would be building snapshots and verify releas

Re: [VOTE] Release 2.14.0, release candidate #1

2019-07-31 Thread Chamikara Jayalath
+1 for continuing with RC1 with a notice about this issue. I marked this as a blocker for 2.15.0. Thanks, Cham On Wed, Jul 31, 2019 at 3:28 PM Anton Kedin wrote: > I am planning to close the vote tonight and this is the only potential > blocker discovered so far. So I strongly prefer to finaliz

Re: [VOTE] Release 2.14.0, release candidate #1

2019-07-31 Thread Anton Kedin
I am planning to close the vote tonight and this is the only potential blocker discovered so far. So I strongly prefer to finalize the current candidate to not delay the release further, since, as Ahmet mentioned, this is a new functionality and not a regression, and is experimental. However, given

Re: [ANNOUNCE] New committer: Jan Lukavský

2019-07-31 Thread Rui Wang
Congratulations! -Rui On Wed, Jul 31, 2019 at 10:51 AM Robin Qiu wrote: > Congrats! > > On Wed, Jul 31, 2019 at 10:31 AM Aizhamal Nurmamat kyzy < > aizha...@apache.org> wrote: > >> Congratulations, Jan! Thank you for your contributions! >> >> On Wed, Jul 31, 2019 at 10:04 AM Tanay Tummalapalli

Re: [VOTE] Release 2.14.0, release candidate #1

2019-07-31 Thread Eugene Kirpichov
I would recommend that the known issue notice about this source at least be strongly worded - this source in the current state should be marked "DO NOT USE" - it will produce data loss in *most* production use cases. That still leaves the risk that people will use it anyway; up to folks driving the

Re: [VOTE] Release 2.14.0, release candidate #1

2019-07-31 Thread Ahmet Altay
Since the python mongodb source is new in this release (not a regression) and experimental, I agree with adding a known issues notice to the release notes instead of starting a RC2 only for this issue. On Wed, Jul 31, 2019 at 2:47 PM Chamikara Jayalath wrote: > FYI we found a critical issue with

[PROPOSAL] An initial Schema API in Python

2019-07-31 Thread Brian Hulette
tl;dr: I have a PR at [1] that defines an initial Schema API in python based on the typing module, and uses typing.NamedTuple to represent a Schema. There are some risks with that approach but I propose we move forward with it as a first draft and iterate. I've opened up a PR [1] that implements

Re: [VOTE] Release 2.14.0, release candidate #1

2019-07-31 Thread Chamikara Jayalath
FYI we found a critical issue with the Python MongoDB source that is included with this release: https://issues.apache.org/jira/browse/BEAM-7866 I suggest we include a clear notice in the release about this issue if the release vote has already been finalized or make this a blocker if we are going

[Update] Beam 2.15 Release Progress

2019-07-31 Thread Yifan Zou
Hi, The release branch is cut https://github.com/apache/beam/tree/release-2.15.0 . The next step would be building snapshots and verify release branch. Regards. Yifan

Re: [ANNOUNCE] New committer: Jan Lukavský

2019-07-31 Thread Robin Qiu
Congrats! On Wed, Jul 31, 2019 at 10:31 AM Aizhamal Nurmamat kyzy wrote: > Congratulations, Jan! Thank you for your contributions! > > On Wed, Jul 31, 2019 at 10:04 AM Tanay Tummalapalli > wrote: > >> Congratulations! >> >> On Wed, Jul 31, 2019 at 10:05 PM Ahmet Altay wrote: >> >>> Congratulat

Re: [ANNOUNCE] New committer: Jan Lukavský

2019-07-31 Thread Aizhamal Nurmamat kyzy
Congratulations, Jan! Thank you for your contributions! On Wed, Jul 31, 2019 at 10:04 AM Tanay Tummalapalli wrote: > Congratulations! > > On Wed, Jul 31, 2019 at 10:05 PM Ahmet Altay wrote: > >> Congratulations Jan! Thank you for your contributions! >> >> On Wed, Jul 31, 2019 at 2:30 AM Ankur G

Re: [ANNOUNCE] New committer: Jan Lukavský

2019-07-31 Thread Tanay Tummalapalli
Congratulations! On Wed, Jul 31, 2019 at 10:05 PM Ahmet Altay wrote: > Congratulations Jan! Thank you for your contributions! > > On Wed, Jul 31, 2019 at 2:30 AM Ankur Goenka wrote: > >> Congratulations Jan! >> >> On Wed, Jul 31, 2019, 1:23 AM David Morávek wrote: >> >>> Congratulations Jan, w

Re: [ANNOUNCE] New committer: Jan Lukavský

2019-07-31 Thread Ahmet Altay
Congratulations Jan! Thank you for your contributions! On Wed, Jul 31, 2019 at 2:30 AM Ankur Goenka wrote: > Congratulations Jan! > > On Wed, Jul 31, 2019, 1:23 AM David Morávek wrote: > >> Congratulations Jan, well deserved! ;) >> >> D. >> >> On Wed, Jul 31, 2019 at 10:17 AM Ryan Skraba wrote

Re: [DISCUSS] Turn `WindowedValue` into `T` in the FnDataService and BeamFnDataClient interface definition

2019-07-31 Thread Maximilian Michels
Hi Jincheng, Thanks for getting back to us. > For the next major release of Flink, we plan to add Python user defined > functions(UDF, UDTF, UDAF) support in Flink and I have go over the Beam > portability framework and think that it is perfect for our requirements. > However we also find some

Re: [DISCUSS] Integer coders used in SchemaCoder

2019-07-31 Thread Robert Bradshaw
The standard VARINT coder is used for all sorts of integer values (e.g. the output of the CountElements transform), but the vast majority of them are likely significantly less than a full 64 bits. In Python, declaring an element type to be int will use this. On the other hand, using a VarInt format

Re: [BEAM-7819] -python - parsing message_id from PubSub message to the PubSubMessage attributes property

2019-07-31 Thread Matthew Darwin
Hi Ahmet/Udi, There are a couple of additional tests that failed following my change; apache_beam.io.gcp.tests.pubsub_match_test.PubSubMatcherTest.test_message_matcher_strip_success and apache_beam.io.gcp.tests.pubsub_match_test.PubSubMatcherTest.test_message_matcher_attributes_success, as the

Re: [DISCUSS] Turn `WindowedValue` into `T` in the FnDataService and BeamFnDataClient interface definition

2019-07-31 Thread Robert Bradshaw
Yep, Python support under active development, e.g. https://github.com/apache/beam/pull/9188 On Wed, Jul 31, 2019 at 9:24 AM jincheng sun wrote: > Thanks a lot for sharing the link. I take a quick look at the design and > the implementation in Java and think it could address my concern. It seems

Re: [VOTE] Release 2.14.0, release candidate #1

2019-07-31 Thread Robert Bradshaw
On Wed, Jul 31, 2019 at 11:22 AM Valentyn Tymofieiev wrote: > I have checked Portable Wordcount example on Flink and Spark on Python 2 > and Python 3. > > To do so, I had to checkout Beam from git repo, since using the source > distribution does not include gradlew, and gradelw_orig did not work

Re: [ANNOUNCE] New committer: Jan Lukavský

2019-07-31 Thread Ankur Goenka
Congratulations Jan! On Wed, Jul 31, 2019, 1:23 AM David Morávek wrote: > Congratulations Jan, well deserved! ;) > > D. > > On Wed, Jul 31, 2019 at 10:17 AM Ryan Skraba wrote: > >> Congratulations Jan! >> >> On Wed, Jul 31, 2019 at 10:10 AM Ismaël Mejía wrote: >> > >> > Hi, >> > >> > Please jo

Re: [VOTE] Release 2.14.0, release candidate #1

2019-07-31 Thread Valentyn Tymofieiev
I have checked Portable Wordcount example on Flink and Spark on Python 2 and Python 3. To do so, I had to checkout Beam from git repo, since using the source distribution does not include gradlew, and gradelw_orig did not work for me. Commands I ran: git checkout tags/v2.14.0-RC1 ./gradlew :sdks:

Re: [VOTE] Release 2.14.0, release candidate #1

2019-07-31 Thread Ismaël Mejía
Oups Robert pointed to me that I have probably not counted correctly. There were indeed already 3 PMC +1 votes. Pablo, Robert and Ahmet. Please excuse me for the extra noise. On Wed, Jul 31, 2019 at 9:46 AM Ismaël Mejía wrote: > > To complete the release we need to have at least three +1 binding

Re: [ANNOUNCE] New committer: Jan Lukavský

2019-07-31 Thread David Morávek
Congratulations Jan, well deserved! ;) D. On Wed, Jul 31, 2019 at 10:17 AM Ryan Skraba wrote: > Congratulations Jan! > > On Wed, Jul 31, 2019 at 10:10 AM Ismaël Mejía wrote: > > > > Hi, > > > > Please join me and the rest of the Beam PMC in welcoming a new > > committer: Jan Lukavský. > > > >

Re: [ANNOUNCE] New committer: Jan Lukavský

2019-07-31 Thread Ryan Skraba
Congratulations Jan! On Wed, Jul 31, 2019 at 10:10 AM Ismaël Mejía wrote: > > Hi, > > Please join me and the rest of the Beam PMC in welcoming a new > committer: Jan Lukavský. > > Jan has been contributing to Beam for a while, he was part of the team > that contributed the Euphoria DSL extension,

Re: [DISCUSS] Moving FakeBigQueryServices to main/ rather than test/

2019-07-31 Thread Ryan Skraba
Hello! No objection to the move :/ But what do you think about publishing the test jar created in google-cloud-platform to be reused without moving the code to the main artifact jar? I admit that I'm familiar with this technique with maven, and not at all with gradle, but it's described here: ht

[ANNOUNCE] New committer: Jan Lukavský

2019-07-31 Thread Ismaël Mejía
Hi, Please join me and the rest of the Beam PMC in welcoming a new committer: Jan Lukavský. Jan has been contributing to Beam for a while, he was part of the team that contributed the Euphoria DSL extension, and he has done interesting improvements for the Spark and Direct runner. He has also bee

Re: [VOTE] Release 2.14.0, release candidate #1

2019-07-31 Thread Ismaël Mejía
To complete the release we need to have at least three +1 binding votes (votes from PMC members) as stated in [1]. So far we have only 2. Thomas (and the others). The blog post PR is now open [2] please help us add missing features or maybe to highlight the ones you consider important in the PR co

Re: [DISCUSS] Turn `WindowedValue` into `T` in the FnDataService and BeamFnDataClient interface definition

2019-07-31 Thread jincheng sun
Thanks a lot for sharing the link. I take a quick look at the design and the implementation in Java and think it could address my concern. It seems that it's still not supported in the Python SDK Harness. Is there any plan on that? Robert Bradshaw 于2019年7月30日周二 下午12:33写道: > On Tue, Jul 30, 2019