Re: The trouble with DisplayData tests

2017-11-01 Thread Kenneth Knowles
Hi Pablo, On Wed, Nov 1, 2017 at 6:22 PM, Pablo Estrada wrote: > > > > As of today, the Java and Python Dataflow runners actually transmit the > > entire original pipeline, which has slots for display data on composites. > > So the future in which this is actually used - at least for Dataflow -

Fwd: Jenkins instability and restarts

2017-11-01 Thread Kenneth Knowles
-- Forwarded message -- From: Daniel Pono Takamori Date: Wed, Nov 1, 2017 at 8:36 PM Subject: Jenkins instability and restarts To: bui...@apache.org Hello builds@ enthusiasts. As I'm sure you are aware, we're encountering a rather nasty bug affecting a small subset of jobs but i

Re: The trouble with DisplayData tests

2017-11-01 Thread Pablo Estrada
> > As of today, the Java and Python Dataflow runners actually transmit the > entire original pipeline, which has slots for display data on composites. > So the future in which this is actually used - at least for Dataflow - > might be very near. > I am not 100% sure, but I believe this is not tru

Re: Hello

2017-11-01 Thread Ankur Goenka
Thanks for adding me to the group. On Wed, Nov 1, 2017 at 12:04 PM, Lukasz Cwik wrote: > Welcome, you have been added. > > On Tue, Oct 31, 2017 at 5:31 PM, Ankur Goenka > wrote: > > > Hi Guys, > > > > I recently joined Google and will be contributing to Apache Beam project. > > Please add me to

Re: Hello

2017-11-01 Thread Ahmet Altay
Welcome Ankur! On Wed, Nov 1, 2017 at 12:04 PM, Lukasz Cwik wrote: > Welcome, you have been added. > > On Tue, Oct 31, 2017 at 5:31 PM, Ankur Goenka > wrote: > > > Hi Guys, > > > > I recently joined Google and will be contributing to Apache Beam project. > > Please add me to the apache jira. My

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-01 Thread Kenneth Knowles
I have started one, here: https://github.com/kennknowles/beam/commits/bazel. It is not nearly as far along as Luke's. For the POC I am just putting things in one root BUILD, and learning where we might find the necessary plugins as I go. I am happy to grant push access to this branch. It would be s

Re: The trouble with DisplayData tests

2017-11-01 Thread Lukasz Cwik
I think we should keep the tests around and encourage people to use it. With the work that Ken put in to get the Pipeline proto to passed to the Dataflow workers means that these composites will be flowing through Dataflow to the execution side. I could see that this will eventually show up in the

Re: Hello

2017-11-01 Thread Lukasz Cwik
Welcome, you have been added. On Tue, Oct 31, 2017 at 5:31 PM, Ankur Goenka wrote: > Hi Guys, > > I recently joined Google and will be contributing to Apache Beam project. > Please add me to the apache jira. My jira id is angoenka > > Looking forward to work with you guys! > > Thanks, > Ankur >

Combine.perKey and late data

2017-11-01 Thread Pawel Bartoszek
My simplified scenario is that I want to count how many A and B events I am receiving each minute. I also need to support late data. Let's assume I am receiving 2 A and 2 B events in the same window (00:00:00-00:00:001) So the expected result is two A events , and two B events Then I receive 2 la

Re: Portability overview webpage

2017-11-01 Thread Jean-Baptiste Onofré
Thanks for the update. I will take a look. Regards JB On Nov 1, 2017, 18:21, at 18:21, Henning Rohde wrote: >Hi everyone, > >Although portability is a large and involved effort, it seems it >doesn't >have a high-level overview and plan written down anywhere. I added a >proposed page with a 10,0

Portability overview webpage

2017-11-01 Thread Henning Rohde
Hi everyone, Although portability is a large and involved effort, it seems it doesn't have a high-level overview and plan written down anywhere. I added a proposed page with a 10,000 ft view and links to the webside under 'Contribute (technical references)'. There is a page for ongoing projects, b

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-01 Thread Ahmet Altay
Has anyone started a POC with Bazel? I would be interested in helping that effort. On Wed, Nov 1, 2017 at 9:27 AM, Lukasz Cwik wrote: > I have started a POC for using Gradle here: > https://github.com/lukecwik/incubator-beam/tree/gradle > > Things that work: > * compiling all Java code (src/main

Re: [VOTE] Switch to new JIRA workflow for pending proposals

2017-11-01 Thread Chamikara Jayalath
+1 Thanks, Cham On Wed, Nov 1, 2017 at 9:42 AM Ted Yu wrote: > +1 > Original message From: Henning Rohde > Date: 11/1/17 9:29 AM (GMT-08:00) To: > dev@beam.apache.org Subject: Re: [VOTE] Switch to new JIRA workflow for > pending proposals > + 1. Also +1 to Kenn's suggestion

Re: [VOTE] Switch to new JIRA workflow for pending proposals

2017-11-01 Thread Ted Yu
+1 Original message From: Henning Rohde Date: 11/1/17 9:29 AM (GMT-08:00) To: dev@beam.apache.org Subject: Re: [VOTE] Switch to new JIRA workflow for pending proposals + 1. Also +1 to Kenn's suggestion of adding a "triage" stage prior to "open" for other bugs and features.

Re: [VOTE] Switch to new JIRA workflow for pending proposals

2017-11-01 Thread Henning Rohde
+ 1. Also +1 to Kenn's suggestion of adding a "triage" stage prior to "open" for other bugs and features. On Wed, Nov 1, 2017 at 9:19 AM, Lukasz Cwik wrote: > +1 > > On Wed, Nov 1, 2017 at 5:52 AM, Kenneth Knowles > wrote: > > > +1 I like it. > > > > This is the JIRA alternative to FLIP/KIP/HIP

Re: [DISCUSS] Move away from Apache Maven as build tool

2017-11-01 Thread Lukasz Cwik
I have started a POC for using Gradle here: https://github.com/lukecwik/incubator-beam/tree/gradle Things that work: * compiling all Java code (src/main and src/test) * generating source from protos * generating source from avro * running rat, checkstyle Partially working: * generating maven pom

Re: [VOTE] Switch to new JIRA workflow for pending proposals

2017-11-01 Thread Lukasz Cwik
+1 On Wed, Nov 1, 2017 at 5:52 AM, Kenneth Knowles wrote: > +1 I like it. > > This is the JIRA alternative to FLIP/KIP/HIP etc, yes? I definitely favor > having automation around that rather than just wiki or web page entries. > > Any thoughts on how it interacts with other types of tasks? > > A

Re: [VOTE] Switch to new JIRA workflow for pending proposals

2017-11-01 Thread Kenneth Knowles
+1 I like it. This is the JIRA alternative to FLIP/KIP/HIP etc, yes? I definitely favor having automation around that rather than just wiki or web page entries. Any thoughts on how it interacts with other types of tasks? Also, an analogous initial state seems useful for all bugs & features. Name