Re: real real-time beam

2019-12-03 Thread Kenneth Knowles
Jan - let's try to defrag the threads on your time sorting proposal. This thread may have useful ideas but I want to focus on helping Aaron in this thread. You can link to this thread from other threads or from a design doc. Does this seem OK to you? Aaron - do you have the information you need to

Re: Beam Testing Tools FAQ

2019-12-03 Thread Kenneth Knowles
Just still reading through this. It is very helpful. Thanks! On Tue, Dec 3, 2019 at 6:32 AM Reza Rokni wrote: > Thanx! > > On Wed, 27 Nov 2019, 02:31 Pablo Estrada, wrote: > >> Very cool. Thanks Lukasz! >> >> On Tue, Nov 26, 2019 at 9:41 AM Alan Myrvold wrote: >> >>> Nice, thanks! >>> >>> On T

Version Beam Website Documentation

2019-12-03 Thread Ankur Goenka
Hi, We are constantly adding features to Beam which makes each new Beam version more feature rich and compelling. This also means that the old Beam released don't have the new features and might have different ways to do certain things. (I might be wrong here) - Our Beam website only publish a si

Re: Jenkins jobs are not being displayed on GitHub

2019-12-03 Thread Yifan Zou
Glanced at the build workers. All jenkins executors are busy now. Lots of jobs are in the waiting queue. And the recent completed jobs show a long waiting time to get started, which is around 1h. The Jenkins site also responses very slow. On Tue, Dec 3, 2019 at 2:15 PM Kirill Kozlov wrote: > Hel

Jenkins jobs are not being displayed on GitHub

2019-12-03 Thread Kirill Kozlov
Hello everyone! It looks like for the PRs created within the last 30 minutes status of Jenkins jobs is not being displayed. Seed job appears to be stuck [1]. #5293 is waiting for #5292 to finish, but #5293 shows that it is complete. [1] https://builds.apache.org/job/beam_SeedJob/ -- Kirill

Re: Request for review of PR [Beam-8564]

2019-12-03 Thread Robert Bradshaw
Is there a way to wrap this up as an optional dependency with multiple possible providers, if there's no good library satisfying all of the conditions (in particular (1))? On Tue, Dec 3, 2019 at 9:47 AM Luke Cwik wrote: > > I was hoping that someone in the community would provide some alternative

Re: Request for review of PR [Beam-8564]

2019-12-03 Thread Luke Cwik
I was hoping that someone in the community would provide some alternatives since there are quite a few implementations. On Tue, Dec 3, 2019 at 8:20 AM Amogh Tiwari wrote: > Hi Luke, > > I agree with your thoughts and observations. But, airlift:aircompressor is > the only implementation of LZO in

Re: Per Element File Output Without writeDynamic

2019-12-03 Thread Eugene Kirpichov
Hi Christopher, Thanks for clarifying. Then can you just preprocess the PCollection with a custom FlatMapElements that converts each Document into one or more smaller documents, small enough to be written into individual files? Then pair it with a unique key and follow by FileIO.writeDynamic().by(

Re: Request for review of PR [Beam-8564]

2019-12-03 Thread Amogh Tiwari
Hi Luke, I agree with your thoughts and observations. But, airlift:aircompressor is the only implementation of LZO in pure java. That straight away solves #5. The other implementations that I found either have licensing issues (since LZO natively uses GNU GPL licence) or are implemented using .c,

Re: Per Element File Output Without writeDynamic

2019-12-03 Thread Christopher Larsen
Hi Eugene, Yes I think you've got it correct. In our use case we need to write each Document in the PCollection to a separate file as multiple Documents in a file will cause compilation errors and/or incorrect code to be generated by the Thrift compiler. Additionally there are some Documents that

Re: Beam Testing Tools FAQ

2019-12-03 Thread Reza Rokni
Thanx! On Wed, 27 Nov 2019, 02:31 Pablo Estrada, wrote: > Very cool. Thanks Lukasz! > > On Tue, Nov 26, 2019 at 9:41 AM Alan Myrvold wrote: > >> Nice, thanks! >> >> On Tue, Nov 26, 2019 at 8:04 AM Robert Bradshaw >> wrote: >> >>> Thanks! >>> >>> On Tue, Nov 26, 2019 at 7:43 AM Ɓukasz Gajowy >