Re: Testing streaming pipeline on Dataflow

2017-02-03 Thread Amit Sela
Note to self: stop emailing at 02:00am. Sorry for spamming ;-) . On Sat, Feb 4, 2017, 02:07 Amit Sela wrote: > Hi Davor, > > My team wants to try and test streaming pipelines on Dataflow (Beam > pipelines of course) and I was wondering how this works in terms of > UnboundedSources - Kafka/Pubsub

Testing streaming pipeline on Dataflow

2017-02-03 Thread Amit Sela
Hi Davor, My team wants to try and test streaming pipelines on Dataflow (Beam pipelines of course) and I was wondering how this works in terms of UnboundedSources - Kafka/Pubsub ? We currently use Kafka, and I was wondering if I could "record" a chunk of (public) data we use and import it ? can I

Re: Projects for Google Summer of Code 2017

2017-02-03 Thread Kenneth Knowles
In fact, I have just learned that our deadline to file project _is_ February 9th. Having good ideas is part of the ASF's application process. Here's a TL;DR of the instructions: 0. ASF members and committers can be mentors; find one for any project idea. 1. Mentors: understand what it means to be

Re: [PROPOSAL] New way of passing lambdas

2017-02-03 Thread Jesse Anderson
Excellent! On the redefinition of #2, people are used to it. With Hadoop MapReduce, you had to define types in 1-3 different places. While you're there, we also need also need a lambda that has access to the Context object. On Fri, Feb 3, 2017 at 11:03 AM Kenneth Knowles wrote: > Hi all, > > Ri

[PROPOSAL] New way of passing lambdas

2017-02-03 Thread Kenneth Knowles
Hi all, Right now when you want to use MapElements (and friends) you have two options: 1. Use a SimpleFunction Java 7 style MapElements.via(SimpleFunction() { @Override public B return apply(A input) { return ...expr...; } } and the type descriptors are automat