I'm excited to hear about making Samza capable of running non-Java pipelines.
On Fri, Oct 12, 2018 at 9:46 AM Xinyu Liu <xinyuliu...@gmail.com> wrote: > @Max: absolutely we should work together! FlinkRunner has been our best > reference since the start of our SamzaRunner, and the previous work in > Flink portable runner has been extremely valuable to us too. We haven't got > to the point of portable stateful processing yet. Our next step is to hook > up a streaming source, i.e. Kafka, and test out streaming capabilities such > as watermarks, windowing and triggers. For us, reading from Kafka will > happen on the Java side (we have quite a lot of extensions of Kafka in > LinkedIn), so we will try to create some internal Python API and do the > translation right now. On the other hand, we are following up with > BEAM-2937 for the optimization of Combine in portable runner. Previously we > run into some state problem without this being resolved. Anyway, look > forward to syncing up with you more! > > Thanks, > Xinyu > > > > On Fri, Oct 12, 2018 at 1:40 AM Maximilian Michels <m...@apache.org> wrote: > >> Thanks for the updating, Xinyu and Hai! Great to see another Running >> emerging :) >> >> I'm on the FlinkRunner. Looking forward to working together with you to >> make the Beam Runners even better. Particularly, we should sync on the >> portability, as some things are still to be fleshed out. In Flink, we >> are starting to integrate portable State. >> >> Best, >> Max >> >> On 11.10.18 05:14, Jesse Anderson wrote: >> > Interesting >> > >> > On Wed, Oct 10, 2018, 3:49 PM Kenneth Knowles <k...@apache.org >> > <mailto:k...@apache.org>> wrote: >> > >> > Welcome, Hai! >> > >> > On Wed, Oct 10, 2018 at 3:46 PM Hai Lu <lhai...@gmail.com >> > <mailto:lhai...@gmail.com>> wrote: >> > >> > Hi, all >> > >> > This is Hai from LinkedIn. As Xinyu mentioned, I have been >> > working on portable API for Samza runner and made some solid >> > progress. It's been a very smooth process (although not >> > effortless for sure) and I'm really grateful for the great >> > platform that you all have built. I'm very impressed. Bravo! >> > >> > Excited to work with everyone on Beam. Do expect more questions >> > from me down the road. >> > >> > Thanks, >> > Hai >> > >> > On Wed, Oct 10, 2018 at 12:36 PM Kenneth Knowles >> > <k...@apache.org <mailto:k...@apache.org>> wrote: >> > >> > Clarification: Thomas Groh wrote the fuser, not me! >> > >> > Thanks for the sharing all this. Really cool. >> > >> > Kenn >> > >> > On Wed, Oct 10, 2018 at 11:17 AM Rui Wang < >> ruw...@google.com >> > <mailto:ruw...@google.com>> wrote: >> > >> > Thanks for sharing! it's so exciting to hear that Beam >> > is being used on Samza in production @LinkedIn! Your >> > feedback will be helpful to Beam community! >> > >> > Besides, Beam supports SQL right now and hopefully Beam >> > community could also receive feedback on BeamSQL >> > < >> https://beam.apache.org/documentation/dsls/sql/overview/> in >> > the future. >> > >> > -Rui >> > >> > On Wed, Oct 10, 2018 at 11:10 AM Jean-Baptiste Onofré >> > <j...@nanthrax.net <mailto:j...@nanthrax.net>> wrote: >> > >> > Thanks for sharing and congrats for this great work >> ! >> > >> > Regards >> > JB >> > Le 10 oct. 2018, à 20:23, Xinyu Liu <xinyuliu.us >> > <mailto:%3Ca>@gmail.com <http://gmail.com> >> > target=_blank>xinyuliu.us >> > <http://xinyuliu.us>@gmail.com <http://gmail.com>> >> a >> > écrit: >> > >> > Hi, All, >> > >> > It's been over four months since we added the >> > Samza Runner to Beam, and we've been making a >> > lot of progress after that. Here I would like to >> > update your guys and share some really good news >> > happening here at LinkedIn: >> > >> > 1) First Beam job in production @LInkedIn! >> > After a few rounds of testing and benchmarking, >> > we finally rolled out our first Beam job here! >> > The job uses quite a few features, such as event >> > time, fixed/session windowing, early triggering, >> > and stateful processing. Our first customer is >> > very happy and they highly appraise the >> > easy-to-use Beam API as well as powerful >> > processing model. Due to the limited resources >> > here, we put our full trust in the work you guys >> > are doing, and we didn't run into any surprises. >> > We see extremely attention to details as well as >> > non-compromise in any user experience everywhere >> > in the code base. We would like to thank >> > everyone in the Beam community to contribute to >> > such an amazing framework! >> > >> > 2) A portable Samza Runner prototype >> > We are also starting the work in making Samza >> > Runner portable. So far we just got the python >> > word count example working using portable Samza >> > Runner. Please look out for the PR for this very >> > soon :). Again, this work is not possible >> > without the great Beam portability framework, >> > and the developers like Luke and Ahmet, just to >> > name a few, behind it. The ReferenceRunner has >> > been extremely useful to us to figure out what's >> > needed and how it works. Kudos to Thomas Groh, >> > Ben Sidhom and all the others who makes this >> > available to us. And to Kenn, your fuse work >> rocks. >> > >> > 3) More contributors in Samza Runner >> > The runner has been Chris and my personal >> > project for a while and now it's not the case. >> > We got Hai Lu and Boris Shkolnik from Samza team >> > to contribute. Hai has been focusing on the >> > portability work as mentioned in #2, and Boris >> > will work mostly on supporting our use cases. We >> > will send more emails discussing our use cases, >> > like the "Update state after firing" email I >> > sent out earlier. >> > >> > Finally, a shout-out to our very own Chris >> > Pettitt. Without you, none of the above won't >> > happen! >> > >> > Thanks, >> > Xinyu >> > >> >
