Re: [PROPOSAL] Merge samza-runner to master

2018-06-25 Thread Xinyu Liu
Thanks a lot, Kenn! Finally we linked our runner in. I will work on the rest of the stuff as you mentioned. Thanks again for everyone's comments, too. Thanks, Xinyu On Mon, Jun 25, 2018 at 2:48 PM, Kenneth Knowles wrote: > This is done. Now we need to make sure the build is running, healthy, in

Re: [PROPOSAL] Merge samza-runner to master

2018-06-25 Thread Kenneth Knowles
This is done. Now we need to make sure the build is running, healthy, in the PR template. On Mon, Jun 25, 2018 at 9:10 AM Kenneth Knowles wrote: > I'll do it. I'm working with Xinyu on the PR. > > Kenn > > On Mon, Jun 25, 2018, 08:24 Ismaël Mejía wrote: > >> +1 >> >> It is important to have new

Re: [PROPOSAL] Merge samza-runner to master

2018-06-25 Thread Kenneth Knowles
I'll do it. I'm working with Xinyu on the PR. Kenn On Mon, Jun 25, 2018, 08:24 Ismaël Mejía wrote: > +1 > > It is important to have new runners merged (even if not 100% complete) so > they benefit of the fixes going on, and that they can easily (and > incrementally) start to track the new porta

Re: [PROPOSAL] Merge samza-runner to master

2018-06-25 Thread Ismaël Mejía
+1 It is important to have new runners merged (even if not 100% complete) so they benefit of the fixes going on, and that they can easily (and incrementally) start to track the new portability features as they develop. What is next then ? Who triggers the green button so this happens? On Sat,

Re: [PROPOSAL] Merge samza-runner to master

2018-06-22 Thread Jean-Baptiste Onofré
+1 As the build is fine, it makes sense to merge pretty fast. Thanks, Regards JB On 22/06/2018 00:14, Xinyu Liu wrote: > I updated the merge PR with the gradle integration (there was some > Jenkins Java tests failure with google cloud quota issues. It seems not > related to this patch). Please f

Re: [PROPOSAL] Merge samza-runner to master

2018-06-22 Thread Lukasz Cwik
+1 for merging Samza. Portability is making great progress and it will be great to see Samza join in the effort to be able to run non Java pipelines. On Fri, Jun 22, 2018 at 1:53 PM Robert Bradshaw wrote: > Thanks for the clarification on contributors, that makes me much more > comfortable. I ag

Re: [PROPOSAL] Merge samza-runner to master

2018-06-22 Thread Robert Bradshaw
Thanks for the clarification on contributors, that makes me much more comfortable. I agree that portability isn't ready enough to require it, and am encouraged by the plans to focus on this next quarter. These are my only reservations, and I see lot of benefits of making samza an official runner.

Re: [PROPOSAL] Merge samza-runner to master

2018-06-22 Thread Xinyu Liu
A little clarification on the contributors: Chris Pettitt and I are the main contributors so far. Chris wrote the initial prototype but his commits got squashed into the giant initial commit, and he's been reviewing all incremental changes afterwards. Two more team members (Boris Shkolnik and Hai L

Re: [PROPOSAL] Merge samza-runner to master

2018-06-22 Thread Rafael Fernandez
​I think it's great to go ahead and merge it, so it can continue evolving. As with all things, it'll adopt new stuff as it becomes ready (in fact, it may even prove to be a great example of how to port an existing "legacy" runner to the portability stuff when ready).​ It seems the immediate blocke

Re: [PROPOSAL] Merge samza-runner to master

2018-06-21 Thread Kenneth Knowles
*Contributors* Agree with Robert's concern. But this is a nice opportunity for Beam to connect. It is a different sort of backend and a different sort of community that we are linking in. Consider the Gearpump and Apex runners: both had resumes that met the requirements, but might not today. But t

Re: [PROPOSAL] Merge samza-runner to master

2018-06-21 Thread Robert Bradshaw
Neat to see a new runner on board! I would like to make it a requirement for all new runners to support the portability API, but given that it's still somewhat of a moving target, and you have ongoing work in this direction, that may not be a hard requirement. I'm a bit concerned that there is ar

Re: [PROPOSAL] Merge samza-runner to master

2018-06-21 Thread Xinyu Liu
I updated the merge PR with the gradle integration (there was some Jenkins Java tests failure with google cloud quota issues. It seems not related to this patch). Please feel free to ping me if anything else is needed. Thanks, Xinyu On Mon, Jun 18, 2018 at 5:44 PM, Xinyu Liu wrote: > @Kenn: I a

Re: [PROPOSAL] Merge samza-runner to master

2018-06-18 Thread Xinyu Liu
@Kenn: I am going to add the build.gradle. Is there anything else? @Ahmet, @Robert: here are more details about the samza runner right now: - Missing pieces: timer support in ParDo is not there yet and I plan to add it soon. SplittableParDo is missing but we don't have a use case so far. We are o

Re: [PROPOSAL] Merge samza-runner to master

2018-06-18 Thread Robert Burke
This is exciting! Is it implemented as a portability framework runner too? On Mon, Jun 18, 2018, 4:36 PM Pablo Estrada wrote: > It's very exciting to see a new runner making it into master. : ) > > Best > -P. > > On Mon, Jun 18, 2018 at 3:38 PM Rafael Fernandez > wrote: > >> I've just read this

Re: [PROPOSAL] Merge samza-runner to master

2018-06-18 Thread Pablo Estrada
It's very exciting to see a new runner making it into master. : ) Best -P. On Mon, Jun 18, 2018 at 3:38 PM Rafael Fernandez wrote: > I've just read this and wanted to share my excitement :D > > > > On Mon, Jun 18, 2018 at 3:10 PM Kenneth Knowles wrote: > >> One thing that will be necessary is

Re: [PROPOSAL] Merge samza-runner to master

2018-06-18 Thread Ahmet Altay
Thank you for everyone who contributed to this runner. It is really great to see this. Xinyu, for the people like myself who were not following the development closely, could you talk about missing pieces, work in progress, future plans? On Mon, Jun 18, 2018 at 3:37 PM, Rafael Fernandez wrote:

Re: [PROPOSAL] Merge samza-runner to master

2018-06-18 Thread Rafael Fernandez
I've just read this and wanted to share my excitement :D On Mon, Jun 18, 2018 at 3:10 PM Kenneth Knowles wrote: > One thing that will be necessary is porting the build to Gradle. > > Kenn > > On Mon, Jun 18, 2018 at 11:57 AM Xinyu Liu wrote: > >> Hi, Folks, >> >> On behalf of the Samza team,

Re: [PROPOSAL] Merge samza-runner to master

2018-06-18 Thread Kenneth Knowles
One thing that will be necessary is porting the build to Gradle. Kenn On Mon, Jun 18, 2018 at 11:57 AM Xinyu Liu wrote: > Hi, Folks, > > On behalf of the Samza team, I would like to propose to merge the > samza-runner branch into master. The branch was created on Jan when we > first introduced

[PROPOSAL] Merge samza-runner to master

2018-06-18 Thread Xinyu Liu
Hi, Folks, On behalf of the Samza team, I would like to propose to merge the samza-runner branch into master. The branch was created on Jan when we first introduced the Samza Runner [1], and we've been adding features and refining it afterwards. Now the runner satisfies the criteria outlined in [2