Yi, That's a good amount of history to know. I will take a look at 680 and then see if I can implement something as well. If there's some stuff that's already done, would be glad to re-use it too. Thanks again
On Wed, Aug 31, 2016 at 10:58 PM, Yi Pan <nickpa...@gmail.com> wrote: > Hi, Sriram, > > The story behind delaying the integration of SAMZA-375 is that there are > tons of repeated code in SamzaAppMaster that exist in both samza-yarn and > Mesos. W/o the change we recently made in SAMZA-680, we are going to copy > the SamzaAppMaster code for every distributed execution system that we > added support in Samza. Now, w/ the change in SAMZA-680, we have inverted > the JobCoordinator and the AppMaster logic, which makes it much easier to > have pluggable distributed cluster management system in Samza. As stated in > the JIRA, all we need is now a Mesos-specific implementation of > ClusterResourceManager that can talk to Mesos for container > request/allocation. > > @Jagadish, I remember that you did some proto-type integration w/ Mesos > based on SAMZA-680. Would you mind to share some example code for that? > > Thanks! > > -Yi > > On Tue, Aug 30, 2016 at 8:35 PM, Sriram Ramachandrasekaran < > sri.ram...@gmail.com> wrote: > > > Folks, > > > > We've been using Samza in Production from beginning of this year. It's > been > > quite stable for our needs, although, we don't use it heavily yet. One of > > the things we would like to know is, where is Samza Mesos integration in > > the roadmap? I know, SAMZA-375 > > <https://issues.apache.org/jira/browse/SAMZA-375> is specifically > towards > > that, but, is there something stopping the community from integration > into > > mainline? > > > > I ask this because, we run our Samza jobs on YARN right now and we use > > Mesos infra for other workloads. I really don't want to manage 2 infra > > components which are supposed to do exactly the same thing. We've built > > enough tooling around Mesos infra, so, wouldn't want to move away from it > > too. > > > > The options we're evaluating are: > > 1. Move to KStreams and get away from YARN > > 2. Explore Samza-Mesos integration so that, we can reduce "explicit" > > dependency on Kafka. > > > > > > Some clarity on this would really help us. > > Sriram > > > > -- > > It's just about how deep your longing is! > > > -- It's just about how deep your longing is!
