Hi, Sriram, The story behind delaying the integration of SAMZA-375 is that there are tons of repeated code in SamzaAppMaster that exist in both samza-yarn and Mesos. W/o the change we recently made in SAMZA-680, we are going to copy the SamzaAppMaster code for every distributed execution system that we added support in Samza. Now, w/ the change in SAMZA-680, we have inverted the JobCoordinator and the AppMaster logic, which makes it much easier to have pluggable distributed cluster management system in Samza. As stated in the JIRA, all we need is now a Mesos-specific implementation of ClusterResourceManager that can talk to Mesos for container request/allocation.
@Jagadish, I remember that you did some proto-type integration w/ Mesos based on SAMZA-680. Would you mind to share some example code for that? Thanks! -Yi On Tue, Aug 30, 2016 at 8:35 PM, Sriram Ramachandrasekaran < sri.ram...@gmail.com> wrote: > Folks, > > We've been using Samza in Production from beginning of this year. It's been > quite stable for our needs, although, we don't use it heavily yet. One of > the things we would like to know is, where is Samza Mesos integration in > the roadmap? I know, SAMZA-375 > <https://issues.apache.org/jira/browse/SAMZA-375> is specifically towards > that, but, is there something stopping the community from integration into > mainline? > > I ask this because, we run our Samza jobs on YARN right now and we use > Mesos infra for other workloads. I really don't want to manage 2 infra > components which are supposed to do exactly the same thing. We've built > enough tooling around Mesos infra, so, wouldn't want to move away from it > too. > > The options we're evaluating are: > 1. Move to KStreams and get away from YARN > 2. Explore Samza-Mesos integration so that, we can reduce "explicit" > dependency on Kafka. > > > Some clarity on this would really help us. > Sriram > > -- > It's just about how deep your longing is! >
