Hi Sriram, I had started prototyping it (purely to ensure that the Samza API makes sense with Mesos). The exact API on the Samza-11 trunk is slightly different, but hopefully there're similarities - https://github.com/apache/samza/blob/master/samza-core/src/main/java/org/apache/samza/clustermanager/ClusterResourceManager.java
Find a stub implementation here: (that encapsulates a fair bit of boiler plate from Mesos driver creation etc.) https://github.com/vjagadish/samza-clone/commit/9e5ed9f1774dadf079ad33913ff7f20ed58bc8dc A version of the prototype with the Old API: here <https://github.com/bringhurst/samza/tree/SAMZA-375/samza-mesos/src/main/scala/org/apache/samza/job/mesos> Some interesting implementation notes: - Mesos did not (yet) support a preferred host request. However, that could be implemented via dynamic reservations <http://mesos.apache.org/documentation/latest/reservation/>. - My discussions with the Mesos community here: https://mail-archives.apache.org/mod_mbox/mesos-user/201602.mbox/%3ccamd3yjgxmhg4rtw4gbxgf9msmbv6abzbgpql6ejq5gwmot0...@mail.gmail.com%3E - MESOS-4616 has more context. It'd be awesome you can take a stab at Mesos integration - I'm happy to help out in whatever way I can. Thank you, Jagadish On Wed, Aug 31, 2016 at 10:45 AM, Sriram Ramachandrasekaran < sri.ram...@gmail.com> wrote: > Yi, > That's a good amount of history to know. I will take a look at 680 and then > see if I can implement something as well. If there's some stuff that's > already done, would be glad to re-use it too. > Thanks again > > On Wed, Aug 31, 2016 at 10:58 PM, Yi Pan <nickpa...@gmail.com> wrote: > > > Hi, Sriram, > > > > The story behind delaying the integration of SAMZA-375 is that there are > > tons of repeated code in SamzaAppMaster that exist in both samza-yarn and > > Mesos. W/o the change we recently made in SAMZA-680, we are going to copy > > the SamzaAppMaster code for every distributed execution system that we > > added support in Samza. Now, w/ the change in SAMZA-680, we have inverted > > the JobCoordinator and the AppMaster logic, which makes it much easier to > > have pluggable distributed cluster management system in Samza. As stated > in > > the JIRA, all we need is now a Mesos-specific implementation of > > ClusterResourceManager that can talk to Mesos for container > > request/allocation. > > > > @Jagadish, I remember that you did some proto-type integration w/ Mesos > > based on SAMZA-680. Would you mind to share some example code for that? > > > > Thanks! > > > > -Yi > > > > On Tue, Aug 30, 2016 at 8:35 PM, Sriram Ramachandrasekaran < > > sri.ram...@gmail.com> wrote: > > > > > Folks, > > > > > > We've been using Samza in Production from beginning of this year. It's > > been > > > quite stable for our needs, although, we don't use it heavily yet. One > of > > > the things we would like to know is, where is Samza Mesos integration > in > > > the roadmap? I know, SAMZA-375 > > > <https://issues.apache.org/jira/browse/SAMZA-375> is specifically > > towards > > > that, but, is there something stopping the community from integration > > into > > > mainline? > > > > > > I ask this because, we run our Samza jobs on YARN right now and we use > > > Mesos infra for other workloads. I really don't want to manage 2 infra > > > components which are supposed to do exactly the same thing. We've built > > > enough tooling around Mesos infra, so, wouldn't want to move away from > it > > > too. > > > > > > The options we're evaluating are: > > > 1. Move to KStreams and get away from YARN > > > 2. Explore Samza-Mesos integration so that, we can reduce "explicit" > > > dependency on Kafka. > > > > > > > > > Some clarity on this would really help us. > > > Sriram > > > > > > -- > > > It's just about how deep your longing is! > > > > > > > > > -- > It's just about how deep your longing is! > -- Jagadish V, Graduate Student, Department of Computer Science, Stanford University
