Tracking this at https://issues.apache.org/jira/browse/AURORA-587

-=Bill


On Fri, Jul 11, 2014 at 2:24 PM, Joe Stein <joe.st...@stealth.ly> wrote:

> +1 for adding it to Aurora, looking forward to using it, thanks!!!!
>
> /*******************************************
>  Joe Stein
>  Founder, Principal Consultant
>  Big Data Open Source Security LLC
>  http://www.stealth.ly
>  Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
> ********************************************/
>
>
> On Wed, Jul 9, 2014 at 6:48 PM, Oliver, James <james.oli...@pegs.com>
> wrote:
>
> > *Nudge nudge*
> >
> > We would love to take advantage of this tool if you folks would be so
> kind!
> >
> > Thanks,
> > James
> >
> >
> > On 7/9/14 1:35 PM, "Bill Farner" <wfar...@apache.org> wrote:
> >
> > >FWIW at Twitter we do something that sounds like a mix between (1) and
> > >(2).
> > > We consume the (otherwise unused) Announcer configuration parameter [1]
> > >to
> > >Job, and extend the executor to publish all allocated
> {{thermos.port[x]}}s
> > >to ZooKeeper.  This is something we would love to open source, so please
> > >nudge us to do so if you would like this feature!
> > >
> > >[1]
> > >
> >
> https://github.com/apache/incubator-aurora/blob/master/src/main/python/apa
> > >che/aurora/config/schema/base.py#L69
> > >
> > >-=Bill
> > >
> > >
> > >On Wed, Jul 9, 2014 at 10:24 AM, Oliver, James <james.oli...@pegs.com>
> > >wrote:
> > >
> > >> Good morning,
> > >>
> > >> My company is in the process of adopting Apache's open source stack,
> and
> > >> I've been tasked with building Aurora jobs to deploy a few popular
> open
> > >> source technologies on Mesos. Aurora is an elegant scheduler and in
> our
> > >> estimation will met our company's needs. However, we are struggling to
> > >>meet
> > >> some of the configuration requirements of some of the tools we wish to
> > >> deploy.
> > >>
> > >> Scenario: When a distributed service is deployed, we need to
> > >> programmatically determine all hosts selected for deployment and their
> > >> reserved ports in order to properly configure the service. We've
> solved
> > >> this in a few not-so-elegant ways:
> > >>
> > >>  1.  We wrote a Process to publish host/port information to a
> > >>distributed
> > >> file system, block until {{instances}} were written, read the
> > >>information
> > >> and finally configure the service. This works, but IMO it is not an
> > >>elegant
> > >> solution.
> > >>  2.  Next, we designed a REST API for service registration (based off
> of
> > >> the Aurora job key) and published this information to our ZooKeeper
> > >> ensemble. This solution removes the dependency on a pre-configured
> > >> distributed file system. However, some overhead is still required
> > >>(multiple
> > >> instances are necessary so as to not introduce a point of failure).
> > >>Aurora
> > >> jobs now require some initial configuration to be able to communicate
> > >>with
> > >> the service. Communication to this API is a little non-trivial because
> > >>the
> > >> REST service doesn't block until all information is communicated ­
> this
> > >> would be problematic due to the nature of the HTTP protocol.
> > >>
> > >> At this point, we realized that an even better solution might be to
> > >> communicate directly to Aurora scheduler to get this data. Seeing as
> > >>Aurora
> > >> scripts are just Python, we could probably implement it in a reusable
> > >> fashionŠbut I'm curious if anyone has already gone down this path?
> > >>
> > >> Thank you,
> > >> James O
> > >>
> >
> >
>

Reply via email to