Speeding up Aurora client job creation

2015-02-11 Thread Hussein Elgridly
g up the machinery for doing so) I imagine I can sidestep #1 with a check for "{{" in the job file and bypass Pystachio entirely. Can I also skip the Aurora client entirely and talk directly to the scheduler? If so what does that entail, and are there any risks associated? Thanks, -Husse

Getting secure data into Docker containers

2015-02-18 Thread Hussein Elgridly
open. The ideal solution would be to put the creds on the Mesos slaves and then mount them on the container, but Aurora doesn't have the means to do this yet. If the answer is "wait a week and AURORA-1107 will be done", then great; but if not, anyone have any ideas? Thanks, Hussein

Re: Soliciting feature requests for 0.8.0

2015-02-19 Thread Hussein Elgridly
Things I'd like to see because they'd make my life easier: AURORA-987 Create a first-class REST-like scheduler interface. AURORA-1107: Add support for mounting external volumes into docker containers. +1 on the documentation front also; there's a lot I've had to figure out myself (by asking on IR

Making sense of Aurora terminal states

2015-02-19 Thread Hussein Elgridly
l be rescheduled. 8. These rules are unlikely to change in the future ;) Finally, I noticed something odd: ASSIGNED -> LOST has followups [KILL, RESCHEDULE], but STARTING and RUNNING -> LOST only has [RESCHEDULE] as a followup. Why? Thanks, Hussein Elgridly Senior Software Engineer, DSDE The Broad Institute of MIT and Harvard

Re: Making sense of Aurora terminal states

2015-02-20 Thread Hussein Elgridly
27;s when that might happened" isn't made as explicit as it could be. I know I'd have had an easier time if there had been an explanation of "here's what each state means and what might happen next", and I can imagine [weasel words; citation needed] that other users mig

Re: Making sense of Aurora terminal states

2015-02-20 Thread Hussein Elgridly
back to Aurora that then gets thrown away? Hussein Elgridly Senior Software Engineer, DSDE The Broad Institute of MIT and Harvard On 20 February 2015 at 11:08, Hussein Elgridly wrote: > This is fantastic (and I'm glad that my understanding was mostly correct) > - thanks a lot. > &

Re: Making sense of Aurora terminal states

2015-02-20 Thread Hussein Elgridly
Also (sorry for repeated messages), what's the deal with KILLING -> [FINISHED, FAILED]? User sends kill request but Mesos reports it's done before it gets through so congratulations, you get to keep it? Hussein Elgridly Senior Software Engineer, DSDE The Broad Institute of MIT and Ha

Re: Making sense of Aurora terminal states

2015-02-21 Thread Hussein Elgridly
ossible that it's achievable by remapping nomenclature (where I usually say "job", replace with "task instance")... I'll give it a shot if I can find the time. Hussein Elgridly Senior Software Engineer, DSDE The Broad Institute of MIT and Harvard On 21 February 2015 a

Sharing scripts for deploying Aurora on Mesosphere

2015-02-21 Thread Hussein Elgridly
Hi, There's been some talk on IRC about better documentation for deploying Aurora in a non-Vagrant environment, and requests that users share their deployment notes. Our devops folks are putting together Puppet scripts for our production environment, but meanwhile I've been using Mesosphere as a d

Re: installing Aurora client fails

2015-03-11 Thread Hussein Elgridly
lient successfully, it should be in dist/aurora.pex relative to the root of the Git repo. Hussein Elgridly Senior Software Engineer, DSDE The Broad Institute of MIT and Harvard On 11 March 2015 at 17:43, Bill Farner wrote: > This isn't a solution, but are you able to try to repro in a 14.04 image?

Re: installing Aurora client fails

2015-03-12 Thread Hussein Elgridly
Thanks Bill - I've passed this on. Hussein Elgridly Senior Software Engineer, DSDE The Broad Institute of MIT and Harvard On 11 March 2015 at 18:02, Bill Farner wrote: > Hussein - you and/or your devops folks may benefit from this relic as a > guide: https://reviews.apache.

Re: Speeding up Aurora client job creation

2015-03-16 Thread Hussein Elgridly
executor can > understand. Turns out this is JSON data, so it should not be *too* > prohibitive. > > However, there is another technical limitation you will hit for the > submission rate you are after. The scheduler is backed by a durable store > whose write latency is at minimum the am

Re: Speeding up Aurora client job creation

2015-03-16 Thread Hussein Elgridly
I dug into TRequestsTransport and I get it now. Sending raw bytes across a socket is not the same as doing an HTTP POST with said bytes stuffed in the body! I guess I too will be rolling my own HTTP transport... Hussein Elgridly Senior Software Engineer, DSDE The Broad Institute of MIT and

Re: Speeding up Aurora client job creation

2015-03-16 Thread Hussein Elgridly
b.com/eleme/thriftpy Hussein Elgridly Senior Software Engineer, DSDE The Broad Institute of MIT and Harvard On 16 March 2015 at 19:11, Erb, Stephan wrote: > Just to make sure I get this correctly: You say, you cannot use the > existing python client because it is python 2.7 only so you wa

Re: Speeding up Aurora client job creation

2015-03-16 Thread Hussein Elgridly
master/src/main/python/apache/aurora/common/transport.py Hussein Elgridly Senior Software Engineer, DSDE The Broad Institute of MIT and Harvard On 16 March 2015 at 22:58, Bill Farner wrote: > Exploring the possibilities - can you use python 2.7? If so, you could > leverage some of the pri

Re: Speeding up Aurora client job creation

2015-03-17 Thread Hussein Elgridly
s); 2. Roll my own protocol, based on Thrift's code [1]; or 3. Backport my project to Python 2.7 and use official Thrift. [1] https://github.com/apache/thrift/blob/93fea15b51494a79992a5323c803325537134bd8/lib/py/src/protocol/TJSONProtocol.py Hussein Elgridly Senior Software Engineer, DSDE T

Getting exit codes from Thermos

2015-03-19 Thread Hussein Elgridly
ther the web or JSON interfaces. If it helps, the failed process has a start_time field, but is missing a stop_time. Any clues? Hussein Elgridly Senior Software Engineer, DSDE The Broad Institute of MIT and Harvard

Re: Speeding up Aurora client job creation

2015-03-20 Thread Hussein Elgridly
ke it into official Thrift. But it works for me, so I'm happy :) Hussein Elgridly Senior Software Engineer, DSDE The Broad Institute of MIT and Harvard On 17 March 2015 at 15:18, Hussein Elgridly wrote: > For anyone following along at home, I managed to make my own THTTPClient > f