Thanks for checking out Aurora!
My short answer is that Aurora should handle thousands of short-lived
tasks/jobs per day without trouble. (If you proceed with this approach and
encounter performance issues, feel free to file tickets!) The DSL does
have some mechanisms for parameterization. In y
Thanks, Bill.
Am I correct in understanding that is not possible to parameterize
individual Jobs, just Tasks? Therefore, since I don't know the job
definitions up front, I will have parameterized Task templates, and
generate a new Task every time I need to run a Job?
Is that the recommended route
A job is a group of nearly-identical tasks plus some constraints like rack
diversity. The scheduler considers each task within a job equivalently
schedulable, so you can't vary things like resource footprint. It's
perfectly fine to have several jobs with just a single task, as long as
each has a di
Thanks, Kevin. The idea of always-on workers of varying sizes is
effectively what we have right now in our non-Mesos world. The problem
is that sometimes we end up with not enough workers for certain
classes of jobs (e.g. High Memory), while part of the cluster sits
idle.
Conceptually, in my mind
Can you offer some more details on what the workload execution looks like?
Are these shell commands? An application that's provided different
configuration?
-=Bill
On Wed, Feb 26, 2014 at 12:45 PM, Bryan Helmkamp wrote:
> Thanks, Kevin. The idea of always-on workers of varying sizes is
> effe
>
> The problem
> is that sometimes we end up with not enough workers for certain
> classes of jobs (e.g. High Memory), while part of the cluster sits
> idle.
There's no prior art for this, but the Aurora API is actually designed in a
way that would make it possible to have a 'supervisor' job tha
Sure. Yes, they are shell commands and yes they are provided different
configuration on each run.
In effect we have a number of different job types that are queued up,
and we need to run as quickly as possible. Each job type has different
resource requirements. Every time we run the job, we provid
For a more dynamic approach to resource utilization you can use something
like this:
# dynamic.aurora
*# Enqueue each individual work-item with aurora create -E
work_item=$work_item -E resource_profile=graph_traversals
west/service-account-name/prod/process_$work_item*
class Profile(Struct):
que
Thanks, Kevin. That pretty much looks like exactly what I need.
-Bryan
On Wed, Feb 26, 2014 at 8:16 PM, Kevin Sweeney wrote:
> For a more dynamic approach to resource utilization you can use something
> like this:
>
> # dynamic.aurora
> *# Enqueue each individual work-item with aurora create -E
The example I gave is somewhat syntactically invalid due to coding via
email, but that's more or less what the interface will look like. I also
filed https://issues.apache.org/jira/browse/AURORA-236 for more first-class
support of the semantics I think you want (though currently you can fake it
by
And after a bit of code spelunking the semantics you want already exist
(just undocumented). Updated the ticket to update the documentation.
On Wed, Feb 26, 2014 at 6:00 PM, Kevin Sweeney wrote:
> The example I gave is somewhat syntactically invalid due to coding via
> email, but that's more or
Got it. Thanks. Do finished Jobs and Tasks get garbage collected
automatically at some point?
Otherwise it seems like they will stack up pretty fast. (We might run
hundreds of thousands of jobs in a day.)
BTW, Aurora does not seem to like the resources =
'{{resources[{{resource_profile}}]}}' part
On Wed, Feb 26, 2014 at 7:45 PM, Bryan Helmkamp wrote:
> Got it. Thanks. Do finished Jobs and Tasks get garbage collected
> automatically at some point?
> Otherwise it seems like they will stack up pretty fast. (We might run
> hundreds of thousands of jobs in a day.)
>
Jobs are garbage-collecte
13 matches
Mail list logo
