I'm nothing if not persistent! After feedback from my second try at
proposing a way of doing defaults and aliases, I've got a third draft.
Feedback please!
# Defaults and Shorthands in the Aurora Client
## Motivation
Most of the time, users are doing the same thing, over and over again.
They're
working mainly on one particular service, in one particular workspace. But
they
need to repeat the same parameters, over and over again, and they need to
remember what those parameters are, what order they occur in, what format
they
use, and other details that can be difficult to remember.
In order to avoid these problems, users set up custom scripts that make it
easy
for them to run commands like "`myservice start`", instead of "`aurora job
create west/markcc/prod/myserver
src/main/aurora/myservice/myservice.aurora`"
Scripts aren't necessarily a bad thing. We've designed the aurora client to
be script friendly: every command has at least an option supporting
easy-to-parse
output. But scripts also often cover for problems that really shouldn't
exist.
Scripts get written for a variety of reasons. Among the many reasons that
people implement
scripts, there are scripts that add custom functionality for a specific
group of users;
scripts that can cover up for inadequate or missing core functionality; and
there are
scripts that cover up for poor user interfaces.
In aurora, we've worked hard to eliminate cases where the core
functionality of the
command-line client is inadequate. But our user interface still needs a lot
of work.
In particular, commands in Aurora often require very long parameters, which
users have
a hard time remembering. Look at the example above - it's a pretty typical
case. The
user wants to create a job. They're going to be creating the same job, over
and over again.
But to run it, they need to type out 68 characters for the two parameters!
This is a
prime example of the kind of case where users will write scripts, not to
provide new/special
functionality, nor to cover for inadequate functionality, but just because
it's so
painful to remember and correctly type out an overly long string of
parameters.
To reduce this, we'd like to support a way for users to set up a
configuration
file that defines defaults and shorthands for their everyday work. With
shorthands, a user that only works with a single service could say "aurora
job
create", instead of needing to spell out the full jobkey and configuration
file
location; a user working with multiple datacenters could say "`aurora job
create @east`" or "`aurora job create @west`" to select the correct jobkey.
## Proposal: Aurora Init Files
To allow users to customize shorthands, we'll provide a
builtin capability to allow users to provide a configuration
file, from which their customizations will be loaded.
Many applications use a simple pattern to solve similar problems.
Vagrant uses a file named "Vagrantfile"; when you run vagrant, if you
don't specifically tell the tool where to find a configuration, it looks
in the current directory or one of its parents to find a file named
"Vagrantfile".
We'd like to follow a similar pattern, and create an "AuroraInit" file.
The aurora init file is found by searching the following locations, in
order:
* the contents of the "--init-file" parameter.
* if the "--init-file" parameter is unspecified, then look in the current
directory for a file named "AuroraInit".
* if no "AuroraInit" file exists in the current directory, then look in
the users
home directory for an init file named ".aurora/init".
> **Sidebar: Why fallback to the users home directory?**
>
> Codebases are often shared between multiple projects, each of which lives
in
> a set of subdirectories within the codebase. For example, just look at
the aurora
> sourcecode, which includes the aurora scheduler, the aurora executor,
thermos,
> and the aurora client.
>
> If multiple services live in a codebase, then users can't put an
AuroraInit file in
> the root directory of the codebase, because it would interfere with other
users'
> work.
>
> The home directory fallback provides an easy way for users to set up a
pointer
> to the correct init file, which won't be removed by operations like
switching branches
> in a source repository. Users write shared init files, which are located
in their projects
> in the codebase, and create a symbolic link from their project init file
to their home
> directory.
### What goes into an init file?
We should support the following kinds of things:
1. Universal defaults - user-defined default settings that will be
applied to
all commands. For example, if there is a default config file that
should always
be used if the user doesn't specify one, that would be a universal
default.
2. Command specific defaults - users should be able to specify that they
always want
to use certain parameter settings for a specific command. For example,
if they
want to always use a default batch size of 10 for updates, but don't
want to affect
batch sizes for other commands like kill, they could use a command
specific default.
3. Aliases - shorthand names for longer parameters. A user could
specify shorthands
"east" and "west" for full jobkeys in two different datacenters.
### Defaults
A default specifies a set of _bindings_. If a parameter is omitted
from a command, and there's a binding for that parameter, it will be
automatically inserted into the command as if the user had typed it.
The binding is specified in the configuration file using a Python
dictionary.
For example, if the defaults included `{'jobspec':
'devcluster/me/prod/service'}`,
then if you ran `aurora job create` without specifying any parameters, the
command-line
would automatically substitute `devcluster/me/prod/service` for the
`jobspec` parameter.
Defaults can be declared either globally (in which case they'll be inserted
as parameters
for all commands), or for specific commands (in which case they'll only be
inserted for a
single command).
### Aliases
An alias is a short equivalent for a parameter. When a command line is
provided by a user, aliases will be expanded inline. A user can
specifically
mark an alias for expansion by prefixing it with "@"; if an alias
appears on the command-line surrounded by whitespace, it will be
replaced even if it isn't marked with an "@".
> **Sidebar: why not just use shell substitutions?**
>
> The @-substitution model proposed here is very similar to unix variable
> substitution. So why implement the same thing all over again? Why not
just tell
> users to use their shell?
>
> There are several reasons.
>
> 1. All of the options and aliases that affect aurora command invocations
can be specified
> in one place: the AuroraInit file.
> 2. Command shell substitution is complex - the ways in which the command
shell does
> expansion, tokenization, substitution, and parsing can be very hard to
follow. Adding
> the aurora aliases to that process just adds another layer of confusion
to the process.
> With internal alias substitution, we can avoid the confusions of
expansions and
> tokenization for aurora aliases.
> 3. We can provide better debugging and error messages with our own alias
substitution.
> For example, if a user specifies a job using aliases like "`aurora job
create @c/@me/test/myservice`",
> we can create an error message that shows exactly what they typed, and
how it expanded:
> <pre>
> Job "west/markcc/test/myservice" not found in configuration file
config.aurora.
> Jobkey was generated by alias expansion: original key was
"@c/@me/test/myservice", where:
> - "@c" was expanded to "west"
> - "@me" was expanded to "markcc"
> config_file parameter "config.aurora" was specified from defaults.
> </pre>
### Defining Shorthands and Defaults: Syntax and Examples
The init file is, like aurora job configurations, a python source file
using a Pystachio
schema. The schema is loaded, and an `Init` object is retrieved from the
top-level
variable `init` in that file.
The pystachio schema for init files is:
class CommandDefaults(Struct)
command = Required(String)
defaults=Map(String, String)
class Init(Struct):
defaults=Map(String, String)
command_defaults = Optional(CommandDefaults)
aliases=Map(String, String)
For example, an init file could contain the following:
init=Init(
defaults={
"jobspec": "west/markcc/test/myservice",
"config_file": "./src/aurora/myservice.aurora"
},
aliases={
"east": "east/markcc/test/myservice",
"c": "east",
"me": "markchucarroll"
},
command_defaults(command="job update",
defaults={"--batch-size": 10}
))
With this configuration file, if the user ran "`aurora job create`" without
any parameters,
it would automatically be expanded to "`aurora job create
west/markcc/test/myservice ./src/aurora/myservice.aurora`".
If a user ran "`aurora job create east`", the alias `east` would be
expanded to "east/markcc/test/myservice", and the missing config file
parameter would
be instantiated using the default, to create a command-line:
"`aurora job create east/markcc/test/myservice
./src/aurora/myservice.aurora`".
If a user ran "`aurora job create @c/@me/test/myservice`", the two aliases
would
be expanded, and the omitted configuration parameter would be added:
"`aurora job create east/markchucarroll/test/myservice
./src/aurora/myservice.aurora`".
If a user ran "`aurora job update`", then the `jobspec` and `config_file`
parameters would
get inserted from the global defaults, and the "--batch-size=10" would be
inserted from
the command defaults for "aurora job update".
