"to set up a configuration file”
This line to me just says “we’re reinventing wrapper scripts” where someone
could create an “alias” like ./east.sh and ./west.sh to do deploys to various
clusters. In that sense, these aliases etc are also “hiding” bad interfaces,
just in a different way- and one which isn’t as well-tested/understood as shell
scripts.
It seems like we’re set on putting time into this, so if we are, then ignore me
and move onward :). IF we were going to go with an approach, this seems
reasonable based on ways that I’ve seen people wrap their configs/invocations
of the client.
On Jun 12, 2014, at 7:50 AM, Mark Chu-Carroll <mchucarr...@apache.org> wrote:
> I'm nothing if not persistent! After feedback from my second try at
> proposing a way of doing defaults and aliases, I've got a third draft.
>
> Feedback please!
>
>
>
>
> # Defaults and Shorthands in the Aurora Client
>
> ## Motivation
>
> Most of the time, users are doing the same thing, over and over again.
> They're
> working mainly on one particular service, in one particular workspace. But
> they
> need to repeat the same parameters, over and over again, and they need to
> remember what those parameters are, what order they occur in, what format
> they
> use, and other details that can be difficult to remember.
>
> In order to avoid these problems, users set up custom scripts that make it
> easy
> for them to run commands like "`myservice start`", instead of "`aurora job
> create west/markcc/prod/myserver
> src/main/aurora/myservice/myservice.aurora`"
>
> Scripts aren't necessarily a bad thing. We've designed the aurora client to
> be script friendly: every command has at least an option supporting
> easy-to-parse
> output. But scripts also often cover for problems that really shouldn't
> exist.
>
> Scripts get written for a variety of reasons. Among the many reasons that
> people implement
> scripts, there are scripts that add custom functionality for a specific
> group of users;
> scripts that can cover up for inadequate or missing core functionality; and
> there are
> scripts that cover up for poor user interfaces.
>
> In aurora, we've worked hard to eliminate cases where the core
> functionality of the
> command-line client is inadequate. But our user interface still needs a lot
> of work.
> In particular, commands in Aurora often require very long parameters, which
> users have
> a hard time remembering. Look at the example above - it's a pretty typical
> case. The
> user wants to create a job. They're going to be creating the same job, over
> and over again.
> But to run it, they need to type out 68 characters for the two parameters!
> This is a
> prime example of the kind of case where users will write scripts, not to
> provide new/special
> functionality, nor to cover for inadequate functionality, but just because
> it's so
> painful to remember and correctly type out an overly long string of
> parameters.
>
>
> To reduce this, we'd like to support a way for users to set up a
> configuration
> file that defines defaults and shorthands for their everyday work. With
> shorthands, a user that only works with a single service could say "aurora
> job
> create", instead of needing to spell out the full jobkey and configuration
> file
> location; a user working with multiple datacenters could say "`aurora job
> create @east`" or "`aurora job create @west`" to select the correct jobkey.
>
> ## Proposal: Aurora Init Files
>
> To allow users to customize shorthands, we'll provide a
> builtin capability to allow users to provide a configuration
> file, from which their customizations will be loaded.
> Many applications use a simple pattern to solve similar problems.
> Vagrant uses a file named "Vagrantfile"; when you run vagrant, if you
> don't specifically tell the tool where to find a configuration, it looks
> in the current directory or one of its parents to find a file named
> "Vagrantfile".
>
> We'd like to follow a similar pattern, and create an "AuroraInit" file.
> The aurora init file is found by searching the following locations, in
> order:
>
> * the contents of the "--init-file" parameter.
> * if the "--init-file" parameter is unspecified, then look in the current
> directory for a file named "AuroraInit".
> * if no "AuroraInit" file exists in the current directory, then look in
> the users
> home directory for an init file named ".aurora/init".
>
>> **Sidebar: Why fallback to the users home directory?**
>>
>> Codebases are often shared between multiple projects, each of which lives
> in
>> a set of subdirectories within the codebase. For example, just look at
> the aurora
>> sourcecode, which includes the aurora scheduler, the aurora executor,
> thermos,
>> and the aurora client.
>>
>> If multiple services live in a codebase, then users can't put an
> AuroraInit file in
>> the root directory of the codebase, because it would interfere with other
> users'
>> work.
>>
>> The home directory fallback provides an easy way for users to set up a
> pointer
>> to the correct init file, which won't be removed by operations like
> switching branches
>> in a source repository. Users write shared init files, which are located
> in their projects
>> in the codebase, and create a symbolic link from their project init file
> to their home
>> directory.
>
> ### What goes into an init file?
>
> We should support the following kinds of things:
>
> 1. Universal defaults - user-defined default settings that will be
> applied to
> all commands. For example, if there is a default config file that
> should always
> be used if the user doesn't specify one, that would be a universal
> default.
> 2. Command specific defaults - users should be able to specify that they
> always want
> to use certain parameter settings for a specific command. For example,
> if they
> want to always use a default batch size of 10 for updates, but don't
> want to affect
> batch sizes for other commands like kill, they could use a command
> specific default.
> 3. Aliases - shorthand names for longer parameters. A user could
> specify shorthands
> "east" and "west" for full jobkeys in two different datacenters.
>
>
> ### Defaults
>
> A default specifies a set of _bindings_. If a parameter is omitted
> from a command, and there's a binding for that parameter, it will be
> automatically inserted into the command as if the user had typed it.
> The binding is specified in the configuration file using a Python
> dictionary.
>
> For example, if the defaults included `{'jobspec':
> 'devcluster/me/prod/service'}`,
> then if you ran `aurora job create` without specifying any parameters, the
> command-line
> would automatically substitute `devcluster/me/prod/service` for the
> `jobspec` parameter.
>
> Defaults can be declared either globally (in which case they'll be inserted
> as parameters
> for all commands), or for specific commands (in which case they'll only be
> inserted for a
> single command).
>
>
> ### Aliases
>
> An alias is a short equivalent for a parameter. When a command line is
> provided by a user, aliases will be expanded inline. A user can
> specifically
> mark an alias for expansion by prefixing it with "@"; if an alias
> appears on the command-line surrounded by whitespace, it will be
> replaced even if it isn't marked with an "@".
>
>> **Sidebar: why not just use shell substitutions?**
>>
>> The @-substitution model proposed here is very similar to unix variable
>> substitution. So why implement the same thing all over again? Why not
> just tell
>> users to use their shell?
>>
>> There are several reasons.
>>
>> 1. All of the options and aliases that affect aurora command invocations
> can be specified
>> in one place: the AuroraInit file.
>> 2. Command shell substitution is complex - the ways in which the command
> shell does
>> expansion, tokenization, substitution, and parsing can be very hard to
> follow. Adding
>> the aurora aliases to that process just adds another layer of confusion
> to the process.
>> With internal alias substitution, we can avoid the confusions of
> expansions and
>> tokenization for aurora aliases.
>> 3. We can provide better debugging and error messages with our own alias
> substitution.
>> For example, if a user specifies a job using aliases like "`aurora job
> create @c/@me/test/myservice`",
>> we can create an error message that shows exactly what they typed, and
> how it expanded:
>> <pre>
>> Job "west/markcc/test/myservice" not found in configuration file
> config.aurora.
>> Jobkey was generated by alias expansion: original key was
> "@c/@me/test/myservice", where:
>> - "@c" was expanded to "west"
>> - "@me" was expanded to "markcc"
>> config_file parameter "config.aurora" was specified from defaults.
>> </pre>
>
>
> ### Defining Shorthands and Defaults: Syntax and Examples
>
> The init file is, like aurora job configurations, a python source file
> using a Pystachio
> schema. The schema is loaded, and an `Init` object is retrieved from the
> top-level
> variable `init` in that file.
>
> The pystachio schema for init files is:
>
> class CommandDefaults(Struct)
> command = Required(String)
> defaults=Map(String, String)
>
> class Init(Struct):
> defaults=Map(String, String)
> command_defaults = Optional(CommandDefaults)
> aliases=Map(String, String)
>
>
> For example, an init file could contain the following:
>
> init=Init(
> defaults={
> "jobspec": "west/markcc/test/myservice",
> "config_file": "./src/aurora/myservice.aurora"
> },
> aliases={
> "east": "east/markcc/test/myservice",
> "c": "east",
> "me": "markchucarroll"
> },
> command_defaults(command="job update",
> defaults={"--batch-size": 10}
> ))
>
>
> With this configuration file, if the user ran "`aurora job create`" without
> any parameters,
> it would automatically be expanded to "`aurora job create
> west/markcc/test/myservice ./src/aurora/myservice.aurora`".
>
> If a user ran "`aurora job create east`", the alias `east` would be
> expanded to "east/markcc/test/myservice", and the missing config file
> parameter would
> be instantiated using the default, to create a command-line:
> "`aurora job create east/markcc/test/myservice
> ./src/aurora/myservice.aurora`".
>
> If a user ran "`aurora job create @c/@me/test/myservice`", the two aliases
> would
> be expanded, and the omitted configuration parameter would be added:
> "`aurora job create east/markchucarroll/test/myservice
> ./src/aurora/myservice.aurora`".
>
> If a user ran "`aurora job update`", then the `jobspec` and `config_file`
> parameters would
> get inserted from the global defaults, and the "--batch-size=10" would be
> inserted from
> the command defaults for "aurora job update".
