I actually don't agree with standardizing the config file name.
Here's the issue: if you're working with typical modern tools - gradle,
maven, pants, sbt, ..., you're doing your work in the root of a fairly deep
source hierarchy. Living in that hierarchy, there are likely multiple
aurora job definitions.
There's no way of just picking a default config file name that's not going
to cause confusion and disruption, because the config file sitting in the
root of the project directory is going to be wrong for many users.
That was the point of the home-directory fallback thing; it provided users
with a way to define a default config file from outside of the workspace,
so that different users could set it differently.
In any case: I'm punting on this for now. It doesn't seem like there's
anything close to a consensus that this is worth doing, so I'm going to
shelve it, and we can come back to it if the consensus changes later.
-Mark
On Thu, Jun 19, 2014 at 4:08 PM, James Oliver <jdo...@gmail.com> wrote:
> +1 for standardizing the config file name
> On Jun 19, 2014 1:01 PM, "Bill Farner" <wfar...@apache.org> wrote:
>
> > I agree with Joe, and tend to disagree with the motivation (for aliases,
> at
> > least). The end-result seems to be obfuscation (i'd likely need to peek
> in
> > the init file to figure out what a command does, or what command to run),
> > and i'm doubtful it will result in converting users away from wrapper
> > scripts.
> >
> > I also feel like our configuration files are already very complex (nature
> > of the beast), and aliases make them even more so.
> >
> > I am, however, more comfortable with the standard/assumed config file
> > naming (i.e. like Vagrantfile).
> >
> >
> > -=Bill
> >
> >
> > On Fri, Jun 13, 2014 at 11:27 AM, Joseph Smith <yasumo...@gmail.com>
> > wrote:
> >
> > > "to set up a configuration file”
> > >
> > > This line to me just says “we’re reinventing wrapper scripts” where
> > > someone could create an “alias” like ./east.sh and ./west.sh to do
> > deploys
> > > to various clusters. In that sense, these aliases etc are also “hiding”
> > bad
> > > interfaces, just in a different way- and one which isn’t as
> > > well-tested/understood as shell scripts.
> > >
> > > It seems like we’re set on putting time into this, so if we are, then
> > > ignore me and move onward :). IF we were going to go with an approach,
> > this
> > > seems reasonable based on ways that I’ve seen people wrap their
> > > configs/invocations of the client.
> > >
> > > On Jun 12, 2014, at 7:50 AM, Mark Chu-Carroll <mchucarr...@apache.org>
> > > wrote:
> > >
> > > > I'm nothing if not persistent! After feedback from my second try at
> > > > proposing a way of doing defaults and aliases, I've got a third
> draft.
> > > >
> > > > Feedback please!
> > > >
> > > >
> > > >
> > > >
> > > > # Defaults and Shorthands in the Aurora Client
> > > >
> > > > ## Motivation
> > > >
> > > > Most of the time, users are doing the same thing, over and over
> again.
> > > > They're
> > > > working mainly on one particular service, in one particular
> workspace.
> > > But
> > > > they
> > > > need to repeat the same parameters, over and over again, and they
> need
> > to
> > > > remember what those parameters are, what order they occur in, what
> > format
> > > > they
> > > > use, and other details that can be difficult to remember.
> > > >
> > > > In order to avoid these problems, users set up custom scripts that
> make
> > > it
> > > > easy
> > > > for them to run commands like "`myservice start`", instead of
> "`aurora
> > > job
> > > > create west/markcc/prod/myserver
> > > > src/main/aurora/myservice/myservice.aurora`"
> > > >
> > > > Scripts aren't necessarily a bad thing. We've designed the aurora
> > client
> > > to
> > > > be script friendly: every command has at least an option supporting
> > > > easy-to-parse
> > > > output. But scripts also often cover for problems that really
> shouldn't
> > > > exist.
> > > >
> > > > Scripts get written for a variety of reasons. Among the many reasons
> > that
> > > > people implement
> > > > scripts, there are scripts that add custom functionality for a
> specific
> > > > group of users;
> > > > scripts that can cover up for inadequate or missing core
> functionality;
> > > and
> > > > there are
> > > > scripts that cover up for poor user interfaces.
> > > >
> > > > In aurora, we've worked hard to eliminate cases where the core
> > > > functionality of the
> > > > command-line client is inadequate. But our user interface still
> needs a
> > > lot
> > > > of work.
> > > > In particular, commands in Aurora often require very long parameters,
> > > which
> > > > users have
> > > > a hard time remembering. Look at the example above - it's a pretty
> > > typical
> > > > case. The
> > > > user wants to create a job. They're going to be creating the same
> job,
> > > over
> > > > and over again.
> > > > But to run it, they need to type out 68 characters for the two
> > > parameters!
> > > > This is a
> > > > prime example of the kind of case where users will write scripts, not
> > to
> > > > provide new/special
> > > > functionality, nor to cover for inadequate functionality, but just
> > > because
> > > > it's so
> > > > painful to remember and correctly type out an overly long string of
> > > > parameters.
> > > >
> > > >
> > > > To reduce this, we'd like to support a way for users to set up a
> > > > configuration
> > > > file that defines defaults and shorthands for their everyday work.
> With
> > > > shorthands, a user that only works with a single service could say
> > > "aurora
> > > > job
> > > > create", instead of needing to spell out the full jobkey and
> > > configuration
> > > > file
> > > > location; a user working with multiple datacenters could say "`aurora
> > job
> > > > create @east`" or "`aurora job create @west`" to select the correct
> > > jobkey.
> > > >
> > > > ## Proposal: Aurora Init Files
> > > >
> > > > To allow users to customize shorthands, we'll provide a
> > > > builtin capability to allow users to provide a configuration
> > > > file, from which their customizations will be loaded.
> > > > Many applications use a simple pattern to solve similar problems.
> > > > Vagrant uses a file named "Vagrantfile"; when you run vagrant, if you
> > > > don't specifically tell the tool where to find a configuration, it
> > looks
> > > > in the current directory or one of its parents to find a file named
> > > > "Vagrantfile".
> > > >
> > > > We'd like to follow a similar pattern, and create an "AuroraInit"
> file.
> > > > The aurora init file is found by searching the following locations,
> in
> > > > order:
> > > >
> > > > * the contents of the "--init-file" parameter.
> > > > * if the "--init-file" parameter is unspecified, then look in the
> > > current
> > > > directory for a file named "AuroraInit".
> > > > * if no "AuroraInit" file exists in the current directory, then
> look
> > in
> > > > the users
> > > > home directory for an init file named ".aurora/init".
> > > >
> > > >> **Sidebar: Why fallback to the users home directory?**
> > > >>
> > > >> Codebases are often shared between multiple projects, each of which
> > > lives
> > > > in
> > > >> a set of subdirectories within the codebase. For example, just look
> at
> > > > the aurora
> > > >> sourcecode, which includes the aurora scheduler, the aurora
> executor,
> > > > thermos,
> > > >> and the aurora client.
> > > >>
> > > >> If multiple services live in a codebase, then users can't put an
> > > > AuroraInit file in
> > > >> the root directory of the codebase, because it would interfere with
> > > other
> > > > users'
> > > >> work.
> > > >>
> > > >> The home directory fallback provides an easy way for users to set
> up a
> > > > pointer
> > > >> to the correct init file, which won't be removed by operations like
> > > > switching branches
> > > >> in a source repository. Users write shared init files, which are
> > located
> > > > in their projects
> > > >> in the codebase, and create a symbolic link from their project init
> > file
> > > > to their home
> > > >> directory.
> > > >
> > > > ### What goes into an init file?
> > > >
> > > > We should support the following kinds of things:
> > > >
> > > > 1. Universal defaults - user-defined default settings that will be
> > > > applied to
> > > > all commands. For example, if there is a default config file that
> > > > should always
> > > > be used if the user doesn't specify one, that would be a
> universal
> > > > default.
> > > > 2. Command specific defaults - users should be able to specify that
> > > they
> > > > always want
> > > > to use certain parameter settings for a specific command. For
> > > example,
> > > > if they
> > > > want to always use a default batch size of 10 for updates, but
> > don't
> > > > want to affect
> > > > batch sizes for other commands like kill, they could use a
> command
> > > > specific default.
> > > > 3. Aliases - shorthand names for longer parameters. A user could
> > > > specify shorthands
> > > > "east" and "west" for full jobkeys in two different datacenters.
> > > >
> > > >
> > > > ### Defaults
> > > >
> > > > A default specifies a set of _bindings_. If a parameter is omitted
> > > > from a command, and there's a binding for that parameter, it will be
> > > > automatically inserted into the command as if the user had typed it.
> > > > The binding is specified in the configuration file using a Python
> > > > dictionary.
> > > >
> > > > For example, if the defaults included `{'jobspec':
> > > > 'devcluster/me/prod/service'}`,
> > > > then if you ran `aurora job create` without specifying any
> parameters,
> > > the
> > > > command-line
> > > > would automatically substitute `devcluster/me/prod/service` for the
> > > > `jobspec` parameter.
> > > >
> > > > Defaults can be declared either globally (in which case they'll be
> > > inserted
> > > > as parameters
> > > > for all commands), or for specific commands (in which case they'll
> only
> > > be
> > > > inserted for a
> > > > single command).
> > > >
> > > >
> > > > ### Aliases
> > > >
> > > > An alias is a short equivalent for a parameter. When a command line
> is
> > > > provided by a user, aliases will be expanded inline. A user can
> > > > specifically
> > > > mark an alias for expansion by prefixing it with "@"; if an alias
> > > > appears on the command-line surrounded by whitespace, it will be
> > > > replaced even if it isn't marked with an "@".
> > > >
> > > >> **Sidebar: why not just use shell substitutions?**
> > > >>
> > > >> The @-substitution model proposed here is very similar to unix
> > variable
> > > >> substitution. So why implement the same thing all over again? Why
> not
> > > > just tell
> > > >> users to use their shell?
> > > >>
> > > >> There are several reasons.
> > > >>
> > > >> 1. All of the options and aliases that affect aurora command
> > invocations
> > > > can be specified
> > > >> in one place: the AuroraInit file.
> > > >> 2. Command shell substitution is complex - the ways in which the
> > command
> > > > shell does
> > > >> expansion, tokenization, substitution, and parsing can be very hard
> > to
> > > > follow. Adding
> > > >> the aurora aliases to that process just adds another layer of
> > confusion
> > > > to the process.
> > > >> With internal alias substitution, we can avoid the confusions of
> > > > expansions and
> > > >> tokenization for aurora aliases.
> > > >> 3. We can provide better debugging and error messages with our own
> > alias
> > > > substitution.
> > > >> For example, if a user specifies a job using aliases like "`aurora
> > job
> > > > create @c/@me/test/myservice`",
> > > >> we can create an error message that shows exactly what they typed,
> > and
> > > > how it expanded:
> > > >> <pre>
> > > >> Job "west/markcc/test/myservice" not found in configuration file
> > > > config.aurora.
> > > >> Jobkey was generated by alias expansion: original key was
> > > > "@c/@me/test/myservice", where:
> > > >> - "@c" was expanded to "west"
> > > >> - "@me" was expanded to "markcc"
> > > >> config_file parameter "config.aurora" was specified from defaults.
> > > >> </pre>
> > > >
> > > >
> > > > ### Defining Shorthands and Defaults: Syntax and Examples
> > > >
> > > > The init file is, like aurora job configurations, a python source
> file
> > > > using a Pystachio
> > > > schema. The schema is loaded, and an `Init` object is retrieved from
> > the
> > > > top-level
> > > > variable `init` in that file.
> > > >
> > > > The pystachio schema for init files is:
> > > >
> > > > class CommandDefaults(Struct)
> > > > command = Required(String)
> > > > defaults=Map(String, String)
> > > >
> > > > class Init(Struct):
> > > > defaults=Map(String, String)
> > > > command_defaults = Optional(CommandDefaults)
> > > > aliases=Map(String, String)
> > > >
> > > >
> > > > For example, an init file could contain the following:
> > > >
> > > > init=Init(
> > > > defaults={
> > > > "jobspec": "west/markcc/test/myservice",
> > > > "config_file": "./src/aurora/myservice.aurora"
> > > > },
> > > > aliases={
> > > > "east": "east/markcc/test/myservice",
> > > > "c": "east",
> > > > "me": "markchucarroll"
> > > > },
> > > > command_defaults(command="job update",
> > > > defaults={"--batch-size": 10}
> > > > ))
> > > >
> > > >
> > > > With this configuration file, if the user ran "`aurora job create`"
> > > without
> > > > any parameters,
> > > > it would automatically be expanded to "`aurora job create
> > > > west/markcc/test/myservice ./src/aurora/myservice.aurora`".
> > > >
> > > > If a user ran "`aurora job create east`", the alias `east` would be
> > > > expanded to "east/markcc/test/myservice", and the missing config file
> > > > parameter would
> > > > be instantiated using the default, to create a command-line:
> > > > "`aurora job create east/markcc/test/myservice
> > > > ./src/aurora/myservice.aurora`".
> > > >
> > > > If a user ran "`aurora job create @c/@me/test/myservice`", the two
> > > aliases
> > > > would
> > > > be expanded, and the omitted configuration parameter would be added:
> > > > "`aurora job create east/markchucarroll/test/myservice
> > > > ./src/aurora/myservice.aurora`".
> > > >
> > > > If a user ran "`aurora job update`", then the `jobspec` and
> > `config_file`
> > > > parameters would
> > > > get inserted from the global defaults, and the "--batch-size=10"
> would
> > be
> > > > inserted from
> > > > the command defaults for "aurora job update".
> > >
> > >
> >
>
