That's what I thought - but in pants/blaze/maven/..., I thought most people
normally left their command-line in the project root. I certainly do. If
you're doing that - and a lot of people do - then you can't easily
disambiguate between different services living in the same repos.
-Mark
On Thu, Jun 19, 2014 at 4:40 PM, Bill Farner <wfar...@apache.org> wrote:
> On Thu, Jun 19, 2014 at 1:36 PM, Mark Chu-Carroll <mchucarr...@apache.org>
> wrote:
>
> > Maybe I'm misunderstanding you?
> >
> > Where else should you look for a default configuration file, where it can
> > automatically differentiate between different services stored in the same
> > repository?
> >
>
> Working directory would be most intuitive to me.
>
>
> >
> > -Mark
> >
> >
> > On Thu, Jun 19, 2014 at 4:35 PM, Bill Farner <wfar...@apache.org> wrote:
> >
> > > > in the root of the project directory
> > >
> > > FWIW i don't see any particular reason why it needs to be in the root
> of
> > a
> > > repo/project dir.
> > >
> > > -=Bill
> > >
> > >
> > > On Thu, Jun 19, 2014 at 1:31 PM, Mark Chu-Carroll <
> > mchucarr...@apache.org>
> > > wrote:
> > >
> > > > I actually don't agree with standardizing the config file name.
> > > >
> > > > Here's the issue: if you're working with typical modern tools -
> gradle,
> > > > maven, pants, sbt, ..., you're doing your work in the root of a
> fairly
> > > deep
> > > > source hierarchy. Living in that hierarchy, there are likely multiple
> > > > aurora job definitions.
> > > >
> > > > There's no way of just picking a default config file name that's not
> > > going
> > > > to cause confusion and disruption, because the config file sitting in
> > the
> > > > root of the project directory is going to be wrong for many users.
> > > >
> > > > That was the point of the home-directory fallback thing; it provided
> > > users
> > > > with a way to define a default config file from outside of the
> > workspace,
> > > > so that different users could set it differently.
> > > >
> > > > In any case: I'm punting on this for now. It doesn't seem like
> there's
> > > > anything close to a consensus that this is worth doing, so I'm going
> to
> > > > shelve it, and we can come back to it if the consensus changes later.
> > > >
> > > > -Mark
> > > >
> > > >
> > > >
> > > >
> > > > On Thu, Jun 19, 2014 at 4:08 PM, James Oliver <jdo...@gmail.com>
> > wrote:
> > > >
> > > > > +1 for standardizing the config file name
> > > > > On Jun 19, 2014 1:01 PM, "Bill Farner" <wfar...@apache.org> wrote:
> > > > >
> > > > > > I agree with Joe, and tend to disagree with the motivation (for
> > > > aliases,
> > > > > at
> > > > > > least). The end-result seems to be obfuscation (i'd likely need
> to
> > > > peek
> > > > > in
> > > > > > the init file to figure out what a command does, or what command
> to
> > > > run),
> > > > > > and i'm doubtful it will result in converting users away from
> > wrapper
> > > > > > scripts.
> > > > > >
> > > > > > I also feel like our configuration files are already very complex
> > > > (nature
> > > > > > of the beast), and aliases make them even more so.
> > > > > >
> > > > > > I am, however, more comfortable with the standard/assumed config
> > file
> > > > > > naming (i.e. like Vagrantfile).
> > > > > >
> > > > > >
> > > > > > -=Bill
> > > > > >
> > > > > >
> > > > > > On Fri, Jun 13, 2014 at 11:27 AM, Joseph Smith <
> > yasumo...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > "to set up a configuration file”
> > > > > > >
> > > > > > > This line to me just says “we’re reinventing wrapper scripts”
> > where
> > > > > > > someone could create an “alias” like ./east.sh and ./west.sh to
> > do
> > > > > > deploys
> > > > > > > to various clusters. In that sense, these aliases etc are also
> > > > “hiding”
> > > > > > bad
> > > > > > > interfaces, just in a different way- and one which isn’t as
> > > > > > > well-tested/understood as shell scripts.
> > > > > > >
> > > > > > > It seems like we’re set on putting time into this, so if we
> are,
> > > then
> > > > > > > ignore me and move onward :). IF we were going to go with an
> > > > approach,
> > > > > > this
> > > > > > > seems reasonable based on ways that I’ve seen people wrap their
> > > > > > > configs/invocations of the client.
> > > > > > >
> > > > > > > On Jun 12, 2014, at 7:50 AM, Mark Chu-Carroll <
> > > > mchucarr...@apache.org>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > I'm nothing if not persistent! After feedback from my second
> > try
> > > at
> > > > > > > > proposing a way of doing defaults and aliases, I've got a
> third
> > > > > draft.
> > > > > > > >
> > > > > > > > Feedback please!
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > # Defaults and Shorthands in the Aurora Client
> > > > > > > >
> > > > > > > > ## Motivation
> > > > > > > >
> > > > > > > > Most of the time, users are doing the same thing, over and
> over
> > > > > again.
> > > > > > > > They're
> > > > > > > > working mainly on one particular service, in one particular
> > > > > workspace.
> > > > > > > But
> > > > > > > > they
> > > > > > > > need to repeat the same parameters, over and over again, and
> > they
> > > > > need
> > > > > > to
> > > > > > > > remember what those parameters are, what order they occur in,
> > > what
> > > > > > format
> > > > > > > > they
> > > > > > > > use, and other details that can be difficult to remember.
> > > > > > > >
> > > > > > > > In order to avoid these problems, users set up custom scripts
> > > that
> > > > > make
> > > > > > > it
> > > > > > > > easy
> > > > > > > > for them to run commands like "`myservice start`", instead of
> > > > > "`aurora
> > > > > > > job
> > > > > > > > create west/markcc/prod/myserver
> > > > > > > > src/main/aurora/myservice/myservice.aurora`"
> > > > > > > >
> > > > > > > > Scripts aren't necessarily a bad thing. We've designed the
> > aurora
> > > > > > client
> > > > > > > to
> > > > > > > > be script friendly: every command has at least an option
> > > supporting
> > > > > > > > easy-to-parse
> > > > > > > > output. But scripts also often cover for problems that really
> > > > > shouldn't
> > > > > > > > exist.
> > > > > > > >
> > > > > > > > Scripts get written for a variety of reasons. Among the many
> > > > reasons
> > > > > > that
> > > > > > > > people implement
> > > > > > > > scripts, there are scripts that add custom functionality for
> a
> > > > > specific
> > > > > > > > group of users;
> > > > > > > > scripts that can cover up for inadequate or missing core
> > > > > functionality;
> > > > > > > and
> > > > > > > > there are
> > > > > > > > scripts that cover up for poor user interfaces.
> > > > > > > >
> > > > > > > > In aurora, we've worked hard to eliminate cases where the
> core
> > > > > > > > functionality of the
> > > > > > > > command-line client is inadequate. But our user interface
> still
> > > > > needs a
> > > > > > > lot
> > > > > > > > of work.
> > > > > > > > In particular, commands in Aurora often require very long
> > > > parameters,
> > > > > > > which
> > > > > > > > users have
> > > > > > > > a hard time remembering. Look at the example above - it's a
> > > pretty
> > > > > > > typical
> > > > > > > > case. The
> > > > > > > > user wants to create a job. They're going to be creating the
> > same
> > > > > job,
> > > > > > > over
> > > > > > > > and over again.
> > > > > > > > But to run it, they need to type out 68 characters for the
> two
> > > > > > > parameters!
> > > > > > > > This is a
> > > > > > > > prime example of the kind of case where users will write
> > scripts,
> > > > not
> > > > > > to
> > > > > > > > provide new/special
> > > > > > > > functionality, nor to cover for inadequate functionality, but
> > > just
> > > > > > > because
> > > > > > > > it's so
> > > > > > > > painful to remember and correctly type out an overly long
> > string
> > > of
> > > > > > > > parameters.
> > > > > > > >
> > > > > > > >
> > > > > > > > To reduce this, we'd like to support a way for users to set
> up
> > a
> > > > > > > > configuration
> > > > > > > > file that defines defaults and shorthands for their everyday
> > > work.
> > > > > With
> > > > > > > > shorthands, a user that only works with a single service
> could
> > > say
> > > > > > > "aurora
> > > > > > > > job
> > > > > > > > create", instead of needing to spell out the full jobkey and
> > > > > > > configuration
> > > > > > > > file
> > > > > > > > location; a user working with multiple datacenters could say
> > > > "`aurora
> > > > > > job
> > > > > > > > create @east`" or "`aurora job create @west`" to select the
> > > correct
> > > > > > > jobkey.
> > > > > > > >
> > > > > > > > ## Proposal: Aurora Init Files
> > > > > > > >
> > > > > > > > To allow users to customize shorthands, we'll provide a
> > > > > > > > builtin capability to allow users to provide a configuration
> > > > > > > > file, from which their customizations will be loaded.
> > > > > > > > Many applications use a simple pattern to solve similar
> > problems.
> > > > > > > > Vagrant uses a file named "Vagrantfile"; when you run
> vagrant,
> > if
> > > > you
> > > > > > > > don't specifically tell the tool where to find a
> configuration,
> > > it
> > > > > > looks
> > > > > > > > in the current directory or one of its parents to find a file
> > > named
> > > > > > > > "Vagrantfile".
> > > > > > > >
> > > > > > > > We'd like to follow a similar pattern, and create an
> > "AuroraInit"
> > > > > file.
> > > > > > > > The aurora init file is found by searching the following
> > > locations,
> > > > > in
> > > > > > > > order:
> > > > > > > >
> > > > > > > > * the contents of the "--init-file" parameter.
> > > > > > > > * if the "--init-file" parameter is unspecified, then look
> in
> > > the
> > > > > > > current
> > > > > > > > directory for a file named "AuroraInit".
> > > > > > > > * if no "AuroraInit" file exists in the current directory,
> > then
> > > > > look
> > > > > > in
> > > > > > > > the users
> > > > > > > > home directory for an init file named ".aurora/init".
> > > > > > > >
> > > > > > > >> **Sidebar: Why fallback to the users home directory?**
> > > > > > > >>
> > > > > > > >> Codebases are often shared between multiple projects, each
> of
> > > > which
> > > > > > > lives
> > > > > > > > in
> > > > > > > >> a set of subdirectories within the codebase. For example,
> just
> > > > look
> > > > > at
> > > > > > > > the aurora
> > > > > > > >> sourcecode, which includes the aurora scheduler, the aurora
> > > > > executor,
> > > > > > > > thermos,
> > > > > > > >> and the aurora client.
> > > > > > > >>
> > > > > > > >> If multiple services live in a codebase, then users can't
> put
> > an
> > > > > > > > AuroraInit file in
> > > > > > > >> the root directory of the codebase, because it would
> interfere
> > > > with
> > > > > > > other
> > > > > > > > users'
> > > > > > > >> work.
> > > > > > > >>
> > > > > > > >> The home directory fallback provides an easy way for users
> to
> > > set
> > > > > up a
> > > > > > > > pointer
> > > > > > > >> to the correct init file, which won't be removed by
> operations
> > > > like
> > > > > > > > switching branches
> > > > > > > >> in a source repository. Users write shared init files, which
> > are
> > > > > > located
> > > > > > > > in their projects
> > > > > > > >> in the codebase, and create a symbolic link from their
> project
> > > > init
> > > > > > file
> > > > > > > > to their home
> > > > > > > >> directory.
> > > > > > > >
> > > > > > > > ### What goes into an init file?
> > > > > > > >
> > > > > > > > We should support the following kinds of things:
> > > > > > > >
> > > > > > > > 1. Universal defaults - user-defined default settings that
> > will
> > > > be
> > > > > > > > applied to
> > > > > > > > all commands. For example, if there is a default config
> > file
> > > > that
> > > > > > > > should always
> > > > > > > > be used if the user doesn't specify one, that would be a
> > > > > universal
> > > > > > > > default.
> > > > > > > > 2. Command specific defaults - users should be able to
> > specify
> > > > that
> > > > > > > they
> > > > > > > > always want
> > > > > > > > to use certain parameter settings for a specific command.
> > For
> > > > > > > example,
> > > > > > > > if they
> > > > > > > > want to always use a default batch size of 10 for
> updates,
> > > but
> > > > > > don't
> > > > > > > > want to affect
> > > > > > > > batch sizes for other commands like kill, they could use
> a
> > > > > command
> > > > > > > > specific default.
> > > > > > > > 3. Aliases - shorthand names for longer parameters. A user
> > > could
> > > > > > > > specify shorthands
> > > > > > > > "east" and "west" for full jobkeys in two different
> > > > datacenters.
> > > > > > > >
> > > > > > > >
> > > > > > > > ### Defaults
> > > > > > > >
> > > > > > > > A default specifies a set of _bindings_. If a parameter is
> > > omitted
> > > > > > > > from a command, and there's a binding for that parameter, it
> > will
> > > > be
> > > > > > > > automatically inserted into the command as if the user had
> > typed
> > > > it.
> > > > > > > > The binding is specified in the configuration file using a
> > Python
> > > > > > > > dictionary.
> > > > > > > >
> > > > > > > > For example, if the defaults included `{'jobspec':
> > > > > > > > 'devcluster/me/prod/service'}`,
> > > > > > > > then if you ran `aurora job create` without specifying any
> > > > > parameters,
> > > > > > > the
> > > > > > > > command-line
> > > > > > > > would automatically substitute `devcluster/me/prod/service`
> for
> > > the
> > > > > > > > `jobspec` parameter.
> > > > > > > >
> > > > > > > > Defaults can be declared either globally (in which case
> they'll
> > > be
> > > > > > > inserted
> > > > > > > > as parameters
> > > > > > > > for all commands), or for specific commands (in which case
> > > they'll
> > > > > only
> > > > > > > be
> > > > > > > > inserted for a
> > > > > > > > single command).
> > > > > > > >
> > > > > > > >
> > > > > > > > ### Aliases
> > > > > > > >
> > > > > > > > An alias is a short equivalent for a parameter. When a
> command
> > > line
> > > > > is
> > > > > > > > provided by a user, aliases will be expanded inline. A user
> > can
> > > > > > > > specifically
> > > > > > > > mark an alias for expansion by prefixing it with "@"; if an
> > alias
> > > > > > > > appears on the command-line surrounded by whitespace, it will
> > be
> > > > > > > > replaced even if it isn't marked with an "@".
> > > > > > > >
> > > > > > > >> **Sidebar: why not just use shell substitutions?**
> > > > > > > >>
> > > > > > > >> The @-substitution model proposed here is very similar to
> unix
> > > > > > variable
> > > > > > > >> substitution. So why implement the same thing all over
> again?
> > > Why
> > > > > not
> > > > > > > > just tell
> > > > > > > >> users to use their shell?
> > > > > > > >>
> > > > > > > >> There are several reasons.
> > > > > > > >>
> > > > > > > >> 1. All of the options and aliases that affect aurora command
> > > > > > invocations
> > > > > > > > can be specified
> > > > > > > >> in one place: the AuroraInit file.
> > > > > > > >> 2. Command shell substitution is complex - the ways in which
> > the
> > > > > > command
> > > > > > > > shell does
> > > > > > > >> expansion, tokenization, substitution, and parsing can be
> > very
> > > > hard
> > > > > > to
> > > > > > > > follow. Adding
> > > > > > > >> the aurora aliases to that process just adds another layer
> of
> > > > > > confusion
> > > > > > > > to the process.
> > > > > > > >> With internal alias substitution, we can avoid the
> confusions
> > > of
> > > > > > > > expansions and
> > > > > > > >> tokenization for aurora aliases.
> > > > > > > >> 3. We can provide better debugging and error messages with
> our
> > > own
> > > > > > alias
> > > > > > > > substitution.
> > > > > > > >> For example, if a user specifies a job using aliases like
> > > > "`aurora
> > > > > > job
> > > > > > > > create @c/@me/test/myservice`",
> > > > > > > >> we can create an error message that shows exactly what they
> > > > typed,
> > > > > > and
> > > > > > > > how it expanded:
> > > > > > > >> <pre>
> > > > > > > >> Job "west/markcc/test/myservice" not found in configuration
> > file
> > > > > > > > config.aurora.
> > > > > > > >> Jobkey was generated by alias expansion: original key was
> > > > > > > > "@c/@me/test/myservice", where:
> > > > > > > >> - "@c" was expanded to "west"
> > > > > > > >> - "@me" was expanded to "markcc"
> > > > > > > >> config_file parameter "config.aurora" was specified from
> > > > defaults.
> > > > > > > >> </pre>
> > > > > > > >
> > > > > > > >
> > > > > > > > ### Defining Shorthands and Defaults: Syntax and Examples
> > > > > > > >
> > > > > > > > The init file is, like aurora job configurations, a python
> > source
> > > > > file
> > > > > > > > using a Pystachio
> > > > > > > > schema. The schema is loaded, and an `Init` object is
> retrieved
> > > > from
> > > > > > the
> > > > > > > > top-level
> > > > > > > > variable `init` in that file.
> > > > > > > >
> > > > > > > > The pystachio schema for init files is:
> > > > > > > >
> > > > > > > > class CommandDefaults(Struct)
> > > > > > > > command = Required(String)
> > > > > > > > defaults=Map(String, String)
> > > > > > > >
> > > > > > > > class Init(Struct):
> > > > > > > > defaults=Map(String, String)
> > > > > > > > command_defaults = Optional(CommandDefaults)
> > > > > > > > aliases=Map(String, String)
> > > > > > > >
> > > > > > > >
> > > > > > > > For example, an init file could contain the following:
> > > > > > > >
> > > > > > > > init=Init(
> > > > > > > > defaults={
> > > > > > > > "jobspec": "west/markcc/test/myservice",
> > > > > > > > "config_file": "./src/aurora/myservice.aurora"
> > > > > > > > },
> > > > > > > > aliases={
> > > > > > > > "east": "east/markcc/test/myservice",
> > > > > > > > "c": "east",
> > > > > > > > "me": "markchucarroll"
> > > > > > > > },
> > > > > > > > command_defaults(command="job update",
> > > > > > > > defaults={"--batch-size": 10}
> > > > > > > > ))
> > > > > > > >
> > > > > > > >
> > > > > > > > With this configuration file, if the user ran "`aurora job
> > > create`"
> > > > > > > without
> > > > > > > > any parameters,
> > > > > > > > it would automatically be expanded to "`aurora job create
> > > > > > > > west/markcc/test/myservice ./src/aurora/myservice.aurora`".
> > > > > > > >
> > > > > > > > If a user ran "`aurora job create east`", the alias `east`
> > would
> > > be
> > > > > > > > expanded to "east/markcc/test/myservice", and the missing
> > config
> > > > file
> > > > > > > > parameter would
> > > > > > > > be instantiated using the default, to create a command-line:
> > > > > > > > "`aurora job create east/markcc/test/myservice
> > > > > > > > ./src/aurora/myservice.aurora`".
> > > > > > > >
> > > > > > > > If a user ran "`aurora job create @c/@me/test/myservice`",
> the
> > > two
> > > > > > > aliases
> > > > > > > > would
> > > > > > > > be expanded, and the omitted configuration parameter would be
> > > > added:
> > > > > > > > "`aurora job create east/markchucarroll/test/myservice
> > > > > > > > ./src/aurora/myservice.aurora`".
> > > > > > > >
> > > > > > > > If a user ran "`aurora job update`", then the `jobspec` and
> > > > > > `config_file`
> > > > > > > > parameters would
> > > > > > > > get inserted from the global defaults, and the
> > "--batch-size=10"
> > > > > would
> > > > > > be
> > > > > > > > inserted from
> > > > > > > > the command defaults for "aurora job update".
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
