Sorry, meant gradle. Back when I worked at Google, I wrote part of blaze,
so it's easy to slip and say that when I'm talking about build tools.
-Mark
On Thu, Jun 19, 2014 at 4:45 PM, Kevin Sweeney <kevi...@apache.org> wrote:
> What's blaze ;-p?
>
>
> On Thu, Jun 19, 2014 at 1:44 PM, Mark Chu-Carroll <mchucarr...@apache.org>
> wrote:
>
>> That's what I thought - but in pants/blaze/maven/..., I thought most
>> people
>> normally left their command-line in the project root. I certainly do. If
>> you're doing that - and a lot of people do - then you can't easily
>> disambiguate between different services living in the same repos.
>>
>> -Mark
>>
>>
>> On Thu, Jun 19, 2014 at 4:40 PM, Bill Farner <wfar...@apache.org> wrote:
>>
>> > On Thu, Jun 19, 2014 at 1:36 PM, Mark Chu-Carroll <
>> mchucarr...@apache.org>
>> > wrote:
>> >
>> > > Maybe I'm misunderstanding you?
>> > >
>> > > Where else should you look for a default configuration file, where it
>> can
>> > > automatically differentiate between different services stored in the
>> same
>> > > repository?
>> > >
>> >
>> > Working directory would be most intuitive to me.
>> >
>> >
>> > >
>> > > -Mark
>> > >
>> > >
>> > > On Thu, Jun 19, 2014 at 4:35 PM, Bill Farner <wfar...@apache.org>
>> wrote:
>> > >
>> > > > > in the root of the project directory
>> > > >
>> > > > FWIW i don't see any particular reason why it needs to be in the
>> root
>> > of
>> > > a
>> > > > repo/project dir.
>> > > >
>> > > > -=Bill
>> > > >
>> > > >
>> > > > On Thu, Jun 19, 2014 at 1:31 PM, Mark Chu-Carroll <
>> > > mchucarr...@apache.org>
>> > > > wrote:
>> > > >
>> > > > > I actually don't agree with standardizing the config file name.
>> > > > >
>> > > > > Here's the issue: if you're working with typical modern tools -
>> > gradle,
>> > > > > maven, pants, sbt, ..., you're doing your work in the root of a
>> > fairly
>> > > > deep
>> > > > > source hierarchy. Living in that hierarchy, there are likely
>> multiple
>> > > > > aurora job definitions.
>> > > > >
>> > > > > There's no way of just picking a default config file name that's
>> not
>> > > > going
>> > > > > to cause confusion and disruption, because the config file
>> sitting in
>> > > the
>> > > > > root of the project directory is going to be wrong for many users.
>> > > > >
>> > > > > That was the point of the home-directory fallback thing; it
>> provided
>> > > > users
>> > > > > with a way to define a default config file from outside of the
>> > > workspace,
>> > > > > so that different users could set it differently.
>> > > > >
>> > > > > In any case: I'm punting on this for now. It doesn't seem like
>> > there's
>> > > > > anything close to a consensus that this is worth doing, so I'm
>> going
>> > to
>> > > > > shelve it, and we can come back to it if the consensus changes
>> later.
>> > > > >
>> > > > > -Mark
>> > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > On Thu, Jun 19, 2014 at 4:08 PM, James Oliver <jdo...@gmail.com>
>> > > wrote:
>> > > > >
>> > > > > > +1 for standardizing the config file name
>> > > > > > On Jun 19, 2014 1:01 PM, "Bill Farner" <wfar...@apache.org>
>> wrote:
>> > > > > >
>> > > > > > > I agree with Joe, and tend to disagree with the motivation
>> (for
>> > > > > aliases,
>> > > > > > at
>> > > > > > > least). The end-result seems to be obfuscation (i'd likely
>> need
>> > to
>> > > > > peek
>> > > > > > in
>> > > > > > > the init file to figure out what a command does, or what
>> command
>> > to
>> > > > > run),
>> > > > > > > and i'm doubtful it will result in converting users away from
>> > > wrapper
>> > > > > > > scripts.
>> > > > > > >
>> > > > > > > I also feel like our configuration files are already very
>> complex
>> > > > > (nature
>> > > > > > > of the beast), and aliases make them even more so.
>> > > > > > >
>> > > > > > > I am, however, more comfortable with the standard/assumed
>> config
>> > > file
>> > > > > > > naming (i.e. like Vagrantfile).
>> > > > > > >
>> > > > > > >
>> > > > > > > -=Bill
>> > > > > > >
>> > > > > > >
>> > > > > > > On Fri, Jun 13, 2014 at 11:27 AM, Joseph Smith <
>> > > yasumo...@gmail.com>
>> > > > > > > wrote:
>> > > > > > >
>> > > > > > > > "to set up a configuration file”
>> > > > > > > >
>> > > > > > > > This line to me just says “we’re reinventing wrapper
>> scripts”
>> > > where
>> > > > > > > > someone could create an “alias” like ./east.sh and
>> ./west.sh to
>> > > do
>> > > > > > > deploys
>> > > > > > > > to various clusters. In that sense, these aliases etc are
>> also
>> > > > > “hiding”
>> > > > > > > bad
>> > > > > > > > interfaces, just in a different way- and one which isn’t as
>> > > > > > > > well-tested/understood as shell scripts.
>> > > > > > > >
>> > > > > > > > It seems like we’re set on putting time into this, so if we
>> > are,
>> > > > then
>> > > > > > > > ignore me and move onward :). IF we were going to go with an
>> > > > > approach,
>> > > > > > > this
>> > > > > > > > seems reasonable based on ways that I’ve seen people wrap
>> their
>> > > > > > > > configs/invocations of the client.
>> > > > > > > >
>> > > > > > > > On Jun 12, 2014, at 7:50 AM, Mark Chu-Carroll <
>> > > > > mchucarr...@apache.org>
>> > > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > > I'm nothing if not persistent! After feedback from my
>> second
>> > > try
>> > > > at
>> > > > > > > > > proposing a way of doing defaults and aliases, I've got a
>> > third
>> > > > > > draft.
>> > > > > > > > >
>> > > > > > > > > Feedback please!
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > # Defaults and Shorthands in the Aurora Client
>> > > > > > > > >
>> > > > > > > > > ## Motivation
>> > > > > > > > >
>> > > > > > > > > Most of the time, users are doing the same thing, over and
>> > over
>> > > > > > again.
>> > > > > > > > > They're
>> > > > > > > > > working mainly on one particular service, in one
>> particular
>> > > > > > workspace.
>> > > > > > > > But
>> > > > > > > > > they
>> > > > > > > > > need to repeat the same parameters, over and over again,
>> and
>> > > they
>> > > > > > need
>> > > > > > > to
>> > > > > > > > > remember what those parameters are, what order they occur
>> in,
>> > > > what
>> > > > > > > format
>> > > > > > > > > they
>> > > > > > > > > use, and other details that can be difficult to remember.
>> > > > > > > > >
>> > > > > > > > > In order to avoid these problems, users set up custom
>> scripts
>> > > > that
>> > > > > > make
>> > > > > > > > it
>> > > > > > > > > easy
>> > > > > > > > > for them to run commands like "`myservice start`",
>> instead of
>> > > > > > "`aurora
>> > > > > > > > job
>> > > > > > > > > create west/markcc/prod/myserver
>> > > > > > > > > src/main/aurora/myservice/myservice.aurora`"
>> > > > > > > > >
>> > > > > > > > > Scripts aren't necessarily a bad thing. We've designed the
>> > > aurora
>> > > > > > > client
>> > > > > > > > to
>> > > > > > > > > be script friendly: every command has at least an option
>> > > > supporting
>> > > > > > > > > easy-to-parse
>> > > > > > > > > output. But scripts also often cover for problems that
>> really
>> > > > > > shouldn't
>> > > > > > > > > exist.
>> > > > > > > > >
>> > > > > > > > > Scripts get written for a variety of reasons. Among the
>> many
>> > > > > reasons
>> > > > > > > that
>> > > > > > > > > people implement
>> > > > > > > > > scripts, there are scripts that add custom functionality
>> for
>> > a
>> > > > > > specific
>> > > > > > > > > group of users;
>> > > > > > > > > scripts that can cover up for inadequate or missing core
>> > > > > > functionality;
>> > > > > > > > and
>> > > > > > > > > there are
>> > > > > > > > > scripts that cover up for poor user interfaces.
>> > > > > > > > >
>> > > > > > > > > In aurora, we've worked hard to eliminate cases where the
>> > core
>> > > > > > > > > functionality of the
>> > > > > > > > > command-line client is inadequate. But our user interface
>> > still
>> > > > > > needs a
>> > > > > > > > lot
>> > > > > > > > > of work.
>> > > > > > > > > In particular, commands in Aurora often require very long
>> > > > > parameters,
>> > > > > > > > which
>> > > > > > > > > users have
>> > > > > > > > > a hard time remembering. Look at the example above - it's
>> a
>> > > > pretty
>> > > > > > > > typical
>> > > > > > > > > case. The
>> > > > > > > > > user wants to create a job. They're going to be creating
>> the
>> > > same
>> > > > > > job,
>> > > > > > > > over
>> > > > > > > > > and over again.
>> > > > > > > > > But to run it, they need to type out 68 characters for the
>> > two
>> > > > > > > > parameters!
>> > > > > > > > > This is a
>> > > > > > > > > prime example of the kind of case where users will write
>> > > scripts,
>> > > > > not
>> > > > > > > to
>> > > > > > > > > provide new/special
>> > > > > > > > > functionality, nor to cover for inadequate functionality,
>> but
>> > > > just
>> > > > > > > > because
>> > > > > > > > > it's so
>> > > > > > > > > painful to remember and correctly type out an overly long
>> > > string
>> > > > of
>> > > > > > > > > parameters.
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > To reduce this, we'd like to support a way for users to
>> set
>> > up
>> > > a
>> > > > > > > > > configuration
>> > > > > > > > > file that defines defaults and shorthands for their
>> everyday
>> > > > work.
>> > > > > > With
>> > > > > > > > > shorthands, a user that only works with a single service
>> > could
>> > > > say
>> > > > > > > > "aurora
>> > > > > > > > > job
>> > > > > > > > > create", instead of needing to spell out the full jobkey
>> and
>> > > > > > > > configuration
>> > > > > > > > > file
>> > > > > > > > > location; a user working with multiple datacenters could
>> say
>> > > > > "`aurora
>> > > > > > > job
>> > > > > > > > > create @east`" or "`aurora job create @west`" to select
>> the
>> > > > correct
>> > > > > > > > jobkey.
>> > > > > > > > >
>> > > > > > > > > ## Proposal: Aurora Init Files
>> > > > > > > > >
>> > > > > > > > > To allow users to customize shorthands, we'll provide a
>> > > > > > > > > builtin capability to allow users to provide a
>> configuration
>> > > > > > > > > file, from which their customizations will be loaded.
>> > > > > > > > > Many applications use a simple pattern to solve similar
>> > > problems.
>> > > > > > > > > Vagrant uses a file named "Vagrantfile"; when you run
>> > vagrant,
>> > > if
>> > > > > you
>> > > > > > > > > don't specifically tell the tool where to find a
>> > configuration,
>> > > > it
>> > > > > > > looks
>> > > > > > > > > in the current directory or one of its parents to find a
>> file
>> > > > named
>> > > > > > > > > "Vagrantfile".
>> > > > > > > > >
>> > > > > > > > > We'd like to follow a similar pattern, and create an
>> > > "AuroraInit"
>> > > > > > file.
>> > > > > > > > > The aurora init file is found by searching the following
>> > > > locations,
>> > > > > > in
>> > > > > > > > > order:
>> > > > > > > > >
>> > > > > > > > > * the contents of the "--init-file" parameter.
>> > > > > > > > > * if the "--init-file" parameter is unspecified, then
>> look
>> > in
>> > > > the
>> > > > > > > > current
>> > > > > > > > > directory for a file named "AuroraInit".
>> > > > > > > > > * if no "AuroraInit" file exists in the current
>> directory,
>> > > then
>> > > > > > look
>> > > > > > > in
>> > > > > > > > > the users
>> > > > > > > > > home directory for an init file named ".aurora/init".
>> > > > > > > > >
>> > > > > > > > >> **Sidebar: Why fallback to the users home directory?**
>> > > > > > > > >>
>> > > > > > > > >> Codebases are often shared between multiple projects,
>> each
>> > of
>> > > > > which
>> > > > > > > > lives
>> > > > > > > > > in
>> > > > > > > > >> a set of subdirectories within the codebase. For example,
>> > just
>> > > > > look
>> > > > > > at
>> > > > > > > > > the aurora
>> > > > > > > > >> sourcecode, which includes the aurora scheduler, the
>> aurora
>> > > > > > executor,
>> > > > > > > > > thermos,
>> > > > > > > > >> and the aurora client.
>> > > > > > > > >>
>> > > > > > > > >> If multiple services live in a codebase, then users can't
>> > put
>> > > an
>> > > > > > > > > AuroraInit file in
>> > > > > > > > >> the root directory of the codebase, because it would
>> > interfere
>> > > > > with
>> > > > > > > > other
>> > > > > > > > > users'
>> > > > > > > > >> work.
>> > > > > > > > >>
>> > > > > > > > >> The home directory fallback provides an easy way for
>> users
>> > to
>> > > > set
>> > > > > > up a
>> > > > > > > > > pointer
>> > > > > > > > >> to the correct init file, which won't be removed by
>> > operations
>> > > > > like
>> > > > > > > > > switching branches
>> > > > > > > > >> in a source repository. Users write shared init files,
>> which
>> > > are
>> > > > > > > located
>> > > > > > > > > in their projects
>> > > > > > > > >> in the codebase, and create a symbolic link from their
>> > project
>> > > > > init
>> > > > > > > file
>> > > > > > > > > to their home
>> > > > > > > > >> directory.
>> > > > > > > > >
>> > > > > > > > > ### What goes into an init file?
>> > > > > > > > >
>> > > > > > > > > We should support the following kinds of things:
>> > > > > > > > >
>> > > > > > > > > 1. Universal defaults - user-defined default settings
>> that
>> > > will
>> > > > > be
>> > > > > > > > > applied to
>> > > > > > > > > all commands. For example, if there is a default
>> config
>> > > file
>> > > > > that
>> > > > > > > > > should always
>> > > > > > > > > be used if the user doesn't specify one, that would
>> be a
>> > > > > > universal
>> > > > > > > > > default.
>> > > > > > > > > 2. Command specific defaults - users should be able to
>> > > specify
>> > > > > that
>> > > > > > > > they
>> > > > > > > > > always want
>> > > > > > > > > to use certain parameter settings for a specific
>> command.
>> > > For
>> > > > > > > > example,
>> > > > > > > > > if they
>> > > > > > > > > want to always use a default batch size of 10 for
>> > updates,
>> > > > but
>> > > > > > > don't
>> > > > > > > > > want to affect
>> > > > > > > > > batch sizes for other commands like kill, they could
>> use
>> > a
>> > > > > > command
>> > > > > > > > > specific default.
>> > > > > > > > > 3. Aliases - shorthand names for longer parameters. A
>> user
>> > > > could
>> > > > > > > > > specify shorthands
>> > > > > > > > > "east" and "west" for full jobkeys in two different
>> > > > > datacenters.
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > ### Defaults
>> > > > > > > > >
>> > > > > > > > > A default specifies a set of _bindings_. If a parameter is
>> > > > omitted
>> > > > > > > > > from a command, and there's a binding for that parameter,
>> it
>> > > will
>> > > > > be
>> > > > > > > > > automatically inserted into the command as if the user had
>> > > typed
>> > > > > it.
>> > > > > > > > > The binding is specified in the configuration file using a
>> > > Python
>> > > > > > > > > dictionary.
>> > > > > > > > >
>> > > > > > > > > For example, if the defaults included `{'jobspec':
>> > > > > > > > > 'devcluster/me/prod/service'}`,
>> > > > > > > > > then if you ran `aurora job create` without specifying any
>> > > > > > parameters,
>> > > > > > > > the
>> > > > > > > > > command-line
>> > > > > > > > > would automatically substitute
>> `devcluster/me/prod/service`
>> > for
>> > > > the
>> > > > > > > > > `jobspec` parameter.
>> > > > > > > > >
>> > > > > > > > > Defaults can be declared either globally (in which case
>> > they'll
>> > > > be
>> > > > > > > > inserted
>> > > > > > > > > as parameters
>> > > > > > > > > for all commands), or for specific commands (in which case
>> > > > they'll
>> > > > > > only
>> > > > > > > > be
>> > > > > > > > > inserted for a
>> > > > > > > > > single command).
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > ### Aliases
>> > > > > > > > >
>> > > > > > > > > An alias is a short equivalent for a parameter. When a
>> > command
>> > > > line
>> > > > > > is
>> > > > > > > > > provided by a user, aliases will be expanded inline. A
>> user
>> > > can
>> > > > > > > > > specifically
>> > > > > > > > > mark an alias for expansion by prefixing it with "@"; if
>> an
>> > > alias
>> > > > > > > > > appears on the command-line surrounded by whitespace, it
>> will
>> > > be
>> > > > > > > > > replaced even if it isn't marked with an "@".
>> > > > > > > > >
>> > > > > > > > >> **Sidebar: why not just use shell substitutions?**
>> > > > > > > > >>
>> > > > > > > > >> The @-substitution model proposed here is very similar to
>> > unix
>> > > > > > > variable
>> > > > > > > > >> substitution. So why implement the same thing all over
>> > again?
>> > > > Why
>> > > > > > not
>> > > > > > > > > just tell
>> > > > > > > > >> users to use their shell?
>> > > > > > > > >>
>> > > > > > > > >> There are several reasons.
>> > > > > > > > >>
>> > > > > > > > >> 1. All of the options and aliases that affect aurora
>> command
>> > > > > > > invocations
>> > > > > > > > > can be specified
>> > > > > > > > >> in one place: the AuroraInit file.
>> > > > > > > > >> 2. Command shell substitution is complex - the ways in
>> which
>> > > the
>> > > > > > > command
>> > > > > > > > > shell does
>> > > > > > > > >> expansion, tokenization, substitution, and parsing can
>> be
>> > > very
>> > > > > hard
>> > > > > > > to
>> > > > > > > > > follow. Adding
>> > > > > > > > >> the aurora aliases to that process just adds another
>> layer
>> > of
>> > > > > > > confusion
>> > > > > > > > > to the process.
>> > > > > > > > >> With internal alias substitution, we can avoid the
>> > confusions
>> > > > of
>> > > > > > > > > expansions and
>> > > > > > > > >> tokenization for aurora aliases.
>> > > > > > > > >> 3. We can provide better debugging and error messages
>> with
>> > our
>> > > > own
>> > > > > > > alias
>> > > > > > > > > substitution.
>> > > > > > > > >> For example, if a user specifies a job using aliases
>> like
>> > > > > "`aurora
>> > > > > > > job
>> > > > > > > > > create @c/@me/test/myservice`",
>> > > > > > > > >> we can create an error message that shows exactly what
>> they
>> > > > > typed,
>> > > > > > > and
>> > > > > > > > > how it expanded:
>> > > > > > > > >> <pre>
>> > > > > > > > >> Job "west/markcc/test/myservice" not found in
>> configuration
>> > > file
>> > > > > > > > > config.aurora.
>> > > > > > > > >> Jobkey was generated by alias expansion: original key was
>> > > > > > > > > "@c/@me/test/myservice", where:
>> > > > > > > > >> - "@c" was expanded to "west"
>> > > > > > > > >> - "@me" was expanded to "markcc"
>> > > > > > > > >> config_file parameter "config.aurora" was specified from
>> > > > > defaults.
>> > > > > > > > >> </pre>
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > ### Defining Shorthands and Defaults: Syntax and Examples
>> > > > > > > > >
>> > > > > > > > > The init file is, like aurora job configurations, a python
>> > > source
>> > > > > > file
>> > > > > > > > > using a Pystachio
>> > > > > > > > > schema. The schema is loaded, and an `Init` object is
>> > retrieved
>> > > > > from
>> > > > > > > the
>> > > > > > > > > top-level
>> > > > > > > > > variable `init` in that file.
>> > > > > > > > >
>> > > > > > > > > The pystachio schema for init files is:
>> > > > > > > > >
>> > > > > > > > > class CommandDefaults(Struct)
>> > > > > > > > > command = Required(String)
>> > > > > > > > > defaults=Map(String, String)
>> > > > > > > > >
>> > > > > > > > > class Init(Struct):
>> > > > > > > > > defaults=Map(String, String)
>> > > > > > > > > command_defaults = Optional(CommandDefaults)
>> > > > > > > > > aliases=Map(String, String)
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > For example, an init file could contain the following:
>> > > > > > > > >
>> > > > > > > > > init=Init(
>> > > > > > > > > defaults={
>> > > > > > > > > "jobspec": "west/markcc/test/myservice",
>> > > > > > > > > "config_file": "./src/aurora/myservice.aurora"
>> > > > > > > > > },
>> > > > > > > > > aliases={
>> > > > > > > > > "east": "east/markcc/test/myservice",
>> > > > > > > > > "c": "east",
>> > > > > > > > > "me": "markchucarroll"
>> > > > > > > > > },
>> > > > > > > > > command_defaults(command="job update",
>> > > > > > > > > defaults={"--batch-size": 10}
>> > > > > > > > > ))
>> > > > > > > > >
>> > > > > > > > >
>> > > > > > > > > With this configuration file, if the user ran "`aurora job
>> > > > create`"
>> > > > > > > > without
>> > > > > > > > > any parameters,
>> > > > > > > > > it would automatically be expanded to "`aurora job create
>> > > > > > > > > west/markcc/test/myservice
>> ./src/aurora/myservice.aurora`".
>> > > > > > > > >
>> > > > > > > > > If a user ran "`aurora job create east`", the alias `east`
>> > > would
>> > > > be
>> > > > > > > > > expanded to "east/markcc/test/myservice", and the missing
>> > > config
>> > > > > file
>> > > > > > > > > parameter would
>> > > > > > > > > be instantiated using the default, to create a
>> command-line:
>> > > > > > > > > "`aurora job create east/markcc/test/myservice
>> > > > > > > > > ./src/aurora/myservice.aurora`".
>> > > > > > > > >
>> > > > > > > > > If a user ran "`aurora job create @c/@me/test/myservice`",
>> > the
>> > > > two
>> > > > > > > > aliases
>> > > > > > > > > would
>> > > > > > > > > be expanded, and the omitted configuration parameter
>> would be
>> > > > > added:
>> > > > > > > > > "`aurora job create east/markchucarroll/test/myservice
>> > > > > > > > > ./src/aurora/myservice.aurora`".
>> > > > > > > > >
>> > > > > > > > > If a user ran "`aurora job update`", then the `jobspec`
>> and
>> > > > > > > `config_file`
>> > > > > > > > > parameters would
>> > > > > > > > > get inserted from the global defaults, and the
>> > > "--batch-size=10"
>> > > > > > would
>> > > > > > > be
>> > > > > > > > > inserted from
>> > > > > > > > > the command defaults for "aurora job update".
>> > > > > > > >
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
>
