On 11/23/2015 06:58 AM, Jay Dobies wrote: > On 11/20/2015 07:05 PM, Ben Nemec wrote: >> Thinking about this some more makes me wonder if we need a sample config >> generator like oslo.config. It would work off something similar to the >> capabilities map, where you would say >> >> SSL: >> templates: >> -puppet/extraconfig/tls/tls-cert-inject.yaml >> output: >> -environments/enable-ssl.yaml >> >> And the tool would look at that, read all the params from >> tls-cert-inject.yaml and generate the sample env file. We'd have to be >> able to do a few new things with the params in order for this to work: >> >> -Need to specify whether a param is intended to be set as a top-level >> param, parameter_defaults (which we informally do today with the Can be >> overridden by parameter_defaults comment), or internal, to define params >> that shouldn't be exposed in the sample config and are only intended as >> an interface between templates. There wouldn't be any enforcement of >> the internal type, but Python relies on convention for its private >> members so there's precedent. :-) > > There is new functionality in Heat that will let you pass in a series of > templates and environments and it will return: > > - The list of top-level parameters, the same way template-validate > always did > - A list of all nested parameters, keyed by resource. > > Take a look at > https://github.com/openstack/heat-specs/blob/master/specs/liberty/nested-validation.rst > > for details and an example. > > That's not entirely what you're getting at, I realize that. I'm glad to > see you suggest a convention-based approach because I think that's the > only way we're going to be able to convey some of this information. > > I think at the same time we add a mechanism to distinguish between > internal and external parameters, we need to add something to indicate > required v. optional. > > With a nested stack, anything that's not part of the top-level parameter > contract is defaulted. The problem is that it loses information on what > is a valid default v. what's simply defaulted to pass validation.
I thought the nested validation spec was supposed to handle that though? To me, required vs. optional should be as simple as "Does the parameter definition have a 'default' key? If yes, then it's optional, if no, then it's required for the user to pass a value via a parameter or parameter_default". I realize we may not have been following that up to now for various reasons, but it seems like Heat is already providing a pretty explicit mechanism for marking params as required, so we ought to use it. > > I've been noticing this more and more on the vendor integrations. They > have parameters that are required (such as a username) and others that > are less likely to be changed (I can't think of an example, but I think > everyone can see where I'm going with this). > > So I think there are two sorts of things (at least, I'm also thinking > off the top of my head) we'd like this tool/sample file to convey: > > - Parameters a user would want to change, as compared to those used for > internal data shuffling > - Information on if the user must supply a value, as compared to > parameters with an actual default > > All that said, I dig this idea of a tool that would generate a skeleton > environment file. Cool, I'll try to get a spec written up at some point where we can bikeshed^W discuss the details. > >> -There would have to be some way to pick out only certain params from a >> template, since I think there are almost certainly features that are >> configured using a subset of say puppet/controller.yaml which obviously >> can't just take the params from an entire file. Although maybe this is >> an indication that we could/should refactor the templates to move some >> of these optional params into their own separate files (at this point I >> think I should take a moment to mention that this is somewhat of a brain >> dump, so I haven't thought through all of the implications yet and I'm >> not sure it all makes sense). > >> The nice thing about generating these programmatically is we would >> formalize the interface of the templates somewhat, and it would be >> easier to keep sample envs in sync with the actual implementation. > > You could go so far as to put CI on top of it like we do with the oslo > config stuff, which would be neat. > >> You'd never have to worry about someone adding a param to a file but >> forgetting to update the env (or at least it would be easy to catch and >> fix when they did, just run "tox -e genconfig"). >> >> I'm not saying this is a simple or short-term solution, but I'm curious >> what people think about setting this as a longer-term goal, because as I >> think our discussion in Tokyo exposed, we're probably going to have a >> bit of an explosion of sample envs soon and we're going to need some way >> to keep them sane. >> >> Some more comments inline. >> >> On 11/19/2015 10:16 AM, Steven Hardy wrote: >>> On Mon, Nov 16, 2015 at 08:15:48PM +0100, Giulio Fidente wrote: >>>> On 11/16/2015 04:25 PM, Steven Hardy wrote: >>>>> Hi all, >>>>> >>>>> I wanted to start some discussion re $subject, because it's been apparrent >>>>> that we have a lack of clarity on this issue (and have done ever since we >>>>> started using parameter_defaults). >>>> >>>> [...] >>>> >>>>> How do people feel about this example, and others like it, where we're >>>>> enabling common, but not mandatory functionality? >>>> >>>> At first I was thinking about something as simple as: "don't use top-level >>>> params for resources which the registry doesn't enable by default". >>>> >>>> It seems to be somewhat what we tried to do with the existing pluggable >>>> resources. >>>> >>>> Also, not to hijack the thread but I wanted to add another question related >>>> to a similar issue: >>>> >>>> Is there a reason to prefer use of parameters: instead of >>>> parameter_defaults: in the environment files? >>>> >>>> It looks to me that by defaulting to parameter_defaults: users won't need >>>> to >>>> update their environment files in case the parameter is moved from >>>> top-level >>>> into a specific nested stack so I'm inclined to prefer this. Are there >>>> reasons not to? >>> >>> The main reason is scope - if you use "parameters", you know the data flow >>> happens via the parent template (e.g overcloud-without-mergepy) and you >>> never have to worry about naming collisions outside of that template. >>> >>> But if you use parameter_defaults, all parameters values defined that way >>> are effectively global, and you then have to be much more careful that you >>> never shadow a parameter name and get an unexpected value passed in to it. >>> >>> Here's another example of why we need to decide this btw: >>> >>> https://review.openstack.org/#/c/229471/ >>> >>> Here, we have some workers parameters, going only into controller.yaml - >>> this is fine, but the new options are completely invisible to users who >>> look at the overcloud_without_mergepy parameters schema as their interface >>> (in particular I'm thinking of any UI here). >>> >>> My personal preference is to say: >>> >>> 1. Any templates which are included in the default environment (e.g >>> overcloud-resource-registry-puppet.yaml), must expose their parameters >>> via overcloud-without-mergepy.yaml >>> >>> 2. Any templates which are included in the default environment, but via a >>> "noop" implementation *may* expose their parameters provided they are >>> common and not implementation/vendor specific. >> >> This seems like a reasonable approach, although that "may" still leaves >> a lot of room for bikeshedding. ;-) >> >> It might be good to say that in this case it is "preferred" to use a >> top-level param, but if there's a reason not to then it's acceptable to >> use parameter_defaults. An example for the SSL case would be the >> certificate path - I specifically do not want that visibly exposed to >> the user at this point, so I wouldn't want it added to the top-level >> template. I consider that an implementation detail where if you know >> what you're doing you can override it, but otherwise you shouldn't touch it. >> >>> >>> 3. Any templates exposing vendor specific interfaces (e.g at least anything >>> related to the OS::TripleO::*ExtraConfig* interfaces) must not expose any >>> parameters via the top level template. >>> >>> How does this sound? >>> >>> This does mean we suffer some template bloat from (1) and (2), but it makes >>> the job of any UI or other tool requiring user input much easier, I think? >>> >>> Steve >>> >>> __________________________________________________________________________ >>> OpenStack Development Mailing List (not for usage questions) >>> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >>> >> >> >> __________________________________________________________________________ >> OpenStack Development Mailing List (not for usage questions) >> Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev >> > > __________________________________________________________________________ > OpenStack Development Mailing List (not for usage questions) > Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > __________________________________________________________________________ OpenStack Development Mailing List (not for usage questions) Unsubscribe: openstack-dev-requ...@lists.openstack.org?subject:unsubscribe http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
