On 7 February 2017 at 17:13, jcbollinger <john.bollin...@stjude.org> wrote:
> > > On Tuesday, January 31, 2017 at 10:04:27 AM UTC-6, David Schmitt wrote: >> >> >> The type and provider API has been the bane of my existence since I >> [started writing native resources](https://github.com/ >> DavidS/puppet-mysql-old/commit/d33c7aa10e3a4bd9e97e947c471ee3ed36e9d1e2). >> > > > Honestly, I've never thought the type / provider system was too bad. > Certainly it works pretty well for simple things. Now, I'm more of a Dev > who does Ops than an Op who Devs, but from my perspective, the biggest > problem with Puppet's types & providers has always been inadequate > documentation. The docs in this area are a bit better now than when I > started with Puppet around eight years ago, but they have always missed > presenting many relevant details about how the host Puppet instance, > resource instances, and their associated provider instances interact. > What I read here is that the current state is complex enough that a couple of books and a crack team of tech writers couldn't explain it well. > This new effort doesn't seem to be targeting any of that. > Looping back to the "lack of documentation", I see this lack grounded partially in the huge surface area that needs to be covered, since the existing T&P API is deeply entangled into the catalog compilation, and the agent transactions. This new API has two entry points that take a small number of easily explained arguments, making a full documentation realistic. From this (and other's) reaction I do understand now, that I should have started with a more README-driven approach, to highlight the difference more. Let's try this again: The *SimpleResource* (working title) consists of three distinct, but cooperating parts. The *Definition* describes the shape of a resource, its attributes, their types, which is the namevar, and the doc strings. The *get method* returns a list of hashes, each matching the shape of the resource as defined in the Definition. For example, a list of apt_key resources (here rendered in JSON) would look like this: [ { "name": "BBCB188AD7B3228BCF05BD554C0BE21B5FF054BD", "ensure": "present", "fingerprint": "BBCB188AD7B3228BCF05BD554C0BE21B5FF054BD", "long": "4C0BE21B5FF054BD", "short": "5FF054BD", "size": "2048", "type": "rsa", "created": "2013-06-07 23:55:31 +0100", "expiry": null, "expired": false }, { "name": "B71ACDE6B52658D12C3106F44AB781597254279C", "ensure": "present", "fingerprint": "B71ACDE6B52658D12C3106F44AB781597254279C", "long": "4AB781597254279C", "short": "7254279C", "size": "1024", "type": "dsa", "created": "2007-03-08 20:17:10 +0000", "expiry": null, "expired": false }, ... The *set method* takes a list of target goals in the same format and is responsible for enforcing those goals on the target system. When calling the set method, the caller may provide the current state (as received from a recent get call) to allow the set method to shortcut reading the existing state again. The basic pattern for the set method looks like this: def set(current_state, target_state, noop = false) existing_keys = Hash[current_state.collect { |k| [k[:name], k] }] target_state.each do |resource| # additional validation for this resource goes here current = existing_keys[resource[:name]] if current && resource[:ensure].to_s == 'absent' # delete the resource elsif current && resource[:ensure].to_s == 'present' # update the resource elsif !current && resource[:ensure].to_s == 'present' # create the resource end end end > Now, finally, we'll do something about it. I'm currently working on >> designing a nicer API for types and providers. My primary goals are to >> provide a smooth and simple ruby developer experience for both scripters >> and coders. Secondary goals were to eliminate server side code, and make >> puppet 4 data types available. Currently this is completely aspirational >> (i.e. no real code has been written), but early private feedback was >> encouraging. >> >> > > I'm confused and dubious: how is it that the guiding goals for this effort > are all about form rather than function? Surely, if you intend for the > result to serve any particular function(s) then you need to be guided at > least in part by which functions those are. Perhaps for you that's tied up > in what you mean by "a type and provider API", but I am confident that some > of the details of what that means to you differ from what it means to > others. You really need to be explicit with this if you want good > feedback. Moreover, inasmuch as you seem to imagine this API eventually > supplanting the existing one, it may save a lot of future grief and expense > to make sure everyone is on the same page early on. > Thanks for pointing out where things didn't make it out of my head into this text. Part of the motivation is hidden in the note around libral. There are a few connected efforts/conversations around making the providers a independent entity that can provide value in more situations than the agent transaction: * libral is addressing the local runtime environment * this proposal is more focused on the shape of the data required to "drive" a provider * remote resources (e.g. AWS, network devices) are becoming more prevalent and might be better served by something other than puppet agent. For example puppet device is a thing, but is not well integrated in the general ecosystem The underlying connection is that the functionality of providers (e.g. "how to install a package", or "how to procure a VM in AWS") have a value beyond "can be driven in a puppet agent transaction. Easy examples are "puppet resource" as a exploratory tool, and "puppet device" for remote resources. Gareth's puppetlabs-inventory , everyone hooking into PuppetDB, and by extensions efforts like augeas, libelektra, openlmi all were about providing uniform API access to heterogeneous sets of resources. > Moreover, none of the goals given are addressed the needs of the API > client, which is Puppet itself. Or do you have a different concept of who > the client is? If so, then you and I have even more different ideas of > what a "type and provider API" is than I had supposed. > As argued above, there is value to be unlocked by enabling callers of the Resource API other than puppet agent. That said, the puppet agent will in any reasonable future remain the primary driver for all of this, which also leads to the balancing act of providing a easily callable API, that still hooks up into the current infrastructure. > > Furthermore, as wonderful as "a smooth and simple ruby developer > experience for both scripters and coders" sounds, I think it's utterly > unrealistic. People who aren't good at writing native code or who just > want to avoid it have much different wants and expectations from those who > are comfortable with writing native code. In fact, Puppet already *has* > support for "scripters": this is what defined types are all about. People > who don't want to be writing Ruby are unlikely ever to have a smooth or > simple Ruby developer experience. The most likely outcome of trying to > simplify any more than serves coders well is producing something > insufficiently powerful / flexible. > In my book scripters are smart people who build upon existing code snippets (be that from existing pieces, or documentation), while coders take a more formal approach revolving around exploring a wider swatch of the surrounding infrastructure and principles. My hope is that a runtime model that can be fully explained in a few sentences (see above), and is more inspectable will help both groups to be able to focus on the real problems they are tasked to solve. The `Puppet::SimpleResource.implement()` call receives the `current_state = >> get()` and `set(current_state, target_state, noop)` methods. `get` returns >> a list of discovered resources, while `set` takes the target state and >> enforces those goals on the subject. There is only a single (ruby) object >> throughout an agent run, that can easily do caching and what ever else is >> required for a good functioning of the provider. The state descriptions >> passed around are simple lists of key/value hashes describing resources. >> This will allow the implementation wide latitude in how to organise itself >> for simplicity and efficiency. >> >> > > So there is no mechanism for obtaining the current state of an individual > resource? That seems a bit of an oversight. There are resource types that > cannot feasibly be fully enumerated (e.g. files), and even resource types > that cannot be enumerated at all. > As mentioned in the original text: "Some resources (e.g. file, ssh_authorized_keys, concat) cannot or should not be prefetched. While it might not be convenient, a provider could always return nothing on the get() and do a more customized enforce motion in the set()." This is a deliberate tradeoff now, to build a simple, robust foundation for future evolution. Adding an optional `find_by_name` function that can do this. Also, though you say elsewhere that the provider's set() method receives a > list of resources to update, that is not borne out by your example code. > The example code only shows the Resource itself. The scaffolding around it to hook it up into the existing system does not exist yet. On my last flight I wrote a very simple type and provider to forward puppet's API into calls of get/set functions as they are described here, and that worked surprisingly well, so I'm confident now that I can really implement something to dynamically create this scaffolding based on the resource definition. > The `Puppet::SimpleResource.define()` call provides a data-only >> description of the Type. This is all that is needed on the server side to >> compile a manifest. Thanks to puppet 4 data type checking, this will >> already be much more strict (with less effort) than possible with the >> current APIs, while providing more automatically readable documentation >> about the meaning of the attributes. >> >> > > I'm not at all convinced that your concept of a resource type definition > indeed covers everything needed -- or at least wanted -- on the server > side. For example, I'm all for engaging the type system, but > > 1. The type system does not cover all potential validation needs. In > particular, it cannot perform joint validation on multiple parameters. > 2. Using the type system for any but pretty simple validity checks > will run against your objective of providing a simple interface. > > The first point is true already for most existing non-data-type validations, except for the example you brought up. The second point is the reason I don't propose doing that. > Also, if handling types does not involve any server-side code, does that > mean that the type system is not engaged to validate parameters until the > data reach the client? If so, that in itself is a controversial > interpretation of what is and is not needed on the server side. > No server-side code provided by the developer of the Resource. Of course the puppet compiler will be able to cross-check the types from the define() call with whatever is passed in through the manifest. > > I'm not prepared to offer feedback on the details at this time, because I > don't think you've sufficiently established what the effort is trying to > accomplish. > > Thank you for your time and work. I really appreciate the honest and detailed feedback! David PS: I'm travelling this and next week. While it might take me a day or two to get around to answering, I will process all mails before any decision on this project is made. -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CALF7fHZXDHtap%2BNGkmwXpnFXV7kjCdXCSc_zGA%3DMtb_Vp6Hzbw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
