On Wednesday, October 19, 2016 at 4:51:20 PM UTC-5, re-g...@wiu.edu wrote: > > > > On Tuesday, October 18, 2016 at 9:49:23 AM UTC-5, jcbollinger wrote: >> >> >> >> On Monday, October 17, 2016 at 3:15:51 PM UTC-5, re-g...@wiu.edu wrote: >>> >>> >>> >>> On Friday, October 14, 2016 at 5:08:31 PM UTC-5, re-g...@wiu.edu wrote: >>>> >>>> >>>> >>>> On Thursday, October 13, 2016 at 9:15:58 AM UTC-5, Rob Nelson wrote: >>>>> >>>>> I've worked with saz in the past and believe he would be very >>>>> receptive to PRs that address this issue, as well, if RG or anyone else >>>>> wanted to contribute them. That would be one of the better solutions to >>>>> the >>>>> problem. >>>>> >>>>> Rob Nelson >>>>> rnel...@gmail.com >>>>> >>>>> >>>>> I think I've given you a pretty good handle on the nature of the >>>>>> problem, and I'm inclined to think that a solution that addresses the >>>>>> problem at its root will take care of the whole suite of issues. >>>>>> >>>>> >>>> Lots and lots of thanks to jcbollinger. >>>> Here is what I have done as a result of this thread: >>>> >>>> Submitted an issue with saz-rsyslog: >>>> Duplicate Declaration Class[Rsyslog] with The Foreman as ENC during >>>> catalog compilation #237 >>>> https://github.com/saz/puppet-rsyslog/issues/237 >>>> >>>> Posted a follow-up question to a related thread on The Foreman mail >>>> list to determine if The Foreman also may be exhibiting a bug, or if the >>>> issue may be my configuration: >>>> Duplicate declaration error. >>>> https://groups.google.com/d/topic/foreman-users/k0JgABtszdU/discussion >>>> (my post is waiting for approval as of 22:00 GMT 10/14/2016) >>>> >>>> My current thought at this time is two points: >>>> 1. saz-rsyslog is written in such a way that declaring a subclass >>>> before the parent class causes duplicate declaration errors - but I am >>>> told >>>> the module inheritance should be addressed, and would resolve this issue. >>>> 2. The Foreman, under undetermined circumstances (maybe my >>>> misconfiguration), will randomly declare parent classes and subclass out >>>> of >>>> order - but I am told this is not an issue with puppet modules that use >>>> inheritance correctly - thus there might be an underlying bug with The >>>> Foreman because this error will not appear under modules' expected >>>> inheritance. >>>> >>>> >>> This is what was posted on The Foreman mail list group: >>> https://groups.google.com/d/topic/foreman-users/k0JgABtszdU/discussion >>> >>>> This is an issue with the format of the ENC YAML used between Foreman >>>> and Puppet, and is best fixed in the module. >>>> The list of classes is given as a hash/dictionary and so has no >>>> particular order defined - it's down to the Puppet master/server to >>>> iterate over it, and it probably does so in no particular order. It >>>> isn't under Foreman's control. >>> >>> >>> I now believe this is truly the root of my problem. >>> >>> The evidence is looking at the ENC output and the Puppet-generated >>> Catalog file on each of the master puppet servers. >>> As you can see below, the puppet-generated catalog on the >>> secondary-master puppet server has put rsyslog::client before rsyslog - >>> this causes the failure. >>> However, The Foreman ENC output is the same on both primary and >>> secondary master puppet servers. >>> >>> >> >> You have taken a step backward there. I remind you that duplicate >> declaration errors occur *during catalog building*, and that catalog >> building fails if one occurs. It is therefore impossible to compare a >> catalog from the failure case to a catalog from the success case, because >> no catalog is ever created in the failure case. >> >> > Then I was mistaken in thinking that the node file > (/var/lib/puppet/yaml/node/server1.mydomain.example.com.yaml) was the data > (catalog pre-data) written to a file before the catalog compilation > procedure. >
But you said you were looking at the *catalog* not "catalog pre-data". I'm sure you appreciate the tremendous difference between those. > I assumed this because both the foreman and node files are created new > every time server1 performs a puppet agent update, even when the result is > a 400 error for the catalog compilation. > If I am mistaken in this, then I need someone to tell me how to get the > data of the catalog that is failing to be compiled, so that I can compare > it with other sources. > > Here is the procedure I have been assuming (I am not an expert on puppet > internals): > 1. (server1)puppet-agent -> puppet-master > 2. puppet-master -> (exec)puppet[external_nodes] > server1.mydomain.example.com -> foreman ENC > 3. foreman ENC -> > /var/lib/puppet/yaml/foreman/server1.mydomain.example.com.yaml -> > puppet-master > 4. puppet-master > -> /var/lib/puppet/yaml/node/server1.mydomain.example.com.yaml -> > puppet-master(compile-catalog) > 5. puppet-master(compile-catalog) -> Error -> 400 Server Error -> > (server1)puppet-agent > > I was assuming the node file is the data of the catalog to be compiled. I > had noticed it is updated in every puppet agent run, even when the run is > in error. And it was the differing piece between primary and secondary > puppet masters. > Please correct me. > The YAML files in question are just byproducts of catalog building, not, as I understand it, intermediate files in the catalog-building process. In any event, an ENC certainly delivers its results to Puppet via its standard output, not by writing them to a regular file. Nor would I expect any ENC to create or update files under /var/lib/puppet in any case. > Please let me know how to get the data of the catalog the puppet master is > failing to compile for the puppet agent. > If you are not assigning any classes or resources directly via a site manifest, and you are not using Hiera or any other form of external data besides variables assigned via Foreman, then the facts reported by the node, the ENC output for that node, and the master's standard top-scope variables together constitute the data from which catalog building proceeds. It is possible, however, for evaluation of specific manifests to result in arbitrary data being gleaned from the catalog-building environment by use of Puppet functions, such as generate(), which can run an external command, and template() / inline_template(), which can run arbitrary Ruby code. You can get the latest facts reported by a given node via the 'puppet facts find <node_certname>' command on the master. If you're using puppetdb, then you can get them from there, too. > > >> >> >>> Is this correct, that the yaml file in /var/lib/puppet/yaml/node/ is >>> exclusively generated by Puppet based on the ENC yaml? >>> If this is correct, then this is the root of the problem. >>> >> >> >> I'm not sure what significance you're attributing to those particular >> files, but it's probably the wrong significance. Yes, to the best of my >> knowledge and understanding, Puppet generates those files based on the >> node's certificate and facts, and the associated ENC output (if any). But >> no, there is no reason to think that the root of your problem lies here. >> >> > As I said earlier, I now assume I misunderstood this as being the file > holding the data to be compiled as the catalog. > I apologize for the misunderstanding, but that I intended it as the "root" > of the catalog-compilation "problem". You are right that it is not the root > of the entire underlying problem. > Again, I need help to find the catalog data trying to be compiled, but > which fails, if this /.../node/.. file is not that data. > The starting point for classes to be applied is the combination of ENC output with the site manifest. I cannot speak to what may be in your site manifest. What file or files constitute your site manifest depends on your Puppet configuration. Some of the more likely possibilities are '<puppet root>/manifests/site.pp', '<puppet root>/environments/production/manifests/site.pp', or jointly all files of the form '<puppet root>/environments/production/manifests/*.pp'. Within the site manifest, declarations at top scope and within the node block best matching the target machine (if any match at all) are relevant. > >> >>> Is there a configuration issue causing this issue? Where would I begin >>> looking for something that may be causing this puppet-catalog problem? >>> >> >> >> There is an unsupported-use-of-module issue ultimately causing all your >> problems, as I have been saying since the beginning. It would be best to >> solve *that* problem, either by choosing a different module or by >> restricting yourself to supported uses of the module you have chosen (e.g. >> by using automated data binding to assign values to class parameters). >> >> > "restricting yourself to supported uses of the module" > So using The Foreman as ENC has not been an official tested use of the > module and thus is not supported? This is true for all modules on forge I > have reviewed, except for the ones published by The Foreman group. > To be supported, should I instead restrict my use of either declaring my > classes in a site.pp or use hiera? > Puppet supports declaring classes via an ENC generally. Inasmuch as its implementation violates the DSL's documented constraints, however, the ' rsyslog::client' class is not supported at all in the sense of the term I was using. Since avoiding that class is not a viable solution (unless by choosing an altogether different module), the next best thing would be to stick to uses that are well known to work despite not actually being supported -- in this case, that would be to avoid using (the ENC analog of) a resource-like declaration for the base class, rsyslog. You have exactly one option for customizing that class's parameters without modifying it or using a resource-like declaration: automated data binding. > > "using automated data binding to assign values to class parameters" > What is automated data binding? Is that what I get when using hiera? > It is documented here: https://docs.puppet.com/hiera/3.2/puppet.html#automatic-parameter-lookup. It is based on Hiera, but it would not be quite correct to say it's "what I get when using hiera", because Hiera supports other uses as well. > Would that mean I would forgo the use of The Forman to instead use hiera? > It would mean that you forgo using The Foreman for setting the parameters of class rsyslog, at least for those nodes to which you also assign another class from the same module, such as rsyslog::client. You should otherwise be able to continue using The Foreman as you currently do. > I have been assuming hiera was relatively the same as using The Foreman as > ENC? Reference: > https://ask.puppet.com/question/527/are-hiera-and-foreman-parameters-mutually-exclusive/ > > <https://www.google.com/url?q=https%3A%2F%2Fask.puppet.com%2Fquestion%2F527%2Fare-hiera-and-foreman-parameters-mutually-exclusive%2F&sa=D&sntz=1&usg=AFQjCNFi7mdJ0ZSw80VSK1NGgNAFWZuVTA> > > No, they are not the same thing at all, though they can serve many of the same purposes (and the answer you linked says pretty much the same thing). Hiera is not an ENC, and Puppet does not use it in the same way or via the same interface that it uses an ENC. Binding values to class parameters via (hiera-supported) automatic data binding has different semantics than does binding parameter values via an ENC. From a Foreman perspective, you might view automated data binding as affecting the default values for class parameters. > I am grasping at straws. I have been assuming what I can find different > between the servers would reveal the issue. > Indeed, the problems are mysterious, and it is entirely reasonable to suppose that differing behavior arises from discoverable differences between machines. Myself, I had high hopes for an explanation in terms of differing versions of Ruby. I guess there's still a chance that a difference in the installed Ruby modules might factor in. And I wasn't kidding when I suggested replacing the machine on which the misbehavior is observed with a clone of one of those that behave as you want. However, I also hold out some suspicion that the problem may not be in the machine at all, but rather in the inputs (ENC output varying over time; differences in other parts of the ENC output; node facts). Or the difference may be in the dynamic operating environment, including Puppet runtime state. I'm not sure how to test that directly, but it might be interesting to swap masters to see whether the problem sticks with the machine or with the workload. > I guess I need someone to tell me that if a puppet module is having the > issue we describe in this thread, that due to the intended and expected > nature of some function/process "X" in The Foreman or Puppet, I can expect > to see the module's inheritance issue cause catalog-compilation errors at > random times. > > > Example: it was suggested that the hash keys are not always ordered the > same. So due to python sometimes ordering the hash keys differently, a > module with inheritance issues like saz-rsyslog will sometimes cause a > catalog compilation error, and sometimes not be an issue. This means Puppet > expects modules to not have an inheritance issue like we are describing in > order to guarantee there will not be issues during deployment, like in my > case. > Well, the problem with hashes is that hash iteration order is not necessarily predictable in advance. However, given the same hash implementation, initialized the same way, and subjected to the same sequence of additions and removals, you should see the same iteration order on every trial. On the other hand, if the sequence of additions and removals changes, and especially if more, fewer, or different keys are added in different trials, then you cannot rely on iteration order to be consistent, even in part. That's one reason why I earlier pointed out that the problem does not have to be in the portion of the ENC output that pertains directly to the rsyslog classes. So no, I cannot tell you that with everything remaining the same, you should expect that Puppet's catalog building behavior might vary from run to run. Quite the opposite. But on the other hand, you can never have *everything* exactly the same. I sympathize with you in your difficulty discovering the specific difference(s) that explain the variation in outcome, and some of my suggestions have been aimed at helping you there. But you should not assume that figuring it out will lead you to a solution you like better than those I have proposed. It might do. It also might not. And I cannot evaluate for you whether it's worthwhile to keep looking. John -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/f5e35c10-df99-4f38-8a89-d71b925a2cad%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
