> We've come across a rather strange problem where the parameters of some
> resources in PuppetDB are now empty.
>
> We have a Nagios server collecting resources from PuppetDB and we've started
> to get failures like this for one resource type:
>
> Error: Could not retrieve catalog from remote server: Error 400 on SERVER:
> Must pass host_alias to Nagios::Config::Host[hostname] on node nagiosserver
>
> The Puppet manifest that defines that resources is this, it is impossible to
> not populate host_alias:
>
> *******************************
> define nagios::host::host($host = $::fqdn, $tag = undef) {
> @@nagios::config::host { $host:
> host_alias => $host,
> address => $host,
> tag => $use_tag,
> }
> }
> *******************************
>
> If we query PuppetDB directly (redacted), there are indeed no parameters at
> all on this resource:
>
> *******************************
> # curl -H 'Accept: application/json' -X GET
> 'https://puppet:8081/v2/resources' --cacert
> /var/lib/puppet/ssl/ca/ca_crt.pem --cert
> /var/lib/puppet/ssl/certs/puppet.pem --key
> /var/lib/puppet/ssl/private_keys/puppet.pem --data-urlencode 'query=["=",
> "type", "Nagios::Config::Host"]' | jgrep "certname=hostname"
> [
> {
> "resource": "8ba4379c364b9dba9d18836ef52ce5f4f82d0468",
> "parameters": {
> },
> "title": "hostname",
> "exported": true,
> "certname": "hostname",
> "type": "Nagios::Config::Host",
> "sourceline": 27,
> "sourcefile":
> "/etc/puppet/environments/production/modules/nagios/manifests/host/host.pp",
> "tags": [
> ]
> }
> ]
> *******************************
>
> After a Puppet run and a new catalog, this resource now looks normal:
>
> *******************************
> # curl -H 'Accept: application/json' -X GET
> 'https://puppet:8081/v2/resources' --cacert
> /var/lib/puppet/ssl/ca/ca_crt.pem --cert
> /var/lib/puppet/ssl/certs/puppet.pem --key
> /var/lib/puppet/ssl/private_keys/puppet.pem --data-urlencode 'query=["=",
> "type", "Nagios::Config::Host"]' | jgrep "certname=hostname"
> [
> {
> "type": "Nagios::Config::Host",
> "sourceline": 27,
> "title": "hostname",
> "certname": "hostname",
> "resource": "8ba4379c364b9dba9d18836ef52ce5f4f82d0468",
> "parameters": {
> "address": "hostname",
> "tag": "tag",
> "host_alias": "hostname"
> },
> "exported": true,
> "sourcefile":
> "/etc/puppet/environments/production/modules/nagios/manifests/host/host.pp",
> "tags": [
> ]
> }
> ]
> *******************************
>
> These nodes do not have Puppet run on them regularly. We did upgrade from
> PuppetDB 1.0.1-1.el6.noarch to 1.3.2-1.el6.noarch about 3 weeks ago. We
> don't do any automatic report or node expiry.
>
> This started happening back on 2nd August, just halfway through the day the
> Puppet runs on the Nagios server start failing with this error. Now if I
> think back, at this time I think I had a broken Nagios module and a lot of
> manifests were failing to compile, but I fixed this and re-ran the failures,
> and everything was ok. PuppetDB only stores the last catalog, so there's no
> way a broken catalog could have stayed there, right?
>
> I've fixed this by refreshing the catalog of all nodes in PuppetDB, but I've
> got no idea how it got into this state. Any ideas?
No good idea yet, but there is something suspicious in your curl
responses - the "resource" hash, did you obfuscate this yourself on
purpose? The two hashes between the first and second requests are
identical. That hash is calculated based on the sum of the resource,
including parameters - so it seems impossible that PuppetDB arrived at
the same hash with and without parameters.
Maybe just maybe the responses were identical, and somehow PuppetDB
was not returning parameters as a fault. This might indicate some sort
of integrity problem, whereby the link to the parameters in the RDBMS
was lost somehow, although this is the first time I've heard of it
Luke. Maybe this was an upgrade schema change failure between 1.0.1
and 1.3.2? I'd have to consult what changed in the schema between
those two points to determine if this is likely however. Does the
timing of your upgrade, and the first time you saw this fault line up
with such a possibility? Remember a schema change will only occur
after a restart of PuppetDB ... (so perhaps consult your logs to see
when this happened after upgrade).
Let me at least try to replicate while I await your responses.
ken.
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscr...@googlegroups.com.
To post to this group, send email to puppet-users@googlegroups.com.
Visit this group at http://groups.google.com/group/puppet-users.
For more options, visit https://groups.google.com/groups/opt_out.
