[Puppet Users] PuppetDB API permissions

2012-10-26 Thread ak0ska
Hello, Is it possible to control from which nodes is it allowed to execute commands like "replace catalog" and "replace facts", and which nodes can only do queries (but no changes)? It seems like once someone could access the service through http or https (depending on jetty.ini settings) can d

[Puppet Users] Re: PuppetDB API permissions

2012-11-07 Thread ak0ska
s wrote: > > On Friday, October 26, 2012 7:24:18 AM UTC-7, ak0ska wrote: > >> Hello, >> >> Is it possible to control from which nodes is it allowed to execute >> commands like "replace catalog" and "replace facts", and which nodes can >>

[Puppet Users] PuppetDB: gather nodes changed recently

2012-11-08 Thread ak0ska
Hello, Is it possible to (efficiently) gather those nodes from PuppetDB which have changed recently? Either their facts or their catalogues. In case we want to do something with the nodes that have changed, and quickly collect them using PuppetDB. If I understood correctly, Puppetdb checks if t

[Puppet Users] Re: PuppetDB: gather nodes changed recently

2012-11-09 Thread ak0ska
Thanks for the info! Sounds nice, looking forward to it! :) On Thursday, November 8, 2012 6:24:15 PM UTC+1, Chris Price wrote: > > ak0ska, > > You are correct--right now the timestamps we return only indicate the last > submission of catalogs/facts. We don't actually

[Puppet Users] Question about PuppetDB queries

2012-11-19 Thread ak0ska
o PuppetDB. Especially the second one looks weird, and it also makes our logfile grow fast. Example for type 1 : http://pastebin.com/exNG2bCj Example for type 2 (this is actually one single entry in the log) : http://pastebin.com/ym9AHPSk Thanks in advance! Cheers, ak0ska -- You received th

[Puppet Users] PuppetDB HA

2013-01-22 Thread ak0ska
Hello, I would like to ask what to pay attention to, when someone wants to set up multiple PuppetDB instances (and point them to the same DB), and put them behind a proxy/load balancer, as mentioned in the documentation

Re: [Puppet Users] PuppetDB HA

2013-01-23 Thread ak0ska
I assume that each PuppetDB instance maintains its own message queue, and commands sent by the master wait in these queues. In that case, is the following scenario possible: - the master sends facts for a node to PuppetDB through the load balancer - the load balancer gives the task to PuppetDB1 -

[Puppet Users] PuppetDB queue problem

2013-02-28 Thread ak0ska
Hello, We have upgraded Puppetdb from 1.0.2 to 1.1.1 last week, and recently we noticed that queue is not processed. We don't know if this problem is directly linked to the upgrade. There are now 25 000 messages waiting in the queue. We also see errors in the Postgres log: ERROR: insert or upd

[Puppet Users] Re: PuppetDB queue problem

2013-02-28 Thread ak0ska
We flushed the queue and also restarted Postgres. Didn't help. On Thursday, February 28, 2013 10:14:16 AM UTC+1, ak0ska wrote: > > Hello, > > We have upgraded Puppetdb from 1.0.2 to 1.1.1 last week, and recently we > noticed that queue is not processed. We don'

[Puppet Users] Re: PuppetDB queue problem

2013-02-28 Thread ak0ska
where else to look for a problem in a similar case On Thursday, February 28, 2013 10:38:42 AM UTC+1, ak0ska wrote: > > We flushed the queue and also restarted Postgres. Didn't help. > > On Thursday, February 28, 2013 10:14:16 AM UTC+1, ak0ska wrote: >> >> Hello, >>

[Puppet Users] Re: PuppetDB queue problem

2013-02-28 Thread ak0ska
http://pastebin.com/2B4azRib On Thursday, February 28, 2013 1:09:15 PM UTC+1, ak0ska wrote: > > Or maybe it did. The queue size went up to like 900 and after a while it > normalized and started slowly decreasing. Right now it's empty. > There were 2 insert errors in postgres log since its

Re: [Puppet Users] Re: PuppetDB queue problem

2013-02-28 Thread ak0ska
Hi, thanks for trying to help! :) If you clear the queue and rollback to the original version does the > problem disappear? If you're having processing problems at the latest > version thats what I would do, as I presume we're talking production > here right? > > Yes, this is production. We wo

Re: [Puppet Users] Re: PuppetDB queue problem

2013-02-28 Thread ak0ska
> I'm @ken_barber on irc btw if that is easier. > Can't use IRC here, sorry. :( > Okay. Did you clear the ActiveMQ queues after doing this? I usually > just move the old KahaDB directory out of the way when I do this. > I haven't though about myself, but it makes sense, so I just flushe

Re: [Puppet Users] Re: PuppetDB queue problem

2013-03-01 Thread ak0ska
> You mean, you've only been watching it for a few minutes, and so far > so good - or it crashed? Sorry - just want to be clear :-). > I was watching it for a few minutes and it seemed good. However the queue grew up to 4000 items overnight. Also we have more of the constraint violation err

Re: [Puppet Users] Re: PuppetDB queue problem

2013-03-01 Thread ak0ska
KahaDB.old2 1.1G./scheduler 6.2G./KahaDB.old The big one is a 25 000 queue. I would happily send the logs over, but this is something I have discuss with my colleagues. Again, thank you for helping! :) ak0ska -- You received this message because you are subscribed to the Google Groups &

Re: [Puppet Users] Re: PuppetDB queue problem

2013-03-01 Thread ak0ska
> > But I'm guessing you can't stay broken for very long - so I would take > > a snapshot of the current broken state so we can take a look and try > > to replicate it, and make moves on getting back to a working state as > > suggested above. > > > > I&#

Re: [Puppet Users] Re: PuppetDB queue problem

2013-03-01 Thread ak0ska
Yes, maybe not. The next step will be to recreate it from scratch. On Friday, March 1, 2013 5:47:06 PM UTC+1, Ken Barber wrote: > > > Well, I don't think a vacuum will help you - I imagine something is > wrong with the schema right now or some data migration failed during > upgrade. Esp. if you

Re: [Puppet Users] Re: PuppetDB queue problem

2013-03-04 Thread ak0ska
ted, but that cannot be done before the next weekend. Perhaps we should rebuild the indexes first, and see if it further improves the situation. It's only been 1 day since the restart, but so far it looks a lot better then before. Cheers, ak0ska On Monday, March 4, 2013 10:55:37 PM UTC+1,

Re: [Puppet Users] Re: PuppetDB queue problem

2013-03-05 Thread ak0ska
Hey Mike, Thanks for the suggestions, but we already checked the IO rates, and they seemed fine. And yes, PuppetDB and Postgres are on the same machine for now, but we plan to change that sometime in the future. Cheers, ak0ska On Tuesday, March 5, 2013 12:51:16 AM UTC+1, Mike wrote: > &

Re: [Puppet Users] Re: PuppetDB queue problem

2013-03-06 Thread ak0ska
@Mike iostat -nx Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s avgrq-sz avgqu-sz await svctm %util md0 0.00 0.00 85.55 405.31 9226.53 3234.60 25.39 0.000.00 0.00 0.00 @Ken Wow. Thats still way too large for the amount of nodes. I i

Re: [Puppet Users] Re: PuppetDB queue problem

2013-03-07 Thread ak0ska
On Thursday, March 7, 2013 12:23:13 AM UTC+1, Ken Barber wrote: > > > > So the index 'idx_catalog_resources_tags' was removed in 1.1.0 I > think, so that is no longer needed. > > This points back to making sure you schema matches exactly what a > known good 1.1.1 has, as things have been misse

[Puppet Users] Re: PuppetDB queue problem

2013-03-11 Thread ak0ska
After dropping the obsolete index, and rebuilding the others, the database is now ~ 30 GB. We still get the constraint violation errors when garbage collection starts. Also the "Resources" and "Resource duplication" values on the dashboard are still question marks, so those queries probably time

Re: [Puppet Users] Re: PuppetDB queue problem

2013-03-12 Thread ak0ska
I think my previous comment just got lost. So, I cut three occurrences of this error from the database log and the corresponding part from the puppetdb log. I removed the hostnames, I hope it's still sensible: http://pastebin.com/yvyBDWQE The unversioned api warnings are not from the masters. Th

Re: [Puppet Users] Re: PuppetDB queue problem

2013-03-13 Thread ak0ska
L) Total runtime: 474370.129 ms (13 rows) > And finally, can you give us a dump of your database settings? You can do > this from within psql by doing "SHOW ALL;" > http://pastebin.com/raw.php?i=HfhqfVa4 Thank you for your help! Cheers, ak0ska -- You received this m

Re: [Puppet Users] Re: PuppetDB queue problem

2013-03-14 Thread ak0ska
Hello Ken, I really appreciate you guys looking into this problem, and I'm happy to provide you with the data you ask for. However, I feel like I should ask, whether you think this problem is worth your efforts, if rebuilding the database might solve the issue? Cheers, ak0ska On Thu

[Puppet Users] Low catalog duplication rate

2013-07-12 Thread ak0ska
Hello, We have some performance issues with PuppetDB and I believe our low catalog duplication rate is partly responsible (7.5% atm). I would like to understand this problem better, and ask what's the best way to track down catalogs that change too often. I assumed the catalog hash (in certnam

[Puppet Users] Re: Low catalog duplication rate

2013-10-09 Thread ak0ska
n puppetdb? Cheers, ak0ska On Friday, July 12, 2013 1:30:20 PM UTC+2, ak0ska wrote: > > Hello, > > We have some performance issues with PuppetDB and I believe our low > catalog duplication rate is partly responsible (7.5% atm). I would like to > understand this problem better, an

[Puppet Users] PuppetDB performance tests

2013-10-24 Thread ak0ska
with the live communication between the production PuppetDB and the masters? Suggestions are welcome! :) Thanks, ak0ska -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it

Re: [Puppet Users] PuppetDB performance tests

2013-10-25 Thread ak0ska
he README.md file for more details. I would like to > find the time to make this more performant and able to be used in a > production like environment but for now this is only recommended in > test, or for short periods of time in production. > > ken. > > On Thu, Oct 24, 2

Re: [Puppet Users] Help with scaling puppetdb/postgres

2013-10-29 Thread ak0ska
Just out of curiosity, what is your catalog duplication rate? On Tuesday, October 29, 2013 3:26:20 AM UTC+1, David Mesler wrote: > > I reconfigured postgres based on the recommendations from pgtune and your > document. I still had a lot of agent timeouts and eventually after running > overnight

Re: [Puppet Users] Help with scaling puppetdb/postgres

2013-10-30 Thread ak0ska
Also looking at the reports (Foreman, PuppetDB) might give a clue of what is changing. On Tuesday, October 29, 2013 7:32:54 PM UTC+1, Ryan Senior wrote: > > 1.5% catalog duplication is really low and from a PuppetDB perspective, > means a lot more database I/O. I think that probably explains th

[Puppet Users] Subquery, regular query difference?

2014-01-20 Thread ak0ska
Hello, We wanted to created a query, which would check the hostgroup fact of machines that apply a certain type of resource declared in a certain manifest file. We wanted to use the 'select-resources' subquery as follows: curl -G -H "Accept: application/json" localhost:8080/v3/facts --data-url

[Puppet Users] Re: Subquery, regular query difference?

2014-01-20 Thread ak0ska
Forgot to mention that it's a PuppetDB issue. ;) On Monday, January 20, 2014 4:17:39 PM UTC+1, ak0ska wrote: > > Hello, > > We wanted to created a query, which would check the hostgroup fact of > machines that apply a certain type of resource declared in a certain > mani

Re: [Puppet Users] Re: Subquery, regular query difference?

2014-01-22 Thread ak0ska
On Monday, January 20, 2014 9:17:39 AM UTC-6, ak0ska wrote: > >> > >> Hello, > >> > >> We wanted to created a query, which would check the hostgroup fact of > >> machines that apply a certain type of resource declared in a certain > >> manif

[Puppet Users] alias property flip-flop for concat files

2014-04-03 Thread ak0ska
om/puppetlabs/puppetlabs-concat/blob/1.1.0-rc1/manifests/init.pp#L174 The same is happening to the concat fragments. Any ideas why and how this can happen? Cheers, ak0ska -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsu

[Puppet Users] PDB 1.6: Queries filtering facts with large values might fail

2014-10-15 Thread ak0ska
Hello, We are running PuppetDB 1.6.0. We have fact 'a' in puppetdb that has large numbers occasionally, such as 2930266584. When we launch a query /v3/facts/a with filter '["<", "value", 1950341121]' it returns code 200 with an empty body, while puppetdb.log shows a new error: 2014-10-15 11:20

Re: [Puppet Users] PDB 1.6: Queries filtering facts with large values might fail

2014-10-15 Thread ak0ska
Thanks for the explanation! On Wednesday, October 15, 2014 1:36:49 PM UTC+2, Ken Barber wrote: > > > We are running PuppetDB 1.6.0. We have fact 'a' in puppetdb that has > large > > numbers occasionally, such as 2930266584. When we launch a query > /v3/facts/a > > with filter '["<", "value", 1