Re: [Puppet Users] Re: Puppetserver performance plummeting a few hours after startup
Hi Martijn,
for the small amount of nodes you're managing, you shouldn't need a
separate DB server.
I'd change max-active-instances to 4, that should be plenty for your
environment.
Since your problems clearly are not coming from too much code in memory,
I'd look at other possible causes:
- Check if you're managing very large amounts of file resources
(especially if you use *recurse=>true* anywhere), this can be a cause of
issues
- Check if you're managing the content of a file resource in a bad way
somewhere, e.g. I've seen this really bad idea, which will cause the entire
file content to end up in the catalog:
file { 'huge_file.exe':
content => file('my_module/huge_file.exe')
ensure => present
}
- Check if you're using the puppet server as a fileserver for large
files, this will cause excessive CPU usage for calculating the MD5 hash of
the file each puppet run:
file { 'huge_file.exe':
source => 'puppet:///modules/my_module/huge_file.exe'
ensure => present
}
Kind regards,
*Kevin Reeuwijk*
Principal Sales Engineer, NEMEA
mobile: +31 6 272 33 55 1
kev...@puppet.com | @KevinReeuwijk
*Puppet <http://www.puppet.com>. The shortest path to better software.*
On Mon, Feb 10, 2020 at 10:44 AM Martijn Grendelman
wrote:
> Hi Kevin and others who have responded,
>
> Thanks all for your tips. Unfortunately, no breakthroughs yet.
>
> The current state is this:
>
>- Both Puppetservers typically run at the latest version, currently
>both 6.8.0.
>- The primary server has 8 virtual cores and 12 GB of physical
>(virtualized) RAM, Java is running with -Xms6g -Xmx6g.
>- Max-active-instances is currently set to 7.
>- This morning, I added -XX:ReservedCodeCacheSize=1g to the JVM
>startup config.
>- The size of our 'environments' directory is 131 MB. We currently
>have 3 environments.
>
> I've been looking at JVM stats with 'jstat', and the server doesn't appear
> to spend any significant amount of time doing GC (seems to be about 1%).
>
> After a server restart, compilation times typically drop to 9 seconds on
> average (on the secondary server, it's 5 seconds consistently), but after a
> while, they go back to 30 or 40 seconds.
>
> As I noted in my first post, our server has an average of less than 2
> concurrent agents talking to it, so I can't imagine this happening due to
> lack of resources. The fact that our secondary server handles a bigger load
> than the primary, with a third of the memory and only 2 cores, seems to
> confirm this.
>
> So:
> - enough CPU power (I would think)
> - enough memory
> - no significant garbage collection
> - Puppetserver causing a load of 5
>
> Any more tips? Would it make sense to run PuppetDB and PostgreSQL on a
> different VM?
>
> Thanks,
> Martijn Grendelman.
>
>
>
>
>
>
>
>
>
>
> Op 6-2-2020 om 17:43 schreef KevinR:
>
> Hi Martijn,
>
> it sounds like you have a sub-optimal combination of:
>
>- The amount of JRubies
>- The total amount of java heap memory for puppetserver
>- The size of your code base
>
> This typically causes the kind of problems you're experiencing. What's
> happening in a nutshell is that puppet is loading so much code in memory
> that is starts running out of it and starts performing garbage collection
> more and more aggressively. At the end, 95% of all cpu cycles are spent on
> garbage collection and you don't have any cpu cycles left over to actually
> do work like compile catalogs...
>
> To understand how Puppet loads code into memory:
>
> Your code base is: ( [ size of your control-repo ] + [ size of all the
> modules from the Puppetfile ] ) x [ the amount of puppet code
> environments]
> So let's say:
>
>- your control repo is 5MB in size
>- all modules together are 95MB in size
>- you have 4 code environments: development, testing, acceptance and
>production
>
> That's 100MB of code to load in memory, per environment. For 4
> environments, that's 400MB.
> A different way to get this amount directly is to run *du -h
> /etc/puppetlabs/code/environments* on the puppet master and look at the
> size reported for */etc/puppetlabs/code/environments*
>
> Now every JRuby will load that entire code base into memory. So if you
> have 4 JRubies, that's 1600MB of java heap memory that's actually needed.
> You can imagine what problems will happen if there isn't this much heap
> memory configured...
>
> If you're using the defaults, Puppet will create the same amount of
> JRubies as
Re: [Puppet Users] Re: puppet bolt templates ??
If you target the remote node (that doesn't have a Puppet agent) with an apply() block in a Bolt plan, the prereq apply_prep($nodes) step will install the Puppet agent binaries on that node for you. This allows you to still leverage all the functionality, without activating the agent. Kind regards, *Kevin Reeuwijk* Principal Sales Engineer, NEMEA mobile: +31 6 272 33 55 1 kev...@puppet.com | @KevinReeuwijk *Puppet <http://www.puppet.com>. The shortest path to better software.* On Tue, Feb 11, 2020 at 4:33 PM Alex Dreyer wrote: > > > On Tue, Feb 11, 2020 at 7:24 AM Shirish Shukla > wrote: > >> What about if puppet agent not installed on target >> Is there any way we can use epp file >> > > There are a few options > - You can target localhost for the apply to create the file locally and > then use upload_file to copy it. > - You can write a task that accepts the contents and path as a parameter > and writes the file to disk. > > >> >> On Monday, 16 September 2019 21:51:05 UTC+5:30, Andy Hall wrote: >>> >>> hey there just starting using bolt and has a simple plan which applies a >>> manifest but I'd know like to write a dynamic file from a template like epp >>> in puppet. is this possible ? I do not see anything in the docs except just >>> uploading a static file : >>> https://puppet.com/docs/bolt/latest/running_bolt_commands.html#concept-6839 >>> >>> please do let me know if this is possible - it's great being able to >>> write a manifest and templates would be the icing on the cake ! >>> >>> thanks. >>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "Puppet Users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to puppet-users+unsubscr...@googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/puppet-users/d09a4037-518a-4ecf-bfd1-e729485346fd%40googlegroups.com >> <https://groups.google.com/d/msgid/puppet-users/d09a4037-518a-4ecf-bfd1-e729485346fd%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- > You received this message because you are subscribed to a topic in the > Google Groups "Puppet Users" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/puppet-users/5HOpJEVhqOQ/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > puppet-users+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/puppet-users/CAMahoJtpASY9cwER1sDCWc1EcyYwKaDccJ8Yr200yJYESAbLBw%40mail.gmail.com > <https://groups.google.com/d/msgid/puppet-users/CAMahoJtpASY9cwER1sDCWc1EcyYwKaDccJ8Yr200yJYESAbLBw%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/CAPqNtk_2b5V8BBgOad3aXEhC-2ghU9gyU3OJLhR_LQpotdA_ZQ%40mail.gmail.com.
Re: [Puppet Users] Re: Calling a puppet function inside a ruby task
Hi Raghuram,
The short of it is that you can't make that work in a Task, the Bolt Task
environment is simply not designed for this. You have to use a plan.
You can have the Plan call the function and pass the results as parameters
to your Task though:
plan mymodule::test {
$func_result = mymodule::func1('arg1', 'arg2')
# assuming mymodule::func1() returns a simple string
run_task('mymodule::mytask', 'target.company.com', 'myparam' => $func_result
)
}
-KevinR
On Thu, Feb 4, 2021 at 12:45 PM Raghu Ram Baisani
wrote:
> Thanks Kevin
>
> I'm kind of aware of plan option.
>
> Can you please detail the option for task with an example so that I can
> know how it works.
>
> Cheers
> Raghuram Baisani
>
> On Thu, Feb 4, 2021 at 5:12 PM KevinR wrote:
>
>> Hi Raghuram,
>>
>> this cannot be (easily) achieved from a Task, as the Task execution
>> environment is not Puppet but basically the target's shell.
>> You can do this from a Bolt Plan however:
>>
>>1. In your Bolt module (where your task lives), create a functions
>>folder at the root
>>2. Place your function in that folder, i.e.f /functions/func1.rb
>>3. Namespace the function according to your module name, for example
>>mymodule::func1
>>4. You can now use the module in a Bolt Plan (under /plans) with
>>mymodule::func1(args)
>>
>> - KevinR
>> On Thursday, February 4, 2021 at 7:53:46 AM UTC+1 baisani...@gmail.com
>> wrote:
>>
>>> Hi
>>>
>>> Suppose if we have a module named temp,
>>>
>>> and we have a puppet function named func1
>>> which resides in temp\lib\puppet\functions\temp\func1.rb
>>>
>>> and I have another ruby task in the same module,
>>>
>>> temp\tasks\task1.rb
>>>
>>> How can I call puppet function func1 on task task1?
>>>
>>> Any kind help is appreciated
>>>
>>> Cheers
>>> Raghuram Baisani
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Puppet Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to puppet-users+unsubscr...@googlegroups.com.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/puppet-users/c6d2d365-3cc8-4513-b4c3-bb0f062d56d5n%40googlegroups.com
>>
>> .
>>
> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Puppet Users" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/puppet-users/psFXomJq4tY/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> puppet-users+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/puppet-users/CAFfXrBxmXFxLMkZUaGr1bOaMbOjLaS4TNwZdzrAQPqpnmvoJDg%40mail.gmail.com
>
> .
>
--
You received this message because you are subscribed to the Google Groups
"Puppet Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to puppet-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/puppet-users/CAPqNtk8e%2BZrirF_pT7Lk439-qKto7TxN0Aw8ZFSFxuVvUVcZ_g%40mail.gmail.com.
