Hi Chris, Quite a similar question was posted about two weeks back, you might find that very interesting:
https://groups.google.com/forum/#!topic/puppet-users/NW2yuHMJvsY On Thursday, 1 August 2019 17:01:44 UTC+1, Chris Southall wrote: > > Our site is using a collection of puppet modules to manage various Linux > components using the roles and profiles model. While it works OK for the > most part, I often find it necessary to update a module or profile for some > reason or other. Modules obtained from puppet forge sometimes don't quite > do what is needed, and writing good quality modules on your own can be a > challenge. > There was another recent post about using Forge modules or importing the Puppet code into a personal Git repository directly: https://groups.google.com/forum/#!topic/puppet-users/vcp-pVYC8b0 If you are a confident Puppet Coder, you might prefer to import the source, patch the module to add your feature, then submit the patch back upstream. > When using roles and profiles you end up declaring all the module > parameters again to avoid losing functionality and flexibility. > ... Not sure I agree with that statement. That sounds odd. Why would you be re-declaring module parameters if you're not changing something from the defaults? And if you are intending to change something, then of course you are supplying different parameters? > You also need to be familiar with all the classes, types, and parameters > from all modules in order to use them effectively. > Ideally the README page of a module would contain amazing user level documentation of how the module should work... but not that many do. I often find I have to go read the Puppet code itself to figure out exactly what a parameter does. > To avoid all of the above, I put together the 'basic' module and posted it > on the forge: https://forge.puppet.com/southalc/basic > Ok :-) I'm beginning to see what the core of your problem is. The fact that you've created your own module to effectively do create_resources() hash definitions says to me that you haven't quite grasped the concepts of the Role / Profile design pattern. I know I have a very strong view on this subject and many others will disagree, but personally I think the Role / Profile pattern and the "do-everything-with-Hiera-data" pattern are practically incompatible. This module uses the hiera_hash/create_resources model for all the native > puppet (version 5.5) types, using module parameters that match the type > (exceptions for metaparameters, per the README). The module also includes > the 'file_line' type from puppetlabs/stdlib, the 'archive' type from > puppet/archive, and the local defined type 'binary', which together provide > a simple and powerful way to create complex configurations from hiera. All > module parameters default to an empty hash and also have a merge strategy > of 'hash' to enable a great deal of flexibility. With this approach I've > found it possible to replace many single purpose modules it's much faster > and easier to get the results I'm looking for. > A Hiera-based, data-driven approach will always be faster to produce a "new" result (just like writing Ansible YAML is faster to produce than Puppet code)... It's very easy to brain dump configuration into YAML and have it work, and that's efficient up to a certain point. For your simple use cases, yes, I can completely see why you would be looking at the Role Profile pattern and saying to yourself "WTF for?". I think the tipping point of which design method becomes more efficient directly relates to how complicated (or how much control) you want over your systems. The more complicated you go, the more I think you will find that Hiera just doesn't quite cut it. Hiera is a key value store. You can start using some neat tricks like hash merging, you can look up other keys to de-duplicate data... When you start to model more and more complicated infrastructure, I think you will find that you don't have enough power in Hiera to describe what you want to describe, and that you need an imperative programming language (eg: if statements, loops, map-reduce). The Puppet DSL is imperative. Yes, the hiera data can become quite large, but I find it much easier to > manage data in hiera than coding modules with associated logic, parameters, > templates, etc. Is this suitable for hyper-scale deployment? Maybe not, > but for a few hundred servers with a few dozen configuration variants it > seems to work nicely. Is everyone else using puppet actually happy with > the roles/profiles method? > If you are only making small-to-medium changes to a standard operating system, and/or your machines are short-lived cloud systems that get thrown away after half an hour, then sure, a Hiera-only approach will work fine at the scale you are suggesting. I also think team size and composition is a big factor. If I was in a team of one or two people I'm sure I'd be saying "Yeah! Hiera! I can change anything really really easily!". If I was in a team of a dozen engineers geographically spread across the world with vastly different levels of Puppet knowledge I think I'd be saying "Oh god... Everything's in Hiera... It's so easy for someone to mess up. What on earth has someone changed now". If you haven't guessed already, I've been here before. Personally I think the most useful part of the Role Profile design pattern is the encapsulation of implementation details behind business-specific Profiles. Jesus, what a mouthful. How about "hiding away all the details behind an interface that makes sense to me and my team"? Best demonstrated with a real life example we use here... https://gist.github.com/lukebigum/1a8b9735f604493da38d7b1d924ba4bb The above is the Profile for an LMAX "Statistics Collection Server". A statistics collection server collects statistics. If someone wants to collect statistics, all they have to do is put: include ::profile::statistics::collection_server Somewhere in a node definition and set *AT MOST* nine Hiera parameters for that Profile. That's the real win - an LMAX statistics collection server has only 9 parameters that can be changed. They don't really have to understand exactly what goes into building a Statistics Collection Server if they don't want to (in practice they might need to browse the code to check what a parameter does though, because we are lazy and don't document our Profiles). If you go read that profile in detail you'll see I pull in several component modules: Puppetlabs Apache, Influxdb, a private LVM module that's a wrapper for Puppetlabs' LVM, Grafana, and Chronograf. Apache (with SSL) is set up to proxy Grafana and Chronograf. Our LVM module creates the file system underneath Influx before is installed. Most of the parameters to the component modules are hard coded, and this is a great thing because that means every single one of our Statistics Collection Servers are exactly the same. I even pull in a (private) Nagios module to define monitoring resources, so when one of my Engineers uses that profile they get the monitoring _automatically_. I count 81 parameters to component modules in that Profile, so that would be at least 81 lines of Hiera needed to reproduce that functionality in YAML (and even then, good luck ensuring that the LVM disk is there before Influx is installed). I have condensed that to 9 possible parameters where I think someone should legitimately change something. Otherwise, you use my defaults, and that keeps things the same, reducing entropy across our estate. Yes, writing this profile took a lot longer than doing it in YAML, but our engineers shouldn't need to "figure out" how to build an InfluxDB server ever again. Another big win for me: testing. I can write puppet-rspec units tests for the above Profile to make sure that if someone tries to change it, they don't break any existing functionality. Our current workflow has our engineers committing onto branches and creating Merge Requests in our private Git Lab. All tests must pass before they can merge code to Master. They usually get notified within minutes if something they've pushed hasn't passed tests. You can do testing of Hiera-defined infrastructure, however all approaches I've read about seem awfully cumbersome and wasteful. I won't rant about that today. So tell me, how did I go at convincing you? :-) -Luke -- You received this message because you are subscribed to the Google Groups "Puppet Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to puppet-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/puppet-users/1c759280-fbb3-4432-b26d-af09a461218f%40googlegroups.com.
