Forum: CFEngine Help
Subject: An advanced policy using template and variable substitution
Author: msvob...@linkedin.com
Link to topic: https://cfengine.com/forum/read.php?3,24105,24105#msg-24105
So, I ran into an issue where /etc/sysctl.conf at our organization was getting
out of control. Using MD5 sums to perform straight file copies, I started
running into this situation.
[*] Define a class of hosts that will not get the standard sysctl.conf. This
super class contains machines in classA, classB, classC
[*] Deploy a standard sysctl.conf to all hosts not in the class above.
[*] Deploy a sysctl.conf to machines in classA
[*] Deploy a sysctl.conf to machines in classB
....
So, this works for the simple case. The problem I ran into very quickly is
when a machine belongs in both classA and classB. Cfengine ends up "fighting
itself" by laying down alternate configurations. This is a problem. So
instead of performing a straight file copy, I approached this problem from a
different angle.
[*] Define default kernel tunable values using variables
[*] If I match classA, change a subset of variables
[*] If I match classB, change another subset of variables. I could change
variables I already changed in classA, or, this could be a whole different set.
It doesn't matter.
[*] Once all my variables are defined, expand the variables from the template
and install the final /etc/sysctl.conf
This ends up working remarkably well. We could have hundreds of different
classes adjusting various kernel tunables. A machine could actually end up
with a unique /etc/sysctl.conf depending on the classes it belonged to. Here's
the policy in action.
First, the template. Since the # symbol is a comment for /etc/sysctl.conf, if
a variable isn't expanded, no damage will be done. I also name the variable
the exact same as the kernel tunable.
################################## Networking
###############################################
# Enable / Disable IP packet forwarding
#%%net.ipv4.ip_forward%%
# Controls source route verification
#%%net.ipv4.conf.default.rp_filter%%
# Accept / deny source routing
#%%net.ipv4.conf.default.accept_source_route%%
# Maximum TCP recieve window
#%%net.core.rmem_default%%
#%%net.core.rmem_max%%
# Maximum TCP send window
#%%net.core.wmem_default%%
#%%net.core.wmem_max%%
# Enable configuration of arp_announce option
#%%net.ipv4.conf.all.arp_announce%%
# Enable configuration of arp_ignore option
#%%net.ipv4.conf.all.arp_ignore%%
# defines the local port range that is used by TCP and UDP traffic to choose
the local por
#%%net.ipv4.ip_local_port_range%%
# time default value for tcp_keepalive_time connection
#%%net.ipv4.tcp_keepalive_time%%
# Controls the use of TCP syncookies
#%%net.ipv4.tcp_syncookies%%
################################## Networking
###############################################
################################## Kernel
###############################################
# Controls the System Request debugging functionality of the kernel
#%%kernel.sysrq%%
# Controls whether core dumps will append the PID to the core filename. Useful
for debugging multi-threaded applications
#%%kernel.core_uses_pid%%
# What format the core file is going to be generated as
#%%kernel.core_pattern%%
# If the box does panic, then make sure we reboot after 60s. Hopefully this is
enough time to flush the caches to disk.
#%%kernel.panic_on_oops%%
#%%kernel.panic%%
# SysV IPC tunables, Semaphores, shared memory segment size.
#%%kernel.msgmax%%
#%%kernel.msgmnb%%
#%%kernel.shmmni%%
#%%kernel.shmmax%%
#%%kernel.shmall%%
#%%kernel.sem%%
################################## Kernel
###############################################
################################ Virtual Memory
#############################################
# Behavior of the machine when it runs out of memory. Are we going to panic
the box?
#%%vm.panic_on_oom%%
# Kill the process that caused the OOM condition. Do not scan for other
processes to fail via the OOM killer.
#%%vm.oom_kill_allocating_task%%
# Dont produce a report on what caused the OOM condition
#%%vm.oom_dump_tasks%%
# If we have a ECC memory error, and a process is mapped to that page, then
kill the process.
#%%vm.memory_failure_early_kill%%
# overcommit RAM behavior
#%%vm.overcommit_memory%%
# Ratio of RAM to be committed
#%%vm.overcommit_ratio%%
# Try to keep the box online during a memory failure event.
#%%vm.memory_failure_recovery%%
# There is a limit to the number of mmap() segments that can be active in a
Linux process at any one time.
#%%vm.max_map_count%%
# The minimum amount of RAM the kernel should keep free at all times.
#%%vm.min_free_kbytes%%
# How large the Linux kernel is allowed to grow the filesystem page cache.
RHEL5 only. Not a RHEL6 tunable
#%%vm.pagecache%%
# how much the kernel favors swap over RAM
#%%vm.swappiness%%
################################ Virtual Memory
#############################################
################################### I/O
##################################################
# reconstruction rates on failed drives.
#%%dev.raid.speed_limit_max%%
# maximum number of allowable concurrent requests.
#%%fs.aio-max-nr%%
# File descriptor maximum limit
#%%fs.file-max%%
################################### I/O
##################################################
Simple enough. So, the policy itself defines each of these kernel values by
defining a list variable, and then overwriting these values. Here's the policy.
5 bundle agent build_etc_sysctl_conf
6 {
7 vars:
8 # These are the base values. If we need to adjust, then raise
a class and overwrite the variable.
9 # Some of these values are RHEL6 specific. The RHEL6 generic
get set below linux, which is generic for RHEL5, CentOS, and RHEL6.
10 # policy => free on every variable assignment here means that
we can "overwrite" or "reassign" the value based off of classes
11 # down below.
12
13 linux::
14 "sysctl" string =>
"net.ipv4.ip_forward = 0",
15
policy => "free";
16
17 "sysctl" string =>
"net.ipv4.conf.default.rp_filter = 1",
18
policy => "free";
19
20 "sysctl" string =>
"net.ipv4.conf.default.accept_source_route = 0",
21
policy => "free";
22
23 "sysctl" string =>
"net.ipv4.tcp_syncookies = 1",
24
policy => "free";
25
26 "sysctl" string =>
"net.core.rmem_default = 124928",
27
policy => "free";
28
29 "sysctl" string =>
"net.core.wmem_default = 124928",
30
policy => "free";
31
32 "sysctl" string =>
"net.core.rmem_max = 131071",
33
policy => "free";
34
35 "sysctl" string =>
"net.core.wmem_max = 131071",
36
policy => "free";
37
38 "sysctl" string =>
"net.ipv4.ip_local_port_range = 32768 61000",
39
policy => "free";
40
41 "sysctl" string =>
"net.ipv4.tcp_keepalive_time = 7200",
42
policy => "free";
43
44 "sysctl" string =>
"net.ipv4.conf.all.arp_ignore = 0",
45
policy => "free";
46
47 "sysctl" string =>
"net.ipv4.conf.all.arp_announce = 0",
48
policy => "free";
49
50 "sysctl" string =>
"vm.panic_on_oom = 0",
51
policy => "free";
52
53 "sysctl" string =>
"vm.oom_dump_tasks = 0",
54
policy => "free";
55
56 "sysctl" string =>
"vm.overcommit_memory = 2",
57
policy => "free";
58
59 "sysctl" string =>
"vm.overcommit_ratio = 90",
60
policy => "free";
61
62 "sysctl" string
=> "vm.swappiness = 60",
63
policy => "free";
64
65 "sysctl" string =>
"vm.max_map_count = 65530",
66
policy => "free";
67
68 "sysctl" string =>
"vm.min_free_kbytes = 90112",
69
policy => "free";
70
71 "sysctl" string =>
"kernel.panic_on_oops = 1",
72
policy => "free";
73
74 "sysctl" string
=> "kernel.panic = 60",
75
policy => "free";
76
77 "sysctl" string
=> "kernel.msgmnb = 16384",
78
policy => "free";
79
80 "sysctl" string
=> "kernel.msgmax = 8192",
81
policy => "free";
82
83 "sysctl" string
=> "kernel.shmmax = 33554432",
84
policy => "free";
85
86 "sysctl" string
=> "kernel.shmall = 2097152",
87
policy => "free";
88
89 "sysctl" string
=> "kernel.shmmni = 4096",
90
policy => "free";
91
92 "sysctl" string =>
"kernel.core_pattern = core",
93
policy => "free";
94
95 "sysctl" string
=> "kernel.sem = 250 32000 32 128",
96
policy => "free";
97
98 "sysctl" string
=> "kernel.sysrq = 0",
99
policy => "free";
100
101 "sysctl" string =>
"kernel.core_uses_pid = 1",
102
policy => "free";
103
104 "sysctl" string =>
"dev.raid.speed_limit_max = 100000",
105
policy => "free";
106
107 "sysctl" string
=> "fs.file-max = 4815594",
108
policy => "free";
109
110 "sysctl" string
=> "fs.aio-max-nr = 65536",
111
policy => "free";
112
113 redhat_6::
114 "sysctl" string =>
"vm.oom_kill_allocating_task = 1",
115
policy => "free";
116
117 "sysctl" string =>
"vm.memory_failure_early_kill = 1",
118
policy => "free";
119
120 "sysctl" string =>
"vm.memory_failure_recovery = 1",
121
policy => "free";
122
123 (centos_5|redhat_5)::
124 "sysctl" string
=> "vm.pagecache=1 15 30",
125
policy => "free";
126
127 linux.class1::
128 "sysctl" string =>
"net.core.rmem_default = 262144",
129
policy => "free";
130
131 "sysctl" string =>
"net.core.wmem_default = 262144",
132
policy => "free";
133
134 "sysctl" string =>
"net.core.rmem_max = 4194304",
135
policy => "free";
136
137 "sysctl" string =>
"net.core.wmem_max = 1048576",
138
policy => "free";
139
140 "sysctl" string =>
"net.ipv4.ip_local_port_range = 9000 65500",
141
policy => "free";
142
143 "sysctl" string =>
"vm.overcommit_memory = 0",
144
policy => "free";
145
146 "sysctl" string =>
"vm.overcommit_ratio = 50",
147
policy => "free";
148
149 "sysctl" string =>
"vm.max_map_count = 5242880",
150
policy => "free";
151
152 "sysctl" string
=> "vm.swappiness = 10",
153
policy => "free";
154
155 "sysctl" string
=> "vm.pagecache=1 15 30",
156
policy => "free";
157
158 "sysctl" string =>
"vm.min_free_kbytes = 1024",
159
policy => "free";
160
161 "sysctl" string
=> "kernel.msgmnb = 65536",
162
policy => "free";
163
164 "sysctl" string
=> "kernel.msgmax = 65536",
165
policy => "free";
166
167 "sysctl" string
=> "kernel.shmmax = 68719476736",
168
policy => "free";
169
170 "sysctl" string
=> "kernel.shmall = 4294967296",
171
policy => "free";
172
173 "sysctl" string
=> "kernel.shmmni = 4096",
174
policy => "free";
175
176 "sysctl" string
=> "kernel.panic = 0",
177
policy => "free";
178
179 "sysctl" string =>
"kernel.core_pattern = /var/log/core/core.%u.%e.%p",
180
policy => "free";
181
182 "sysctl" string
=> "kernel.sem = 250 32000 100 2048",
183
policy => "free";
184
185 "sysctl" string
=> "fs.file-max = 6815744",
186
policy => "free";
187
188 "sysctl" string
=> "fs.aio-max-nr = 1048576",
189
policy => "free";
190
191 linux.class2::
192 "sysctl" string =>
"net.core.wmem_max = 1310710",
193
policy => "free";
194
195 "sysctl" string =>
"net.core.wmem_max = 1310710",
196
policy => "free";
197
198 linux.tcp_keepalive_raised::
199 "sysctl" string =>
"net.ipv4.tcp_keepalive_time = 1800",
200
policy => "free";
201
202 linux.class4::
203 "sysctl" string =>
"net.core.rmem_default = 262144",
204
policy => "free";
205
206 "sysctl" string =>
"net.core.wmem_default = 262144",
207
policy => "free";
208
209 "sysctl" string =>
"net.core.rmem_max = 4194304",
210
policy => "free";
211
212 "sysctl" string =>
"net.core.wmem_max = 1048576",
213
policy => "free";
214
215 "sysctl" string =>
"net.ipv4.ip_local_port_range = 9000 65500",
216
policy => "free";
217
218 "sysctl" string
=> "kernel.shmmni = 4096",
219
policy => "free";
220
221 "sysctl" string
=> "kernel.sem = 250 32000 100 128",
222
policy => "free";
223
224 "sysctl" string
=> "fs.aio-max-nr = 1048576",
225
policy => "free";
226
227 "sysctl" string
=> "fs.file-max = 6815744",
228
policy => "free";
229
230 "sysctl" string
=> "kernel.shmmax = 33285996544",
231
policy => "free";
232
233 "sysctl" string
=> "kernel.shmall = 8126464",
234
policy => "free";
235
236 linux.class5::
237 "sysctl" string =>
"net.core.rmem_default = 262144",
238
policy => "free";
239
240 "sysctl" string =>
"net.core.wmem_default = 262144",
241
policy => "free";
242
243 "sysctl" string =>
"net.core.rmem_max = 4194304",
244
policy => "free";
245
246 "sysctl" string =>
"net.core.wmem_max = 1048576",
247
policy => "free";
248
249 "sysctl" string =>
"net.ipv4.ip_local_port_range = 9000 65500",
250
policy => "free";
251
252 "sysctl" string
=> "kernel.shmmni = 4096",
253
policy => "free";
254
255 "sysctl" string
=> "kernel.sem = 250 32000 100 128",
256
policy => "free";
257
258 "sysctl" string
=> "fs.aio-max-nr = 1048576",
259
policy => "free";
260
261 "sysctl" string
=> "fs.file-max = 6815744",
262
policy => "free";
263
264 "sysctl" string
=> "kernel.shmmax = 12884901888",
265
policy => "free";
266
267 "sysctl" string
=> "kernel.shmall = 3145728",
268
policy => "free";
269
270 linux.class6::
271 "sysctl" string =>
"net.ipv4.conf.all.arp_announce = 2",
272
policy => "free";
273
274 "sysctl" string =>
"net.ipv4.conf.all.arp_ignore = 1",
275
policy => "free";
276
277
278 # Grab the index of the above array, so we can loop through
each item and replace / insert into sysctl.conf as needed.
279 linux::
280 "kernel_tunables"
slist => getindices("sysctl");
281
282
283 classes:
284 linux::
285 "tcp_keepalive_raised"
or => {"machine1",
286
"classD",
287
"machine2",
288
"classE", };
289 files:
290 linux::
291 "/etc/sysctl.staging"
292 handle =>
"linux_template_etc_sysctl_conf",
293 copy_from =>
backup_cp_md5_compare("/var/cfengine/inputs/config-linux/build_etc_sysctl_conf/sysctl.conf");
294
295 linux::
296 "/etc/sysctl.final"
297 handle =>
"linux_build_sysctl_final",
298 create => "true",
299 perms =>
mog("0644","root","root"),
300 edit_line =>
edit_sysctl_conf("@(build_etc_sysctl_conf.kernel_tunables)"),
301 edit_defaults => empty;
302
303 linux::
304 "/etc/sysctl.conf"
305 handle =>
"linux_install_sysctl_conf_from_template",
306 perms =>
mog("0644","root","root"),
307 copy_from =>
backup_cp_md5_compare("/etc/sysctl.final"),
308 action => immediate,
309 classes =>
if_repaired("sysctl_conf_modified");
310
311 commands:
312 linux.sysctl_conf_modified::
313 "/sbin/sysctl -p";
314
315 reports:
316 linux.sysctl_conf_modified::
317 "cf3: /etc/sysctl.conf was modified on $(sys.host).
Executing /sbin/sysctl -p for the values to take effect on the system.";
318 }
319 ##############################################################
320 bundle edit_line edit_sysctl_conf(kernel_tunables)
321 {
322 insert_lines:
323 "/etc/sysctl.staging"
324 comment => "Basic template for
/etc/sysctl.conf using variable subsitution for configuration",
325 insert_type => "file";
326
327 replace_patterns:
328 "#%%$(kernel_tunables)%%"
329 replace_with =>
value("$(build_etc_sysctl_conf.sysctl[$(kernel_tunables)])");
330 }
So, some important take away concepts here:
[*] Line 283, I can actually define classes within this policy itself.
Although Cfengine normal ordering processes variables before classes, the
policy is executed multiple times. This means that a class defined in pass 1,
will be applied to the variables in pass 2.
[*] All the variables are stored in an array for two reasons. One, its really
easy to grab all of the variables we want to expand using getindices on line
280. Two, where the expansion actually happens on lines 328 and 329, we use
the variable name in the template (which is also our index) and simply expand
the array at that position.
[*] We pass the slist as an argument at line 300 to the edit_line bundle, but
we dont pass the array. Instead of passing the array, we define the exact
scope of the policy the array lives in on line 329.
[*] 3 Files are involved in this transaction. The first is simply the
template. The second, /etc/sysctl.final, is the template expanded. Finally,
we do a MD5 comparison / file copy from /etc/sysctl.final to /etc/sysctl.conf.
I found that 3 files are necessary.
Yes, technically, this works to do variable expansion using a template --->
/etc/sysctl.conf directly... but... What if a user comes in and modifies
/etc/sysctl.conf after the fact? Keeping a MD5 comparison from the
intermediate /etc/sysctl.final to /etc/sysctl.conf file guarantees that we
control _all_ of the contents of /etc/sysctl.conf.
Also, it allows us to set a class at line 309. This class we use to execute a
command statement in 311-313 to make the config changes applied to the live
system. We only execute sysctl -p to apply the changes if a config change was
detected.
Anyhow, this might have all seemed complex, but, it gives you some great
flexibility in how complex you could have classes defined for config options in
a policy. This approach could really be applied to any configuration --
/etc/sysctl.conf was just our use case.
Finally, here's the policy in action. First, a RHEL6 machine.
# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 6.1 (Santiago)
# /var/cfengine/bin/cf-agent -I -K -b build_etc_sysctl_conf
>> Using command line specified bundlesequence
-> Copying from localhost:/etc/sysctl.final
-> Object /etc/sysctl.conf had permission 600, changed it to 644
-> Executing '/sbin/sysctl -p' ...(timeout=-678,owner=-1,group=-1)
Q: "...bin/sysctl -p": net.ipv4.ip_forward = 0
Q: "...bin/sysctl -p": net.ipv4.conf.default.rp_filter = 1
Q: "...bin/sysctl -p": net.ipv4.conf.default.accept_source_route = 0
Q: "...bin/sysctl -p": net.core.rmem_default = 124928
Q: "...bin/sysctl -p": net.core.rmem_max = 131071
Q: "...bin/sysctl -p": net.core.wmem_default = 124928
Q: "...bin/sysctl -p": net.core.wmem_max = 131071
Q: "...bin/sysctl -p": net.ipv4.conf.all.arp_announce = 0
Q: "...bin/sysctl -p": net.ipv4.conf.all.arp_ignore = 0
Q: "...bin/sysctl -p": net.ipv4.ip_local_port_range = 32768 61000
Q: "...bin/sysctl -p": net.ipv4.tcp_keepalive_time = 7200
Q: "...bin/sysctl -p": net.ipv4.tcp_syncookies = 1
Q: "...bin/sysctl -p": kernel.sysrq = 0
Q: "...bin/sysctl -p": kernel.core_uses_pid = 1
Q: "...bin/sysctl -p": kernel.core_pattern = core
Q: "...bin/sysctl -p": kernel.panic_on_oops = 1
Q: "...bin/sysctl -p": kernel.panic = 60
Q: "...bin/sysctl -p": kernel.msgmax = 8192
Q: "...bin/sysctl -p": kernel.msgmnb = 16384
Q: "...bin/sysctl -p": kernel.shmmni = 4096
Q: "...bin/sysctl -p": kernel.shmmax = 33554432
Q: "...bin/sysctl -p": kernel.shmall = 2097152
Q: "...bin/sysctl -p": kernel.sem = 250 32000 32 128
Q: "...bin/sysctl -p": vm.panic_on_oom = 0
Q: "...bin/sysctl -p": vm.oom_kill_allocating_task = 1
Q: "...bin/sysctl -p": vm.oom_dump_tasks = 0
Q: "...bin/sysctl -p": vm.memory_failure_early_kill = 1
Q: "...bin/sysctl -p": vm.overcommit_memory = 2
Q: "...bin/sysctl -p": vm.overcommit_ratio = 90
Q: "...bin/sysctl -p": vm.memory_failure_recovery = 1
Q: "...bin/sysctl -p": vm.max_map_count = 65530
Q: "...bin/sysctl -p": vm.min_free_kbytes = 90112
Q: "...bin/sysctl -p": vm.swappiness = 60
Q: "...bin/sysctl -p": dev.raid.speed_limit_max = 100000
Q: "...bin/sysctl -p": fs.aio-max-nr = 65536
Q: "...bin/sysctl -p": fs.file-max = 4815594
I: Last 36 quoted lines were generated by promiser "/sbin/sysctl -p"
-> Completed execution of /sbin/sysctl -p
R: cf3: /etc/sysctl.conf was modified on machine1. Executing /sbin/sysctl -p
for the values to take effect on the system.
Next, a RHEL5 box.
# cat /etc/redhat-release
Red Hat Enterprise Linux Server release 5.5 (Tikanga)
# /var/cfengine/bin/cf-agent -I -K -b build_etc_sysctl_conf
>> Using command line specified bundlesequence
-> Copying from localhost:/etc/sysctl.final
-> Object /etc/sysctl.conf had permission 600, changed it to 644
-> Executing '/sbin/sysctl -p' ...(timeout=-678,owner=-1,group=-1)
Q: "...bin/sysctl -p": net.ipv4.ip_forward = 0
Q: "...bin/sysctl -p": net.ipv4.conf.default.rp_filter = 1
Q: "...bin/sysctl -p": net.ipv4.conf.default.accept_source_route = 0
Q: "...bin/sysctl -p": net.core.rmem_default = 262144
Q: "...bin/sysctl -p": net.core.rmem_max = 4194304
Q: "...bin/sysctl -p": net.core.wmem_default = 262144
Q: "...bin/sysctl -p": net.core.wmem_max = 1048576
Q: "...bin/sysctl -p": net.ipv4.conf.all.arp_announce = 0
Q: "...bin/sysctl -p": net.ipv4.conf.all.arp_ignore = 0
Q: "...bin/sysctl -p": net.ipv4.ip_local_port_range = 9000 65500
Q: "...bin/sysctl -p": net.ipv4.tcp_keepalive_time = 7200
Q: "...bin/sysctl -p": net.ipv4.tcp_syncookies = 1
Q: "...bin/sysctl -p": kernel.sysrq = 0
Q: "...bin/sysctl -p": kernel.core_uses_pid = 1
Q: "...bin/sysctl -p": kernel.core_pattern = /var/log/core/core.%%u.%%e.%%p
Q: "...bin/sysctl -p": kernel.panic_on_oops = 1
Q: "...bin/sysctl -p": kernel.panic = 0
Q: "...bin/sysctl -p": kernel.msgmax = 65536
Q: "...bin/sysctl -p": kernel.msgmnb = 65536
Q: "...bin/sysctl -p": kernel.shmmni = 4096
Q: "...bin/sysctl -p": kernel.shmmax = 68719476736
Q: "...bin/sysctl -p": kernel.shmall = 4294967296
Q: "...bin/sysctl -p": kernel.sem = 250 32000 100 2048
Q: "...bin/sysctl -p": vm.panic_on_oom = 0
Q: "...bin/sysctl -p": vm.overcommit_memory = 0
Q: "...bin/sysctl -p": vm.overcommit_ratio = 50
Q: "...bin/sysctl -p": vm.max_map_count = 5242880
Q: "...bin/sysctl -p": vm.min_free_kbytes = 1024
Q: "...bin/sysctl -p": vm.pagecache = 1 15 30
Q: "...bin/sysctl -p": vm.swappiness = 10
Q: "...bin/sysctl -p": dev.raid.speed_limit_max = 100000
Q: "...bin/sysctl -p": fs.aio-max-nr = 1048576
Q: "...bin/sysctl -p": fs.file-max = 6815744
I: Last 34 quoted lines were generated by promiser "/sbin/sysctl -p"
-> Completed execution of /sbin/sysctl -p
R: cf3: /etc/sysctl.conf was modified on machine2. Executing /sbin/sysctl -p
for the values to take effect on the system.
At the end, we get different kernel values applied:
Machine1:
# sysctl net.core.rmem_default
net.core.rmem_default = 124928
Machine2:
# sysctl net.core.rmem_default
net.core.rmem_default = 262144
Anyways, I hope this approach of using an array and expanding variables via a
template helps someone else out trying to solve a complex configuration.
Thanks
Mike
_______________________________________________
Help-cfengine mailing list
Help-cfengine@cfengine.org
https://cfengine.org/mailman/listinfo/help-cfengine
