On Fri, Feb 17, 2017 at 10:47:23AM -0600, Dustin Wenz wrote: > If I needed to go down the road of spreading out the daily maintenance over a > longer period of time, I'd probably use your solution of building in a delay > based on the jail id, or possibly Guy Tabrar's extreme jitter example. > > The pkg-backup job appears somewhat flawed, in my view. The xz compressor is > totally CPU bound for a moderate number of jails . Sure, running it at low > priority isn't as cache friendly, but it's certainly better then the host > becoming unresponsive. The other issue is that in most cases the package > archive doesn't change, and so running a daily backup without checking for > changes is wasteful. In my case, ZFS handles all the rolling backups that I > could ever need, so using pkg-backup for that purpose is redundant. >
Hi, At least in my copy of the /usr/local/etc/periodic/daily/411.pkg-backup script, there is an option to enable backing up the pkg db in the jails from the host OS. If you put daily_backup_pkgng_enable=NO into the periodic.conf in the jails and daily_backup_pkg_jails=YES in the host there should be no more than a single thread consuming CPU doing the pkg backups as they're done sequentially rather than in parallel. In general I think a method of forcing a jail to start it's jobs later based on JID (or some other metric as JID could get rather large and sparse if you stop/start jails a lot) is a good idea that should be looked at for inclusion in base Regards, Gary > > > On Feb 16, 2017, at 5:44 PM, Walter Cramer <w...@mintsol.com> wrote: > > > > Adding something like: > > > > 'sleep $(( $(sysctl -n security.jail.param.jid) * 15 )) && ' > > > > in front of more resource-intensive commands in /etc/crontab can reliably > > spread out the load from a larger number of jails. > > > > (But if you start and stop jails frequently enough to spread out the > > current list of jail id numbers, this method degrades.) > > > > Low priority for 'periodic daily' jobs might not help much, due to disk > > saturation, CPU cache thrashing, etc. > > -Walter > > > > On Thu, 16 Feb 2017, Dustin Wenz wrote: > > > >> The biggest offender that I see is > >> /usr/local/etc/periodic/daily/411.pkg-backup > >> > >> During the high CPU event, my process list contains hundreds of these: > >> > >> 83811 - RJ 0:03.42 xz -c > >> 83816 - S 0:00.02 /usr/local/sbin/pkg shell .dump > >> 83818 - SJ 0:00.02 /usr/local/sbin/pkg shell .dump > >> 83820 - SJ 0:00.03 /usr/local/sbin/pkg shell .dump > >> 83824 - RJ 0:03.41 xz -c > >> 83831 - RJ 0:03.58 xz -c > >> > >> I could probably get away with disabling that in my case. > >> > >> However, instead of jitter, I think I'd prefer if the periodic jobs ran at > >> a lower priority than my user processes. Either through nice, or idprio. I > >> want them to get done as fast as possible, but I don't want them to affect > >> my application. > >> > >> - .Dustin > >> > >> > >>> On Feb 16, 2017, at 4:20 PM, Alan Somers <asom...@freebsd.org> wrote: > >>> > >>> Is the problem caused by newsyslog or by the periodic scripts? > >>> Newsyslog normally runs from cron directly, not through periodic. In > >>> any case, here are a few suggestions: > >>> 1) Turn on cron jitter, as you suggested. Even if 60s isn't enough, > >>> it can't hurt. > >>> 2) Try gz compression instead of xz compression to see if it's faster > >>> 3) Manually edit the jails' /etc/crontab files to hardcode some > >>> variability into their newsyslog and/or periodic run times > >>> 4) If the problem is actually being caused by periodic instead of > >>> newsyslog, tell me which script it is and how much jitter you want. I > >>> am coincidentally changing how periodic manages jitter right now. > >>> > >>> -Alan > >>> > >>> On Thu, Feb 16, 2017 at 2:47 PM, Dustin Wenz <dustinw...@ebureau.com> > >>> wrote: > >>>> I have a number of servers with roughly 60 jails running on each of > >>>> them. On these hosts, I've had to disable the periodic security scans > >>>> due to overly high disk load when they run (which is redundant in jails > >>>> anyway). However, I still have an issue at 3:01am where the CPU is > >>>> consumed by dozens of 'xz -c' processes. This is apparently daily log > >>>> rolling, which I can't exactly disable. > >>>> > >>>> The effect is that our processing applications experience a major > >>>> slowdown for about 15 seconds every morning, which is just enough that > >>>> it's starting to get people's attention. > >>>> > >>>> What is the best way to mitigate this? I'm aware of the cron jitter > >>>> feature, but I'm not sure of the 60-second jitter maximum would be > >>>> enough (especially if I wanted to start utilizing more jails). > >>>> > >>>> - .Dustin > >>>> _______________________________________________ > >>>> freebsd-stable@freebsd.org mailing list > >>>> https://lists.freebsd.org/mailman/listinfo/freebsd-stable > >>>> To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org" > >> > >> _______________________________________________ > >> freebsd-stable@freebsd.org mailing list > >> https://lists.freebsd.org/mailman/listinfo/freebsd-stable > >> To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org" > > > > _______________________________________________ > > freebsd-stable@freebsd.org mailing list > > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > > To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org" > > _______________________________________________ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org" > _______________________________________________ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
