Patchcord Production Line Solutions_Sun Telecom
___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Jailed periodic daily scripts smashing CPU
I have a number of servers with roughly 60 jails running on each of them. On these hosts, I've had to disable the periodic security scans due to overly high disk load when they run (which is redundant in jails anyway). However, I still have an issue at 3:01am where the CPU is consumed by dozens of 'xz -c' processes. This is apparently daily log rolling, which I can't exactly disable. The effect is that our processing applications experience a major slowdown for about 15 seconds every morning, which is just enough that it's starting to get people's attention. What is the best way to mitigate this? I'm aware of the cron jitter feature, but I'm not sure of the 60-second jitter maximum would be enough (especially if I wanted to start utilizing more jails). - .Dustin ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Jailed periodic daily scripts smashing CPU
Is the problem caused by newsyslog or by the periodic scripts? Newsyslog normally runs from cron directly, not through periodic. In any case, here are a few suggestions: 1) Turn on cron jitter, as you suggested. Even if 60s isn't enough, it can't hurt. 2) Try gz compression instead of xz compression to see if it's faster 3) Manually edit the jails' /etc/crontab files to hardcode some variability into their newsyslog and/or periodic run times 4) If the problem is actually being caused by periodic instead of newsyslog, tell me which script it is and how much jitter you want. I am coincidentally changing how periodic manages jitter right now. -Alan On Thu, Feb 16, 2017 at 2:47 PM, Dustin Wenz wrote: > I have a number of servers with roughly 60 jails running on each of them. On > these hosts, I've had to disable the periodic security scans due to overly > high disk load when they run (which is redundant in jails anyway). However, I > still have an issue at 3:01am where the CPU is consumed by dozens of 'xz -c' > processes. This is apparently daily log rolling, which I can't exactly > disable. > > The effect is that our processing applications experience a major slowdown > for about 15 seconds every morning, which is just enough that it's starting > to get people's attention. > > What is the best way to mitigate this? I'm aware of the cron jitter feature, > but I'm not sure of the 60-second jitter maximum would be enough (especially > if I wanted to start utilizing more jails). > > - .Dustin > ___ > freebsd-stable@freebsd.org mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org" ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Jailed periodic daily scripts smashing CPU
The biggest offender that I see is /usr/local/etc/periodic/daily/411.pkg-backup During the high CPU event, my process list contains hundreds of these: 83811 - RJ 0:03.42 xz -c 83816 - S 0:00.02 /usr/local/sbin/pkg shell .dump 83818 - SJ 0:00.02 /usr/local/sbin/pkg shell .dump 83820 - SJ 0:00.03 /usr/local/sbin/pkg shell .dump 83824 - RJ 0:03.41 xz -c 83831 - RJ 0:03.58 xz -c I could probably get away with disabling that in my case. However, instead of jitter, I think I'd prefer if the periodic jobs ran at a lower priority than my user processes. Either through nice, or idprio. I want them to get done as fast as possible, but I don't want them to affect my application. - .Dustin > On Feb 16, 2017, at 4:20 PM, Alan Somers wrote: > > Is the problem caused by newsyslog or by the periodic scripts? > Newsyslog normally runs from cron directly, not through periodic. In > any case, here are a few suggestions: > 1) Turn on cron jitter, as you suggested. Even if 60s isn't enough, > it can't hurt. > 2) Try gz compression instead of xz compression to see if it's faster > 3) Manually edit the jails' /etc/crontab files to hardcode some > variability into their newsyslog and/or periodic run times > 4) If the problem is actually being caused by periodic instead of > newsyslog, tell me which script it is and how much jitter you want. I > am coincidentally changing how periodic manages jitter right now. > > -Alan > > On Thu, Feb 16, 2017 at 2:47 PM, Dustin Wenz wrote: >> I have a number of servers with roughly 60 jails running on each of them. On >> these hosts, I've had to disable the periodic security scans due to overly >> high disk load when they run (which is redundant in jails anyway). However, >> I still have an issue at 3:01am where the CPU is consumed by dozens of 'xz >> -c' processes. This is apparently daily log rolling, which I can't exactly >> disable. >> >> The effect is that our processing applications experience a major slowdown >> for about 15 seconds every morning, which is just enough that it's starting >> to get people's attention. >> >> What is the best way to mitigate this? I'm aware of the cron jitter feature, >> but I'm not sure of the 60-second jitter maximum would be enough (especially >> if I wanted to start utilizing more jails). >> >>- .Dustin >> ___ >> freebsd-stable@freebsd.org mailing list >> https://lists.freebsd.org/mailman/listinfo/freebsd-stable >> To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org" ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Jailed periodic daily scripts smashing CPU
Adding something like: 'sleep $(( $(sysctl -n security.jail.param.jid) * 15 )) && ' in front of more resource-intensive commands in /etc/crontab can reliably spread out the load from a larger number of jails. (But if you start and stop jails frequently enough to spread out the current list of jail id numbers, this method degrades.) Low priority for 'periodic daily' jobs might not help much, due to disk saturation, CPU cache thrashing, etc. -Walter On Thu, 16 Feb 2017, Dustin Wenz wrote: The biggest offender that I see is /usr/local/etc/periodic/daily/411.pkg-backup During the high CPU event, my process list contains hundreds of these: 83811 - RJ 0:03.42 xz -c 83816 - S 0:00.02 /usr/local/sbin/pkg shell .dump 83818 - SJ 0:00.02 /usr/local/sbin/pkg shell .dump 83820 - SJ 0:00.03 /usr/local/sbin/pkg shell .dump 83824 - RJ 0:03.41 xz -c 83831 - RJ 0:03.58 xz -c I could probably get away with disabling that in my case. However, instead of jitter, I think I'd prefer if the periodic jobs ran at a lower priority than my user processes. Either through nice, or idprio. I want them to get done as fast as possible, but I don't want them to affect my application. - .Dustin On Feb 16, 2017, at 4:20 PM, Alan Somers wrote: Is the problem caused by newsyslog or by the periodic scripts? Newsyslog normally runs from cron directly, not through periodic. In any case, here are a few suggestions: 1) Turn on cron jitter, as you suggested. Even if 60s isn't enough, it can't hurt. 2) Try gz compression instead of xz compression to see if it's faster 3) Manually edit the jails' /etc/crontab files to hardcode some variability into their newsyslog and/or periodic run times 4) If the problem is actually being caused by periodic instead of newsyslog, tell me which script it is and how much jitter you want. I am coincidentally changing how periodic manages jitter right now. -Alan On Thu, Feb 16, 2017 at 2:47 PM, Dustin Wenz wrote: I have a number of servers with roughly 60 jails running on each of them. On these hosts, I've had to disable the periodic security scans due to overly high disk load when they run (which is redundant in jails anyway). However, I still have an issue at 3:01am where the CPU is consumed by dozens of 'xz -c' processes. This is apparently daily log rolling, which I can't exactly disable. The effect is that our processing applications experience a major slowdown for about 15 seconds every morning, which is just enough that it's starting to get people's attention. What is the best way to mitigate this? I'm aware of the cron jitter feature, but I'm not sure of the 60-second jitter maximum would be enough (especially if I wanted to start utilizing more jails). - .Dustin ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org" ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org" ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re: Jailed periodic daily scripts smashing CPU
Dustin Wenz wrote on 2017/02/16 22:47: I have a number of servers with roughly 60 jails running on each of them. On these hosts 60 is way more than we have on our jailers. Daily / security scripts are very disk IO intensive so we end up with changing time in /etc/crontab in each jail for periodic tasks. The best way is to randomize these times on jail creation time. Miroslav Lachman ___ freebsd-stable@freebsd.org mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
RE: Jailed periodic daily scripts smashing CPU
What about:
@daily sleep ${RANDOM:0:2}m ; /some/cron/job.sh
Which gives you a jitter of up to 99 minutes.
Cheers,
Guy
-Original Message-
From: owner-freebsd-sta...@freebsd.org
[mailto:owner-freebsd-sta...@freebsd.org] On Behalf Of Miroslav Lachman
Sent: 17 February 2017 01:20
To: Dustin Wenz; freebsd-stable@freebsd.org
Subject: Re: Jailed periodic daily scripts smashing CPU
Dustin Wenz wrote on 2017/02/16 22:47:
> I have a number of servers with roughly 60 jails running on each of
> them. On these hosts
60 is way more than we have on our jailers. Daily / security scripts are very
disk IO intensive so we end up with changing time in /etc/crontab in each jail
for periodic tasks.
The best way is to randomize these times on jail creation time.
Miroslav Lachman
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
___
This e-mail may contain confidential and/or privileged information. If you are
not the intended recipient (or have received this e-mail in error) please
notify the sender immediately and delete this e-mail. Any unauthorised copying,
disclosure or distribution of the material in this e-mail is prohibited.
Please refer to http://www.bnpparibas.co.uk/en/email-disclaimer/ for additional
disclosures.
___
freebsd-stable@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
