Hi Attila - thanks for the pointers, but I'm not sure of how to act upon them.
The start-up for Cadvisor that I'm using doesn't feature any pointer to a parameter list, and despite much googling I don't see any mention of such a thing. Everything keeps referring back to Prometheus and then on to Grafana My Cadvisor start-up (taken directly from the IBM Red Book and slightly modified to comply with local restrictions): docker network create monitoring docker run --name cadvisor -v /sys:/sys:ro -v /var/lib/docker/:/var/lib/docker:ro -v /dev/disk:/dev/disk:ro -d --network monitoring ibmcom/cadvisor-s390x:0.33.0 Perhaps I'm looking at things the wrong way, but my current understanding is: Cadvisor (and also Nodeexporter) collect various usage stats; Prometheus then gathers that data and does some sort of pre-processing of it (it doesn't tell Cadvisor to 'do something' - it just passively makes use of the data that Cadvisor collects) Grafana takes the data from Prometheus and uses it to generate various graphs/tables/reports. My situation is that when I run Cadvisor on it's own - no other containers at all - then it floods as many processors as I define in the zcx start.json file. Whilst Cadvisor is running, I can go to the relevant web-page and I can see that it is producing meters/charts, etc all on its own. Since that is the case, what is the point of Grafana? I have a Prometheus.yml file that features the term 'scrape_interval' (but not 'housekeeping'), but that file is for use by Prometheus, isn't it? How does it affect the amount of work that Cadvisor is doing, since I haven't even started that container yet? Regards Sean On Wed, 26 Aug 2020 at 23:05, Attila Fogarasi <fogar...@gmail.com> wrote: > Check your values for housekeeping interval and scrape_interval. > Recommended is 15s and 30s (which makes for a 60 second rate window). > Small value for housekeeping interval will cause cAdvisor cpu usage to be > high, while scrape_interval affects Prometheus cpu usage. It is entirely > possible to cause data collection to use 100% of the z/OS cpu -- remember > that on Unix systems the rule of thumb is 40% overhead for uncaptured cpu > time while z/OS is far more efficient and runs well under 10%. You will > see this behaviour in zCX containers, it isn't going to measure the same as > z/OS workload. The optimizations in Unix have the premise that cpu time is > low cost (as is memory), while z/OS considers cpu to be high cost and path > length worth saving. Same for the subsystems in z/OS and performance > monitors. > > On Wed, Aug 26, 2020 at 11:43 PM Sean Gleann <sean.gle...@gmail.com> > wrote: > > > Allan - "...count the beans differently...' Yes, I'm beginning to get > used > > to that concept. For instance, with the CPU Utilisation data that I > *have* > > been able to retrieve, the metric given is not 'CPU%', but 'Number of > > cores'. I'm having to do some rapid re-orienting to my way of thinking. > > As for the memory size, I've got "mem-gb" : 2 defined in my start.json > > file, but I've not seen any indication of paging load at all in my > testing. > > > > Michael - 5 zIIPs? I wish! Nope - these are all general-purpose > > processors. > > The z/OS system I'm using is a z/VM guest on a system run by an external > > supplier, so I'm not sure if defining zIIPs would actually achieve > anything > > (Is it possible to dedicate a zIIP engine to a specific z/VM guest? > That's > > a road I've not yet gone down). > > With regard to the WLM definitions, I followed the advice in the red book > > and I'm reasonably certain I've got it right. Having said that, > cross-refer > > to a thread that I started earlier this week, titled "WLM Query" > > The response to that led to me defining a resource group to cap the > > started task to 10MSU, which resulted in a CPU% Util value of roughly 5% > - > > something I could be happy with. > > Under that cap, the started task ran, yes, but it ran like a three-legged > > dog (my apologies to limb-count-challenged canines). > > Start-up of the task, from the START command to the "server is > > listening..." message took over an hour, and > > STOP-command-to-task-termination took approx. 30 minutes. > > (SSH-ing to the task was a bit of a joke, too. Responses to simple > commands > > like 'docker ps -a' could be seen 'painting' across the screen, > > character-by-character...) > > As a result, I've moved away from trying to limit the task for the time > > being. I'm concentrating on attempting to get cadvisor to be a bit less > > greedy. > > > > Regards > > Sean > > > > On Wed, 26 Aug 2020 at 13:49, Michael Babcock <bigironp...@gmail.com> > > wrote: > > > > > I can’t check my zCX out right now since my internet is down. > > > > > > You are running these on zIIP engines correct? Must be nice to have 5 > > > zIIPs! And have the WLM parts in place? Although it probably > wouldn’t > > > make much difference during startup/shutdown. > > > > > > On Wed, Aug 26, 2020 at 3:40 AM Sean Gleann <sean.gle...@gmail.com> > > wrote: > > > > > > > Can anyone offer advice, please, with regard to monitoring the system > > > > > > > > resource consumption of a zcx Container task? > > > > > > > > > > > > > > > > I've got a zcx Container task running on a 'sandbox' system where - > as > > > yet > > > > > > > > - I'm not collecting any RMF/SMF data. Because of that, my only > source > > of > > > > > > > > system usage is the SDSF DA panel. I feel that the numbers I see > there > > > > > > > > are... 'questionable' is the best word I can think of. > > > > > > > > > > > > > > > > Firstly, the EXCP-count for the task goes up to about 15360 during > the > > > > > > > > initial start-up phase, but then it stays there until the STOP > command > > is > > > > > > > > issued. At that point, EXCP-count starts rising again, until the task > > > > > > > > finally terminates. The explanation for that is probably because all > > the > > > > > > > > I/O is being handled internally at the 'Linux' level - the task must > be > > > > > > > > doing *some* I/O, right? - but the data isn't getting back to SDSF > for > > > some > > > > > > > > reason. Without the benefit of SMF data to examine, I'm wondering if > > this > > > > > > > > is part of a larger problem. > > > > > > > > > > > > > > > > The other thing that troubles me is the CPU% busy value. My sandbox > > > system > > > > > > > > has 5 engines defined, and in the 'start.json' file that controls the > > zcx > > > > > > > > Container task, I've specified a 'cpu' value of 4. During the > start-up > > > > > > > > phase for the Container started task, SDSF shows CPU% values of > approx > > > 80%, > > > > > > > > but when the task is finally initialised, this drops to 'tickover' > > rates > > > of > > > > > > > > about 1%. I'm happy with that - the initial start-up of *any* task as > > > > > > > > complex as a zcx Container is likely to cause high CPU usage, and the > > > > > > > > subsequent drop to the 1% levels is fine by me. > > > > > > > > > > > > > > > > But... Once the Container task is started and I've ssh'd into it, I > > then > > > > > > > > want to monitor its 'internal' system consumption. I've been using > the > > > > > > > > 'Getting Started...' redbook as my guide throughout all this project, > > and > > > > > > > > it talks about using "Nodeexporter", "Cadvisor", "Prometheus" and > > > "Grafana" > > > > > > > > as tools for this. I've got all those things installed and I can > start > > > and > > > > > > > > stop them quite happily, but I've found that using Cadvisor on it's > own > > > can > > > > > > > > drive CPU% levels back up to 80% for the entire time it is running. > If > > a > > > > > > > > system is running flat-out when all it is doing is monitoring itself, > > > well, > > > > > > > > there's something wrong somewhere... I'm trying to find an idiot's > > guide > > > to > > > > > > > > controlling what Cadvisor does, but as yet I've been unsuccessful. > > > > > > > > > > > > > > > > Regards > > > > > > > > Sean > > > > > > > > > > > > > > > > > ---------------------------------------------------------------------- > > > > > > > > For IBM-MAIN subscribe / signoff / archive access instructions, > > > > > > > > send email to lists...@listserv.ua.edu with the message: INFO > IBM-MAIN > > > > > > > > -- > > > Michael Babcock > > > OneMain Financial > > > z/OS Systems Programmer, Lead > > > > > > ---------------------------------------------------------------------- > > > For IBM-MAIN subscribe / signoff / archive access instructions, > > > send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN > > > > > > > ---------------------------------------------------------------------- > > For IBM-MAIN subscribe / signoff / archive access instructions, > > send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN > > > > ---------------------------------------------------------------------- > For IBM-MAIN subscribe / signoff / archive access instructions, > send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN > ---------------------------------------------------------------------- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
