On Tuesday, July 21, 2020 at 9:53:41 AM UTC+2 [email protected] wrote:
> > What is the typical CPU utilization for these nodes? Do you notice if any > correlation between CPUs that jump backwards and the load on that CPU at > that time? My question is, when a CPU jumps backwards, is it under high or > low utilization? > It's fairly I/O-heavy workloads, so the CPUs are seldom maxed out. The jumps occur both when the system is idle and when the CPUs are working - I've checked that is the case on an individual CPU level too. At the moment I'm just eye-balling graphs, no hard statistics to tell whether there is correlation, but there is nothing blatantly jumping out at me. It does appear that when a jump occurs, it occurs on multiple CPUs at the same time (even across sockets), and sometimes multiple counters for the same CPU. > When idle jumps backwards, how much does it jump back by? What are the > absolute values for the counter before and after the jump back? Right now > we reset the counters if idle jumps back any amount, assuming this happens > when the kernel hotplugs a CPU. But this was a very big assumption based on > some limited testing. We might want to change things to only reset > everything if there's a jump back of more than X%. > Again, this is just eyeballing logs, but I've looked at maybe 100 instances and they're all jumping back by 0.01 seconds (which is also the resolution). The idle counters are in the range 1-2 million, the others in the range 10k-100k. Cheers Bruce Disclaimer The information contained in this communication from the sender is confidential. It is intended solely for use by the recipient and others authorized to receive it. If you are not the recipient, you are hereby notified that any disclosure, copying, distribution or taking action in relation of the contents of this information is strictly prohibited and may be unlawful. This email has been scanned for viruses and malware, and may have been automatically archived by Mimecast, a leader in email security and cyber resilience. Mimecast integrates email defenses with brand protection, security awareness training, web security, compliance and other essential capabilities. Mimecast helps protect large and small organizations from malicious activity, human error and technology failure; and to lead the movement toward building a more resilient world. To find out more, visit our website. -- You received this message because you are subscribed to the Google Groups "Prometheus Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/prometheus-users/23159371-f491-4f43-921f-61b173a19901n%40googlegroups.com.
