On Wed, May 24, 2023 at 04:53:49PM +0200, Anthony Harivel wrote: > set=UTF-8 > Status: RO > Content-Length: 24102 > Lines: 667 > > Marcelo Tosatti, May 19, 2023 at 20:28: > > Hi Marcelo, > > > > > > + /* Assuming those values are the same accross physical > > > > > system/packages */ > > > > > + maxcpus = get_maxcpus(0); /* Number of CPUS per packages */ > > > > > + maxpkgs = numa_max_node(); /* Number of Packages on the system */ > > > > numa_max_node() returns the highest node number available on the current > > system. > > (See the node numbers in /sys/devices/system/node/ ). Also see > > numa_num_configured_nodes(). > > > > One can find package topology information from > > /sys/devices/system/cpu/cpuX/topology/ > > > > Good point. > I will find a better solution to identify the topology using your hint. > > > > > + /* Allocate memory for each thread stats */ > > > > > + thd_stat = (thread_stat *) calloc(num_threads, > > > > > sizeof(thread_stat)); > > > > Can you keep this pre-allocated ? And all other data as well. > > Ok no problem. > > > > > + /* Retrieve all packages power plane energy counter */ > > > > > + for (int i = 0; i <= maxpkgs; i++) { > > > > > + for (int j = 0; j < num_threads; j++) { > > > > > + /* > > > > > + * Use the first thread we found that ran on the CPU > > > > > + * of the package to read the packages energy counter > > > > > + */ > > > > > + if (thd_stat[j].numa_node_id == i) { > > > > > + pkg_stat[i].e_start = > > > > > read_msr(MSR_PKG_ENERGY_STATUS, i); > > > > > + break; > > > > > + } > > > > > + } > > > > > + } > > > > NUMA node does not map necessarily to one package. > > True. I will update this part at the same time with the topology info > discussed above. > > > > > > > > + /* Sleep a short period while the other threads are working > > > > > */ > > > > > + usleep(MSR_ENERGY_THREAD_SLEEP_US); > > > > > + > > > > > + /* > > > > > + * Retrieve all packages power plane energy counter > > > > > + * Calculate the delta of all packages > > > > > + */ > > > > > + for (int i = 0; i <= maxpkgs; i++) { > > > > > + for (int j = 0; j < num_threads; j++) { > > > > > + /* > > > > > + * Use the first thread we found that ran on the CPU > > > > > + * of the package to read the packages energy counter > > > > > + */ > > > > > + if (thd_stat[j].numa_node_id == i) { > > > > > + pkg_stat[i].e_end = > > > > > + read_msr(MSR_PKG_ENERGY_STATUS, > > > > > thd_stat[j].cpu_id); > > > > This is excessive (to read the MSRs of each package in the system). > > > > Consider 100 Linux guests all of them with this enabled, on a system with > > 4 packages. How many times you'll be reading MSR of each package? > > The problem here is that you can have vCPUs that are running on different > packages. However the energy counter of the different packages are > increasing independently. > Either we "force" somehow users to run only on the same package, either I'm > afraid we are obliged to read all the packages energy counter (when they > are involved in the VM). > > Imagine this: > > |----pkg-0----|----pkg-1----| > |0|1|2|3|4|5|6|0|1|2|3|4|5|6| > | | | | > | vm-0 | vm-1 | vm-2 | > > Only vm-1 that has cores from pkg-0 and pkg-1 would have to read both > pkg energy. vm-0 would only read pkg-0 and vm-2 only pkg-1. > > > > > > Moreover, don't want to readmsr on an isolated CPU. > > > > Could you explain me why ?
Nevermind, its a separate topic. > > > No problem, let me try to explain: > > > a QEMU process is composed of vCPU thread(s) and non-vCPU thread(s) (IO, > > > emulated device,...). Each of those threads can run on different cores > > > that can belongs to the same Package or not. > > > The MSR_PKG_ENERGY_STATUS is a counter that increment for the whole > > > Package domain. If you read this MSR from any core that belongs to the > > > package, you will retrieve the same number. > > > So when I need to read the MSR, I only need to read once for all the > > > threads that are running on cores of each Package. > > > > T=0 read p1v0 (== MSR_PKG_ENERGY_STATUS) > > T=1 vcpu-1 executing in core1 > > vcpu-2 executing in core2 > > vcpu-3 executing in core2 > > T=2 read p1v1 > > > > Won't you be exposing (p1v1-p1v0)/3 by 3 in this case, to the > > virtual MSR_PKG_ENERGY_STATUS? > > > > No, because if we take for exemple a 4 cores per package with 100 ticks per > second, > the maximum number of ticks for all the cores in this packages would be > 400 ticks per second. > So if 2 or more vcpus are sharing the same core, they will be maxout by the > 100 ticks per second anyway and so ratio will still be ok. > > > > > Now let's talk about the implementation of the emulated value. > > > I've created a thread that does in an infinite loop the following: > > > - Retrieve all the QEMU threads + statistics about them > > > - Read the energy counter of each Package involved > > > - Sleep for 1sec (1sec is arbitrary) > > > - Calculate the delta of ticks for each threads so that we know how much > > > time each threads has been scheduled during the Sleep period > > > > Intel docs mention the counter can overflow (faster with higher > > energy consumption). Are you handling the overflow case? > > > > Should we ? On baremetal, anyone reading the MSRs should handle the > overflow case, I guess on a guest this should be the same. > But I'll make sure *not* to output negative energy value for sure! I mean to handle overflow in the emulation code, yes. > > > > - Read again the energy counter of each Package involved and calculate > > > the delta of the counter so that we know how much the counter has > > > increased during the Sleep period > > > - Calculate the ratio for each vCPU thread and deduct the energy spent > > > for each vCPU > > > - Calculate the amount of energy spent for all non-vCPU and evenly > > > spread it to the vCPU > > > - Update each virtual MSR for each vCPU > > > > > > Obviously, this is working better and more consistently with vCPU pinning > > > and proper isolation of the cores in the package. > > > > Do you really need to measure the consumption for QEMU threads? I'd say > > only measuring for the vcpus is sufficient. > > > > What if a process in the guest is actually doing a lot of IO ? > The QEMU workers managing those IO would be consumming, no ? > But if this is unsignicant I can remove it. On a second thought, i am not sure. Perhaps makes sense to keep the non-vcpu threads. > > > > So those virtual MSRs are updated roughly each second (could be changed > > > by updating MSR_ENERGY_THREAD_SLEEP_US). Compared to the real MSRs which > > > are updated by the microcode every 1ms. > > > > How useful are the measurements for the guest, then? (I mean, doesnt a > > 1 second interval render the measurements less useful). > > > > We can always reduce the sleeping time. > I guess we can also set it with a parameter like: > -accel kvm,rapl=true,update-period=500 > > with update-period, the interval in milliseconds between updates. And that would increase the CPU time for emulation significantly (for higher granularity the consumption goes way up). Which perhaps limits the use-cases. I think what would be useful is explanation, on a document, of the behaviour of MSR emulation in QEMU (how MSR value is emulated, low update rate, etc). Including some example measurements. Also consumption of energy by threads does not map 1<->1 to time spent executing (different instructions have different consumption). > > > Concerning the "real" power consumption, we have to not forget that the > > > RAPL interface energy data is not the result of a physical measurement. > > > It is a set of architectural events from each core, processor > > > graphic, IO, etc. and combines them with energy weights to predict the > > > package's active power consumption. > > > > > > IMO it is not really important because the idea behind this patch is to > > > give estimated values to the guests so that software running inside > > > VM can make use of power tools which are all reading those MSRs (or the > > > RAPL driver sys interface) to retrieve power consumption. > > > > Can you describe some use-cases... Because what seems to be useful > > (actually, what seem to be possible) is for users to measure the > > energy consumption of a package. > > > > Yes and here they will do the same. Except that for the moment I don't > have any solution to overcome the fact that each vcpu will have his own > package. > When a software will read MSR_PKG_ENERGY_STATUS of pkg-0, it will also > read the energy of the vcpu-0 because we are in single vcpu per virtual > socket. > I would say it might give better granularity of the power consumption: > If a process in guest is running only on one isolated vcpu, reading the > package MSR of this vcpu would theorically gives the consumption of this > process directly. Whereas in baremetal, we need to take the other cores > of the package into account. > > > So if you think, OK i can read the energy consumption information > > from within a guest, but i know its the energy consumption divided > > by the amount of time the qemu threads executed, how useful that > > measure is ? > > It gives what the guest has actually consume This is not necessarily true due to per-instruction consumption. So a vcpu executing the same sequence of instructions, in a package, sharing a core with a thread executing power hungry instructions, will have exposed consumption different than if that power hungry thread is not executing. I think AVX512 instructions are a good example of power hungry instructions. > and inside the guest there > are processes that are running. > There are many tools [1][2][3] that use thoses MSRs for metrics to calculate > the power consumption of applications/processes in order to optimize > their code. > > What I'm trying to achieve here is enabling this possibilities inside VM. OK. > > > > > Why not expose the actual delta from the MSRs, rather than diving > > by amount of time the qemu threads execute. > > > > Imagine 2 identical VMs on the same package: > VM-1 is hogging cores and VM-2 is doing nothing. > If we only report delta of the MSRs, both VM would consume the same > which is wrong. Yeah. So having the emulation described in a document allows users to know what to expect. > This is why I'm doing ratio of the energy depending on the QEMU threads. > because a vCPU is just a thread and the /proc/<pid>/stat can give us > the amount of time spend by each thread on the cores. > > Thanks a lot for your input, > > Anthony Thanks! This is a nice investigation into RAPL: https://helda.helsinki.fi/bitstream/handle/10138/321707/RAPL_in_Action_Experiences_in_Using_RAPL_for_Power_Measurements.pdf?sequence=1 > > > > [1]: https://github.com/sustainable-computing-io/kepler > [2]: https://powerapi.org/ > [3]: https://github.com/hubblo-org/scaphandre > > > > > > > > + > > > > > + /* Delta of ticks spend by each thread between the sample */ > > > > > + for (int i = 0; i < num_threads; i++) { > > > > > + if (read_thread_stat(&thd_stat[i], pid, 1) != 0) { > > > > > + /* > > > > > + * We don't count the dead thread > > > > > + * i.e threads that existed before the sleep > > > > > + * and not anymore > > > > > + */ > > > > > + thd_stat[i].delta_ticks = 0; > > > > > + } else { > > > > > + delta_ticks(thd_stat, i); > > > > > + } > > > > > + } > > > > > + > > > > > + /* > > > > > + * Identify the vCPU threads > > > > > + * Calculate the Number of vCPU per package > > > > > + */ > > > > > + CPU_FOREACH(cpu) { > > > > > + for (int i = 0; i < num_threads; i++) { > > > > > + if (cpu->thread_id == thd_stat[i].thread_id) { > > > > > + thd_stat[i].is_vcpu = true; > > > > > + thd_stat[i].vcpu_id = cpu->cpu_index; > > > > > + pkg_stat[thd_stat[i].numa_node_id].nb_vcpu++; > > > > > + break; > > > > > + } > > > > > + } > > > > > + } > > > > > + > > > > > + /* Calculate the total energy of all non-vCPU thread */ > > > > > + for (int i = 0; i < num_threads; i++) { > > > > > + double temp; > > > > > + if ((thd_stat[i].is_vcpu != true) && > > > > > + (thd_stat[i].delta_ticks > 0)) { > > > > > + temp = get_ratio(pkg_stat, thd_stat, maxticks, i); > > > > > + pkg_stat[thd_stat[i].numa_node_id].e_ratio > > > > > + += (uint64_t)lround(temp); > > > > > + } > > > > > + } > > > > > + > > > > > + /* Calculate the ratio per non-vCPU thread of each package */ > > > > > + for (int i = 0; i <= maxpkgs; i++) { > > > > > + if (pkg_stat[i].nb_vcpu > 0) { > > > > > + pkg_stat[i].e_ratio = pkg_stat[i].e_ratio / > > > > > pkg_stat[i].nb_vcpu; > > > > > + } > > > > > + } > > > > > + > > > > > + /* Calculate the energy for each vCPU thread */ > > > > > + for (int i = 0; i < num_threads; i++) { > > > > > + double temp; > > > > > + > > > > > + if ((thd_stat[i].is_vcpu == true) && > > > > > + (thd_stat[i].delta_ticks > 0)) { > > > > > + temp = get_ratio(pkg_stat, thd_stat, maxticks, i); > > > > > + vmsr->msr_value[thd_stat[i].vcpu_id] += > > > > > (uint64_t)lround(temp); > > > > > + vmsr->msr_value[thd_stat[i].vcpu_id] \ > > > > > + += pkg_stat[thd_stat[i].numa_node_id].e_ratio; > > > > > + } > > > > > + } > > > > > + > > > > > + /* free all memory */ > > > > > + for (int i = 0; i < num_threads; i++) { > > > > > + free(thd_stat[i].utime); > > > > > + free(thd_stat[i].stime); > > > > > + } > > > > > + free(thd_stat); > > > > > + free(thread_ids); > > > > > + } > > > > > + > > > > > + rcu_unregister_thread(); > > > > > + return NULL; > > > > > +} > > > > > + > > > > > +static int kvm_msr_energy_thread_init(KVMState *s, MachineState *ms) > > > > > +{ > > > > > + struct KVMMsrEnergy *r = &s->msr_energy; > > > > > + > > > > > + /* Retrieve the number of vCPU */ > > > > > + r->cpus = ms->smp.cpus; > > > > > + > > > > > + /* Allocate register memory (MSR_PKG_STATUS) for each vCPU */ > > > > > + r->msr_value = calloc(r->cpus, sizeof(r->msr_value)); > > > > > + > > > > > + qemu_thread_create(&r->msr_thr, "kvm-msr", > > > > > + kvm_msr_energy_thread, > > > > > + s, QEMU_THREAD_JOINABLE); > > > > > + > > > > > + return 0; > > > > > +} > > > > > + > > > > > int kvm_arch_init(MachineState *ms, KVMState *s) > > > > > { > > > > > uint64_t identity_base = 0xfffbc000; > > > > > @@ -2765,6 +2998,46 @@ int kvm_arch_init(MachineState *ms, KVMState > > > > > *s) > > > > > strerror(-ret)); > > > > > exit(1); > > > > > } > > > > > + > > > > > + if (s->msr_energy.enable == true) { > > > > > + > > > > > + r = kvm_filter_msr(s, MSR_RAPL_POWER_UNIT, > > > > > + kvm_rdmsr_rapl_power_unit, NULL); > > > > > + if (!r) { > > > > > + error_report("Could not install MSR_RAPL_POWER_UNIT \ > > > > > + handler: %s", > > > > > + strerror(-ret)); > > > > > + exit(1); > > > > > + } > > > > > + > > > > > + r = kvm_filter_msr(s, MSR_PKG_POWER_LIMIT, > > > > > + kvm_rdmsr_pkg_power_limit, NULL); > > > > > + if (!r) { > > > > > + error_report("Could not install MSR_PKG_POWER_LIMIT \ > > > > > + handler: %s", > > > > > + strerror(-ret)); > > > > > + exit(1); > > > > > + } > > > > > + > > > > > + r = kvm_filter_msr(s, MSR_PKG_POWER_INFO, > > > > > + kvm_rdmsr_pkg_power_info, NULL); > > > > > + if (!r) { > > > > > + error_report("Could not install MSR_PKG_POWER_INFO \ > > > > > + handler: %s", > > > > > + strerror(-ret)); > > > > > + exit(1); > > > > > + } > > > > > + r = kvm_filter_msr(s, MSR_PKG_ENERGY_STATUS, > > > > > + kvm_rdmsr_pkg_energy_status, NULL); > > > > > + if (!r) { > > > > > + error_report("Could not install > > > > > MSR_PKG_ENERGY_STATUS \ > > > > > + handler: %s", > > > > > + strerror(-ret)); > > > > > + exit(1); > > > > > + } else { > > > > > + kvm_msr_energy_thread_init(s, ms); > > > > > + } > > > > > + } > > > > > } > > > > > > > > > > return 0; > > > > > diff --git a/target/i386/kvm/meson.build b/target/i386/kvm/meson.build > > > > > index 322272091bce..9cdc93c6c439 100644 > > > > > --- a/target/i386/kvm/meson.build > > > > > +++ b/target/i386/kvm/meson.build > > > > > @@ -5,6 +5,7 @@ i386_softmmu_kvm_ss = ss.source_set() > > > > > i386_softmmu_kvm_ss.add(files( > > > > > 'kvm.c', > > > > > 'kvm-cpu.c', > > > > > + 'vmsr_energy.c', > > > > > )) > > > > > > > > > > i386_softmmu_kvm_ss.add(when: 'CONFIG_XEN_EMU', if_true: > > > > > files('xen-emu.c')) > > > > > diff --git a/target/i386/kvm/vmsr_energy.c > > > > > b/target/i386/kvm/vmsr_energy.c > > > > > new file mode 100644 > > > > > index 000000000000..8bd86b32becf > > > > > --- /dev/null > > > > > +++ b/target/i386/kvm/vmsr_energy.c > > > > > @@ -0,0 +1,132 @@ > > > > > +/* > > > > > + * QEMU KVM support -- x86 virtual energy-related MSR. > > > > > + * > > > > > + * Copyright 2023 Red Hat, Inc. 2023 > > > > > + * > > > > > + * Author: > > > > > + * Anthony Harivel <ahari...@redhat.com> > > > > > + * > > > > > + * This work is licensed under the terms of the GNU GPL, version 2 > > > > > or later. > > > > > + * See the COPYING file in the top-level directory. > > > > > + * > > > > > + */ > > > > > + > > > > > +#include "vmsr_energy.h" > > > > > + > > > > > +#define MAX_PATH_LEN 50 > > > > > +#define MAX_LINE_LEN 500 > > > > > + > > > > > +uint64_t read_msr(uint32_t reg, unsigned int cpu_id) > > > > > +{ > > > > > + int fd; > > > > > + uint64_t data; > > > > > + > > > > > + char path[MAX_PATH_LEN]; > > > > > + snprintf(path, MAX_PATH_LEN, "/dev/cpu/%u/msr", cpu_id); > > > > > + > > > > > + fd = open(path , O_RDONLY); > > > > > + if (fd < 0) { > > > > > + return 0; > > > > > + } > > > > > + if (pread(fd, &data, sizeof data, reg) != sizeof data) { > > > > > + data = 0; > > > > > + } > > > > > + > > > > > + close(fd); > > > > > + return data; > > > > > +} > > > > > + > > > > > +/* Retrieve the number of physical CPU on the package */ > > > > > +unsigned int get_maxcpus(unsigned int package_num) > > > > > +{ > > > > > + int k, ncpus; > > > > > + unsigned int maxcpus; > > > > > + struct bitmask *cpus; > > > > > + > > > > > + cpus = numa_allocate_cpumask(); > > > > > + ncpus = cpus->size; > > > > > + > > > > > + if (numa_node_to_cpus(package_num, cpus) < 0) { > > > > > + printf("node %u failed to convert\n", package_num); > > > > > + } > > > > > + > > > > > + maxcpus = 0; > > > > > + for (k = 0; k < ncpus; k++) { > > > > > + if (numa_bitmask_isbitset(cpus, k)) { > > > > > + maxcpus++; > > > > > + } > > > > > + } > > > > > + > > > > > + return maxcpus; > > > > > +} > > > > > + > > > > > +int read_thread_stat(struct thread_stat *thread, int pid, int index) > > > > > +{ > > > > > + char path[MAX_PATH_LEN]; > > > > > + snprintf(path, MAX_PATH_LEN, "/proc/%u/task/%d/stat", pid, \ > > > > > + thread->thread_id); > > > > > + > > > > > + FILE *file = fopen(path, "r"); > > > > > + if (file == NULL) { > > > > > + return -1; > > > > > + } > > > > > + > > > > > + if (fscanf(file, "%*d (%*[^)]) %*c %*d %*d %*d %*d %*d %*u %*u > > > > > %*u %*u %*u" > > > > > + " %llu %llu %*d %*d %*d %*d %*d %*d %*u %*u %*d %*u %*u" > > > > > + " %*u %*u %*u %*u %*u %*u %*u %*u %*u %*d %*u %*u %u", > > > > > + &thread->utime[index], &thread->stime[index], > > > > > &thread->cpu_id) != 3) > > > > > + return -1; > > > > > + > > > > > + fclose(file); > > > > > + return 0; > > > > > +} > > > > > + > > > > > +/* Read QEMU stat task folder to retrieve all QEMU threads ID */ > > > > > +pid_t *get_thread_ids(pid_t pid, int *num_threads) > > > > > +{ > > > > > + char path[100]; > > > > > + sprintf(path, "/proc/%d/task", pid); > > > > > + > > > > > + DIR *dir = opendir(path); > > > > > + if (dir == NULL) { > > > > > + perror("opendir"); > > > > > + return NULL; > > > > > + } > > > > > + > > > > > + pid_t *thread_ids = NULL; > > > > > + int thread_count = 0; > > > > > + > > > > > + struct dirent *ent; > > > > > + while ((ent = readdir(dir)) != NULL) { > > > > > + if (ent->d_name[0] == '.') { > > > > > + continue; > > > > > + } > > > > > + pid_t tid = atoi(ent->d_name); > > > > > + if (pid != tid) { > > > > > + thread_ids = realloc(thread_ids, > > > > > + (thread_count + 1) * sizeof(pid_t)); > > > > > + thread_ids[thread_count] = tid; > > > > > + thread_count++; > > > > > + } > > > > > + } > > > > > + > > > > > + closedir(dir); > > > > > + > > > > > + *num_threads = thread_count; > > > > > + return thread_ids; > > > > > +} > > > > > + > > > > > +void delta_ticks(thread_stat *thd_stat, int i) > > > > > +{ > > > > > + thd_stat[i].delta_ticks = (thd_stat[i].utime[1] + > > > > > thd_stat[i].stime[1]) > > > > > + - (thd_stat[i].utime[0] + > > > > > thd_stat[i].stime[0]); > > > > > +} > > > > > + > > > > > +double get_ratio(package_energy_stat *pkg_stat, > > > > > + thread_stat *thd_stat, > > > > > + int maxticks, int i) { > > > > > + > > > > > + return (pkg_stat[thd_stat[i].numa_node_id].e_delta / 100.0) > > > > > + * ((100.0 / maxticks) * thd_stat[i].delta_ticks); > > > > > +} > > > > > + > > > > > diff --git a/target/i386/kvm/vmsr_energy.h > > > > > b/target/i386/kvm/vmsr_energy.h > > > > > new file mode 100644 > > > > > index 000000000000..5f79d2cbe00d > > > > > --- /dev/null > > > > > +++ b/target/i386/kvm/vmsr_energy.h > > > > > @@ -0,0 +1,80 @@ > > > > > +/* > > > > > + * QEMU KVM support -- x86 virtual energy-related MSR. > > > > > + * > > > > > + * Copyright 2023 Red Hat, Inc. 2023 > > > > > + * > > > > > + * Author: > > > > > + * Anthony Harivel <ahari...@redhat.com> > > > > > + * > > > > > + * This work is licensed under the terms of the GNU GPL, version 2 > > > > > or later. > > > > > + * See the COPYING file in the top-level directory. > > > > > + * > > > > > + */ > > > > > + > > > > > +#ifndef VMSR_ENERGY_H > > > > > +#define VMSR_ENERGY_H > > > > > + > > > > > +#include "qemu/osdep.h" > > > > > + > > > > > +#include <numa.h> > > > > > + > > > > > +/* > > > > > + * Define the interval time in micro seconds between 2 samples of > > > > > + * energy related MSRs > > > > > + */ > > > > > +#define MSR_ENERGY_THREAD_SLEEP_US 1000000.0 > > > > > + > > > > > +/* > > > > > + * Thread statistic > > > > > + * @ thread_id: TID (thread ID) > > > > > + * @ is_vcpu: true is thread is vCPU thread > > > > > + * @ cpu_id: CPU number last executed on > > > > > + * @ vcpu_id: vCPU ID > > > > > + * @ numa_node_id:node number of the CPU > > > > > + * @ utime: amount of clock ticks the thread > > > > > + * has been scheduled in User mode > > > > > + * @ stime: amount of clock ticks the thread > > > > > + * has been scheduled in System mode > > > > > + * @ delta_ticks: delta of utime+stime between > > > > > + * the two samples (before/after sleep) > > > > > + */ > > > > > +struct thread_stat { > > > > > + unsigned int thread_id; > > > > > + bool is_vcpu; > > > > > + unsigned int cpu_id; > > > > > + unsigned int vcpu_id; > > > > > + unsigned int numa_node_id; > > > > > + unsigned long long *utime; > > > > > + unsigned long long *stime; > > > > > + unsigned long long delta_ticks; > > > > > +}; > > > > > + > > > > > +/* > > > > > + * Package statistic > > > > > + * @ e_start: package energy counter before the sleep > > > > > + * @ e_end: package energy counter after the sleep > > > > > + * @ e_delta: delta of package energy counter > > > > > + * @ e_ratio: store the energy ratio of non-vCPU thread > > > > > + * @ nb_vcpu: number of vCPU running on this package > > > > > + */ > > > > > +struct packge_energy_stat { > > > > > + uint64_t e_start; > > > > > + uint64_t e_end; > > > > > + uint64_t e_delta; > > > > > + uint64_t e_ratio; > > > > > + unsigned int nb_vcpu; > > > > > +}; > > > > > + > > > > > +typedef struct thread_stat thread_stat; > > > > > +typedef struct packge_energy_stat package_energy_stat; > > > > > + > > > > > +uint64_t read_msr(uint32_t reg, unsigned int cpu_id); > > > > > +void delta_ticks(thread_stat *thd_stat, int i); > > > > > +unsigned int get_maxcpus(unsigned int package_num); > > > > > +int read_thread_stat(struct thread_stat *thread, int pid, int index); > > > > > +pid_t *get_thread_ids(pid_t pid, int *num_threads); > > > > > +double get_ratio(package_energy_stat *pkg_stat, > > > > > + thread_stat *thd_stat, > > > > > + int maxticks, int i); > > > > > + > > > > > +#endif /* VMSR_ENERGY_H */ > > > > > --=2