On Wed, May 24, 2023 at 04:53:49PM +0200, Anthony Harivel wrote:
> set=UTF-8
> Status: RO
> Content-Length: 24102
> Lines: 667
> 
> Marcelo Tosatti, May 19, 2023 at 20:28:
> 
> Hi Marcelo,
> 
> > > > > +    /* Assuming those values are the same accross physical 
> > > > > system/packages */
> > > > > +    maxcpus = get_maxcpus(0); /* Number of CPUS per packages */
> > > > > +    maxpkgs = numa_max_node(); /* Number of Packages on the system */
> >
> > numa_max_node() returns the highest node number available on the current 
> > system. 
> > (See the node numbers in /sys/devices/system/node/ ). Also see 
> > numa_num_configured_nodes().
> >
> > One can find package topology information from
> > /sys/devices/system/cpu/cpuX/topology/
> >
> 
> Good point. 
> I will find a better solution to identify the topology using your hint. 
> 
>  > > > +        /* Allocate memory for each thread stats */
> > > > > +        thd_stat = (thread_stat *) calloc(num_threads, 
> > > > > sizeof(thread_stat));
> >
> > Can you keep this pre-allocated ? And all other data as well.
> 
> Ok no problem.
> 
>  > > > +        /* Retrieve all packages power plane energy counter */
> > > > > +        for (int i = 0; i <= maxpkgs; i++) {
> > > > > +            for (int j = 0; j < num_threads; j++) {
> > > > > +                /*
> > > > > +                 * Use the first thread we found that ran on the CPU
> > > > > +                 * of the package to read the packages energy counter
> > > > > +                 */
> > > > > +                if (thd_stat[j].numa_node_id == i) {
> > > > > +                    pkg_stat[i].e_start = 
> > > > > read_msr(MSR_PKG_ENERGY_STATUS, i);
> > > > > +                    break;
> > > > > +                }
> > > > > +            }
> > > > > +        }
> >
> > NUMA node does not map necessarily to one package.
> 
> True. I will update this part at the same time with the topology info
> discussed above. 
> 
> >
> > > > > +        /* Sleep a short period while the other threads are working 
> > > > > */
> > > > > +        usleep(MSR_ENERGY_THREAD_SLEEP_US);
> > > > > +
> > > > > +        /*
> > > > > +         * Retrieve all packages power plane energy counter
> > > > > +         * Calculate the delta of all packages
> > > > > +         */
> > > > > +        for (int i = 0; i <= maxpkgs; i++) {
> > > > > +            for (int j = 0; j < num_threads; j++) {
> > > > > +                /*
> > > > > +                 * Use the first thread we found that ran on the CPU
> > > > > +                 * of the package to read the packages energy counter
> > > > > +                 */
> > > > > +                if (thd_stat[j].numa_node_id == i) {
> > > > > +                    pkg_stat[i].e_end =
> > > > > +                        read_msr(MSR_PKG_ENERGY_STATUS, 
> > > > > thd_stat[j].cpu_id);
> >
> > This is excessive (to read the MSRs of each package in the system).
> >
> > Consider 100 Linux guests all of them with this enabled, on a system with
> > 4 packages. How many times you'll be reading MSR of each package?
> 
> The problem here is that you can have vCPUs that are running on different
> packages. However the energy counter of the different packages are
> increasing independently. 
> Either we "force" somehow users to run only on the same package, either I'm
> afraid we are obliged to read all the packages energy counter (when they
> are involved in the VM).
> 
> Imagine this:
> 
> |----pkg-0----|----pkg-1----|
> |0|1|2|3|4|5|6|0|1|2|3|4|5|6|
> |       |       |       |
> | vm-0  |  vm-1 |  vm-2 |
> 
> Only vm-1 that has cores from pkg-0 and pkg-1 would have to read both
> pkg energy. vm-0 would only read pkg-0 and vm-2 only pkg-1.
> 
> 
> >
> > Moreover, don't want to readmsr on an isolated CPU.
> >
> 
> Could you explain me why ?

Nevermind, its a separate topic.

> > > No problem, let me try to explain: 
> > > a QEMU process is composed of vCPU thread(s) and non-vCPU thread(s) (IO,
> > > emulated device,...). Each of those threads can run on different cores
> > > that can belongs to the same Package or not.
> > > The MSR_PKG_ENERGY_STATUS is a counter that increment for the whole
> > > Package domain. If you read this MSR from any core that belongs to the 
> > > package, you will retrieve the same number.
> > > So when I need to read the MSR, I only need to read once for all the
> > > threads that are running on cores of each Package.
> >
> > T=0 read p1v0 (== MSR_PKG_ENERGY_STATUS)
> > T=1 vcpu-1 executing in core1
> >     vcpu-2 executing in core2
> >     vcpu-3 executing in core2
> > T=2 read p1v1
> >
> > Won't you be exposing (p1v1-p1v0)/3 by 3 in this case, to the 
> > virtual MSR_PKG_ENERGY_STATUS?
> >
> 
> No, because if we take for exemple a 4 cores per package with 100 ticks per 
> second,
> the maximum number of ticks for all the cores in this packages would be
> 400 ticks per second. 
> So if 2 or more vcpus are sharing the same core, they will be maxout by the
> 100 ticks per second anyway and so ratio will still be ok. 
> 
> 
> > > Now let's talk about the implementation of the emulated value. 
> > > I've created a thread that does in an infinite loop the following:
> > > - Retrieve all the QEMU threads + statistics about them 
> > > - Read the energy counter of each Package involved 
> > > - Sleep for 1sec (1sec is arbitrary)
> > > - Calculate the delta of ticks for each threads so that we know how much
> > >   time each threads has been scheduled during the Sleep period
> >
> > Intel docs mention the counter can overflow (faster with higher 
> > energy consumption). Are you handling the overflow case?
> >
> 
> Should we ? On baremetal, anyone reading the MSRs should handle the
> overflow case, I guess on a guest this should be the same.
> But I'll make sure *not* to output negative energy value for sure!

I mean to handle overflow in the emulation code, yes.

> 
> > > - Read again the energy counter of each Package involved and calculate
> > >   the delta of the counter so that we know how much the counter has
> > >   increased during the Sleep period
> > > - Calculate the ratio for each vCPU thread and deduct the energy spent
> > >   for each vCPU
> > > - Calculate the amount of energy spent for all non-vCPU and evenly
> > >   spread it to the vCPU
> > > - Update each virtual MSR for each vCPU
> > > 
> > > Obviously, this is working better and more consistently with vCPU pinning 
> > > and proper isolation of the cores in the package. 
> >
> > Do you really need to measure the consumption for QEMU threads? I'd say
> > only measuring for the vcpus is sufficient.
> >
> 
> What if a process in the guest is actually doing a lot of IO ?
> The QEMU workers managing those IO would be consumming, no ?
> But if this is unsignicant I can remove it.

On a second thought, i am not sure. Perhaps makes sense to keep 
the non-vcpu threads.

> 
> > > So those virtual MSRs are updated roughly each second (could be changed
> > > by updating MSR_ENERGY_THREAD_SLEEP_US). Compared to the real MSRs which 
> > > are updated by the microcode every 1ms.
> >
> > How useful are the measurements for the guest, then? (I mean, doesnt a
> > 1 second interval render the measurements less useful).
> >
> 
> We can always reduce the sleeping time. 
> I guess we can also set it with a parameter like:
> -accel kvm,rapl=true,update-period=500
> 
> with update-period, the interval in milliseconds between updates. 

And that would increase the CPU time for emulation significantly
(for higher granularity the consumption goes way up).
Which perhaps limits the use-cases.

I think what would be useful is explanation, on a document, of the behaviour of 
MSR emulation
in QEMU (how MSR value is emulated, low update rate, etc).
Including some example measurements.

Also consumption of energy by threads does not map 1<->1 to time spent
executing (different instructions have different consumption).

> > > Concerning the "real" power consumption, we have to not forget that the
> > > RAPL interface energy data is not the result of a physical measurement. 
> > > It is a set of architectural events from each core, processor
> > > graphic, IO, etc. and combines them with energy weights to predict the
> > > package's active power consumption.
> > > 
> > > IMO it is not really important because the idea behind this patch is to
> > > give estimated values to the guests so that software running inside
> > > VM can make use of power tools which are all reading those MSRs (or the
> > > RAPL driver sys interface) to retrieve power consumption.
> >
> > Can you describe some use-cases... Because what seems to be useful
> > (actually, what seem to be possible) is for users to measure the 
> > energy consumption of a package. 
> >
> 
> Yes and here they will do the same. Except that for the moment I don't
> have any solution to overcome the fact that each vcpu will have his own
> package. 
> When a software will read MSR_PKG_ENERGY_STATUS of pkg-0, it will also
> read the energy of the vcpu-0 because we are in single vcpu per virtual
> socket.
> I would say it might give better granularity of the power consumption:
> If a process in guest is running only on one isolated vcpu, reading the
> package MSR of this vcpu would theorically gives the consumption of this
> process directly. Whereas in baremetal, we need to take the other cores
> of the package into account.
> 
> > So if you think, OK i can read the energy consumption information
> > from within a guest, but i know its the energy consumption divided
> > by the amount of time the qemu threads executed, how useful that
> > measure is ?
> 
> It gives what the guest has actually consume 

This is not necessarily true due to per-instruction consumption.

So a vcpu executing the same sequence of instructions, in a package, 
sharing a core with a thread executing power hungry instructions, will
have exposed consumption different than if that power hungry thread is
not executing.

I think AVX512 instructions are a good example of power hungry
instructions.

> and inside the guest there
> are processes that are running.
> There are many tools [1][2][3] that use thoses MSRs for metrics to calculate
> the power consumption of applications/processes in order to optimize
> their code. 
> 
> What I'm trying to achieve here is enabling this possibilities inside VM. 

OK.

> 
> >
> > Why not expose the actual delta from the MSRs, rather than diving
> > by amount of time the qemu threads execute.
> >
> 
> Imagine 2 identical VMs on the same package: 
> VM-1 is hogging cores and VM-2 is doing nothing. 
> If we only report delta of the MSRs, both VM would consume the same
> which is wrong. 

Yeah. So having the emulation described in a document allows users to
know what to expect.

> This is why I'm doing ratio of the energy depending on the QEMU threads. 
> because a vCPU is just a thread and the /proc/<pid>/stat can give us 
> the amount of time spend by each thread on the cores.  
> 
> Thanks a lot for your input,
> 
> Anthony

Thanks!

This is a nice investigation into RAPL:
https://helda.helsinki.fi/bitstream/handle/10138/321707/RAPL_in_Action_Experiences_in_Using_RAPL_for_Power_Measurements.pdf?sequence=1

> 
> 
> 
> [1]: https://github.com/sustainable-computing-io/kepler
> [2]: https://powerapi.org/
> [3]: https://github.com/hubblo-org/scaphandre
> 
> 
> 
> > > > > +
> > > > > +        /* Delta of ticks spend by each thread between the sample */
> > > > > +        for (int i = 0; i < num_threads; i++) {
> > > > > +            if (read_thread_stat(&thd_stat[i], pid, 1) != 0) {
> > > > > +                /*
> > > > > +                 * We don't count the dead thread
> > > > > +                 * i.e threads that existed before the sleep
> > > > > +                 * and not anymore
> > > > > +                 */
> > > > > +                thd_stat[i].delta_ticks = 0;
> > > > > +            } else {
> > > > > +                delta_ticks(thd_stat, i);
> > > > > +            }
> > > > > +        }
> > > > > +
> > > > > +        /*
> > > > > +         * Identify the vCPU threads
> > > > > +         * Calculate the Number of vCPU per package
> > > > > +         */
> > > > > +        CPU_FOREACH(cpu) {
> > > > > +            for (int i = 0; i < num_threads; i++) {
> > > > > +                if (cpu->thread_id == thd_stat[i].thread_id) {
> > > > > +                    thd_stat[i].is_vcpu = true;
> > > > > +                    thd_stat[i].vcpu_id = cpu->cpu_index;
> > > > > +                    pkg_stat[thd_stat[i].numa_node_id].nb_vcpu++;
> > > > > +                    break;
> > > > > +                }
> > > > > +            }
> > > > > +        }
> > > > > +
> > > > > +        /* Calculate the total energy of all non-vCPU thread */
> > > > > +        for (int i = 0; i < num_threads; i++) {
> > > > > +            double temp;
> > > > > +            if ((thd_stat[i].is_vcpu != true) &&
> > > > > +                (thd_stat[i].delta_ticks > 0)) {
> > > > > +                temp = get_ratio(pkg_stat, thd_stat, maxticks, i);
> > > > > +                pkg_stat[thd_stat[i].numa_node_id].e_ratio
> > > > > +                    += (uint64_t)lround(temp);
> > > > > +            }
> > > > > +        }
> > > > > +
> > > > > +        /* Calculate the ratio per non-vCPU thread of each package */
> > > > > +        for (int i = 0; i <= maxpkgs; i++) {
> > > > > +            if (pkg_stat[i].nb_vcpu > 0) {
> > > > > +                pkg_stat[i].e_ratio = pkg_stat[i].e_ratio / 
> > > > > pkg_stat[i].nb_vcpu;
> > > > > +            }
> > > > > +        }
> > > > > +
> > > > > +        /* Calculate the energy for each vCPU thread */
> > > > > +        for (int i = 0; i < num_threads; i++) {
> > > > > +            double temp;
> > > > > +
> > > > > +            if ((thd_stat[i].is_vcpu == true) &&
> > > > > +                (thd_stat[i].delta_ticks > 0)) {
> > > > > +                temp = get_ratio(pkg_stat, thd_stat, maxticks, i);
> > > > > +                vmsr->msr_value[thd_stat[i].vcpu_id] += 
> > > > > (uint64_t)lround(temp);
> > > > > +                vmsr->msr_value[thd_stat[i].vcpu_id] \
> > > > > +                    += pkg_stat[thd_stat[i].numa_node_id].e_ratio;
> > > > > +            }
> > > > > +        }
> > > > > +
> > > > > +        /* free all memory */
> > > > > +        for (int i = 0; i < num_threads; i++) {
> > > > > +            free(thd_stat[i].utime);
> > > > > +            free(thd_stat[i].stime);
> > > > > +        }
> > > > > +        free(thd_stat);
> > > > > +        free(thread_ids);
> > > > > +    }
> > > > > +
> > > > > +    rcu_unregister_thread();
> > > > > +    return NULL;
> > > > > +}
> > > > > +
> > > > > +static int kvm_msr_energy_thread_init(KVMState *s, MachineState *ms)
> > > > > +{
> > > > > +    struct KVMMsrEnergy *r = &s->msr_energy;
> > > > > +
> > > > > +    /* Retrieve the number of vCPU */
> > > > > +    r->cpus = ms->smp.cpus;
> > > > > +
> > > > > +    /* Allocate register memory (MSR_PKG_STATUS) for each vCPU */
> > > > > +    r->msr_value = calloc(r->cpus, sizeof(r->msr_value));
> > > > > +
> > > > > +    qemu_thread_create(&r->msr_thr, "kvm-msr",
> > > > > +                       kvm_msr_energy_thread,
> > > > > +                       s, QEMU_THREAD_JOINABLE);
> > > > > +
> > > > > +    return 0;
> > > > > +}
> > > > > +
> > > > >  int kvm_arch_init(MachineState *ms, KVMState *s)
> > > > >  {
> > > > >      uint64_t identity_base = 0xfffbc000;
> > > > > @@ -2765,6 +2998,46 @@ int kvm_arch_init(MachineState *ms, KVMState 
> > > > > *s)
> > > > >                           strerror(-ret));
> > > > >              exit(1);
> > > > >          }
> > > > > +
> > > > > +        if (s->msr_energy.enable == true) {
> > > > > +
> > > > > +            r = kvm_filter_msr(s, MSR_RAPL_POWER_UNIT,
> > > > > +                               kvm_rdmsr_rapl_power_unit, NULL);
> > > > > +            if (!r) {
> > > > > +                error_report("Could not install MSR_RAPL_POWER_UNIT \
> > > > > +                                handler: %s",
> > > > > +                             strerror(-ret));
> > > > > +                exit(1);
> > > > > +            }
> > > > > +
> > > > > +            r = kvm_filter_msr(s, MSR_PKG_POWER_LIMIT,
> > > > > +                               kvm_rdmsr_pkg_power_limit, NULL);
> > > > > +            if (!r) {
> > > > > +                error_report("Could not install MSR_PKG_POWER_LIMIT \
> > > > > +                                handler: %s",
> > > > > +                             strerror(-ret));
> > > > > +                exit(1);
> > > > > +            }
> > > > > +
> > > > > +            r = kvm_filter_msr(s, MSR_PKG_POWER_INFO,
> > > > > +                               kvm_rdmsr_pkg_power_info, NULL);
> > > > > +            if (!r) {
> > > > > +                error_report("Could not install MSR_PKG_POWER_INFO \
> > > > > +                                handler: %s",
> > > > > +                             strerror(-ret));
> > > > > +                exit(1);
> > > > > +            }
> > > > > +            r = kvm_filter_msr(s, MSR_PKG_ENERGY_STATUS,
> > > > > +                               kvm_rdmsr_pkg_energy_status, NULL);
> > > > > +            if (!r) {
> > > > > +                error_report("Could not install 
> > > > > MSR_PKG_ENERGY_STATUS \
> > > > > +                                handler: %s",
> > > > > +                             strerror(-ret));
> > > > > +                exit(1);
> > > > > +            } else {
> > > > > +                kvm_msr_energy_thread_init(s, ms);
> > > > > +            }
> > > > > +        }
> > > > >      }
> > > > >  
> > > > >      return 0;
> > > > > diff --git a/target/i386/kvm/meson.build b/target/i386/kvm/meson.build
> > > > > index 322272091bce..9cdc93c6c439 100644
> > > > > --- a/target/i386/kvm/meson.build
> > > > > +++ b/target/i386/kvm/meson.build
> > > > > @@ -5,6 +5,7 @@ i386_softmmu_kvm_ss = ss.source_set()
> > > > >  i386_softmmu_kvm_ss.add(files(
> > > > >    'kvm.c',
> > > > >    'kvm-cpu.c',
> > > > > +  'vmsr_energy.c',
> > > > >  ))
> > > > >  
> > > > >  i386_softmmu_kvm_ss.add(when: 'CONFIG_XEN_EMU', if_true: 
> > > > > files('xen-emu.c'))
> > > > > diff --git a/target/i386/kvm/vmsr_energy.c 
> > > > > b/target/i386/kvm/vmsr_energy.c
> > > > > new file mode 100644
> > > > > index 000000000000..8bd86b32becf
> > > > > --- /dev/null
> > > > > +++ b/target/i386/kvm/vmsr_energy.c
> > > > > @@ -0,0 +1,132 @@
> > > > > +/*
> > > > > + * QEMU KVM support -- x86 virtual energy-related MSR.
> > > > > + *
> > > > > + * Copyright 2023 Red Hat, Inc. 2023
> > > > > + *
> > > > > + *  Author:
> > > > > + *      Anthony Harivel <ahari...@redhat.com>
> > > > > + *
> > > > > + * This work is licensed under the terms of the GNU GPL, version 2 
> > > > > or later.
> > > > > + * See the COPYING file in the top-level directory.
> > > > > + *
> > > > > + */
> > > > > +
> > > > > +#include "vmsr_energy.h"
> > > > > +
> > > > > +#define MAX_PATH_LEN 50
> > > > > +#define MAX_LINE_LEN 500
> > > > > +
> > > > > +uint64_t read_msr(uint32_t reg, unsigned int cpu_id)
> > > > > +{
> > > > > +    int fd;
> > > > > +    uint64_t data;
> > > > > +
> > > > > +    char path[MAX_PATH_LEN];
> > > > > +    snprintf(path, MAX_PATH_LEN, "/dev/cpu/%u/msr", cpu_id);
> > > > > +
> > > > > +    fd = open(path , O_RDONLY);
> > > > > +    if (fd < 0) {
> > > > > +        return 0;
> > > > > +    }
> > > > > +    if (pread(fd, &data, sizeof data, reg) != sizeof data) {
> > > > > +        data = 0;
> > > > > +    }
> > > > > +
> > > > > +    close(fd);
> > > > > +    return data;
> > > > > +}
> > > > > +
> > > > > +/* Retrieve the number of physical CPU on the package */
> > > > > +unsigned int get_maxcpus(unsigned int package_num)
> > > > > +{
> > > > > +    int k, ncpus;
> > > > > +    unsigned int maxcpus;
> > > > > +    struct bitmask *cpus;
> > > > > +
> > > > > +    cpus = numa_allocate_cpumask();
> > > > > +    ncpus = cpus->size;
> > > > > +
> > > > > +    if (numa_node_to_cpus(package_num, cpus) < 0) {
> > > > > +        printf("node %u failed to convert\n", package_num);
> > > > > +    }
> > > > > +
> > > > > +    maxcpus = 0;
> > > > > +    for (k = 0; k < ncpus; k++) {
> > > > > +        if (numa_bitmask_isbitset(cpus, k)) {
> > > > > +            maxcpus++;
> > > > > +        }
> > > > > +    }
> > > > > +
> > > > > +    return maxcpus;
> > > > > +}
> > > > > +
> > > > > +int read_thread_stat(struct thread_stat *thread, int pid, int index)
> > > > > +{
> > > > > +    char path[MAX_PATH_LEN];
> > > > > +    snprintf(path, MAX_PATH_LEN, "/proc/%u/task/%d/stat", pid, \
> > > > > +             thread->thread_id);
> > > > > +
> > > > > +    FILE *file = fopen(path, "r");
> > > > > +    if (file == NULL) {
> > > > > +        return -1;
> > > > > +    }
> > > > > +
> > > > > +    if (fscanf(file, "%*d (%*[^)]) %*c %*d %*d %*d %*d %*d %*u %*u 
> > > > > %*u %*u %*u"
> > > > > +        " %llu %llu %*d %*d %*d %*d %*d %*d %*u %*u %*d %*u %*u"
> > > > > +        " %*u %*u %*u %*u %*u %*u %*u %*u %*u %*d %*u %*u %u",
> > > > > +           &thread->utime[index], &thread->stime[index], 
> > > > > &thread->cpu_id) != 3)
> > > > > +        return -1;
> > > > > +
> > > > > +    fclose(file);
> > > > > +    return 0;
> > > > > +}
> > > > > +
> > > > > +/* Read QEMU stat task folder to retrieve all QEMU threads ID */
> > > > > +pid_t *get_thread_ids(pid_t pid, int *num_threads)
> > > > > +{
> > > > > +    char path[100];
> > > > > +    sprintf(path, "/proc/%d/task", pid);
> > > > > +
> > > > > +    DIR *dir = opendir(path);
> > > > > +    if (dir == NULL) {
> > > > > +        perror("opendir");
> > > > > +        return NULL;
> > > > > +    }
> > > > > +
> > > > > +    pid_t *thread_ids = NULL;
> > > > > +    int thread_count = 0;
> > > > > +
> > > > > +    struct dirent *ent;
> > > > > +    while ((ent = readdir(dir)) != NULL) {
> > > > > +        if (ent->d_name[0] == '.') {
> > > > > +            continue;
> > > > > +        }
> > > > > +        pid_t tid = atoi(ent->d_name);
> > > > > +        if (pid != tid) {
> > > > > +            thread_ids = realloc(thread_ids,
> > > > > +                                 (thread_count + 1) * sizeof(pid_t));
> > > > > +            thread_ids[thread_count] = tid;
> > > > > +            thread_count++;
> > > > > +        }
> > > > > +    }
> > > > > +
> > > > > +    closedir(dir);
> > > > > +
> > > > > +    *num_threads = thread_count;
> > > > > +    return thread_ids;
> > > > > +}
> > > > > +
> > > > > +void delta_ticks(thread_stat *thd_stat, int i)
> > > > > +{
> > > > > +    thd_stat[i].delta_ticks = (thd_stat[i].utime[1] + 
> > > > > thd_stat[i].stime[1])
> > > > > +                            - (thd_stat[i].utime[0] + 
> > > > > thd_stat[i].stime[0]);
> > > > > +}
> > > > > +
> > > > > +double get_ratio(package_energy_stat *pkg_stat,
> > > > > +                        thread_stat *thd_stat,
> > > > > +                        int maxticks, int i) {
> > > > > +
> > > > > +    return (pkg_stat[thd_stat[i].numa_node_id].e_delta / 100.0)
> > > > > +            * ((100.0 / maxticks) * thd_stat[i].delta_ticks);
> > > > > +}
> > > > > +
> > > > > diff --git a/target/i386/kvm/vmsr_energy.h 
> > > > > b/target/i386/kvm/vmsr_energy.h
> > > > > new file mode 100644
> > > > > index 000000000000..5f79d2cbe00d
> > > > > --- /dev/null
> > > > > +++ b/target/i386/kvm/vmsr_energy.h
> > > > > @@ -0,0 +1,80 @@
> > > > > +/*
> > > > > + * QEMU KVM support -- x86 virtual energy-related MSR.
> > > > > + *
> > > > > + * Copyright 2023 Red Hat, Inc. 2023
> > > > > + *
> > > > > + *  Author:
> > > > > + *      Anthony Harivel <ahari...@redhat.com>
> > > > > + *
> > > > > + * This work is licensed under the terms of the GNU GPL, version 2 
> > > > > or later.
> > > > > + * See the COPYING file in the top-level directory.
> > > > > + *
> > > > > + */
> > > > > +
> > > > > +#ifndef VMSR_ENERGY_H
> > > > > +#define VMSR_ENERGY_H
> > > > > +
> > > > > +#include "qemu/osdep.h"
> > > > > +
> > > > > +#include <numa.h>
> > > > > +
> > > > > +/*
> > > > > + * Define the interval time in micro seconds between 2 samples of
> > > > > + * energy related MSRs
> > > > > + */
> > > > > +#define MSR_ENERGY_THREAD_SLEEP_US 1000000.0
> > > > > +
> > > > > +/*
> > > > > + * Thread statistic
> > > > > + * @ thread_id: TID (thread ID)
> > > > > + * @ is_vcpu: true is thread is vCPU thread
> > > > > + * @ cpu_id: CPU number last executed on
> > > > > + * @ vcpu_id: vCPU ID
> > > > > + * @ numa_node_id:node number of the CPU
> > > > > + * @ utime: amount of clock ticks the thread
> > > > > + *          has been scheduled in User mode
> > > > > + * @ stime: amount of clock ticks the thread
> > > > > + *          has been scheduled in System mode
> > > > > + * @ delta_ticks: delta of utime+stime between
> > > > > + *          the two samples (before/after sleep)
> > > > > + */
> > > > > +struct thread_stat {
> > > > > +    unsigned int thread_id;
> > > > > +    bool is_vcpu;
> > > > > +    unsigned int cpu_id;
> > > > > +    unsigned int vcpu_id;
> > > > > +    unsigned int numa_node_id;
> > > > > +    unsigned long long *utime;
> > > > > +    unsigned long long *stime;
> > > > > +    unsigned long long delta_ticks;
> > > > > +};
> > > > > +
> > > > > +/*
> > > > > + * Package statistic
> > > > > + * @ e_start: package energy counter before the sleep
> > > > > + * @ e_end: package energy counter after the sleep
> > > > > + * @ e_delta: delta of package energy counter
> > > > > + * @ e_ratio: store the energy ratio of non-vCPU thread
> > > > > + * @ nb_vcpu: number of vCPU running on this package
> > > > > + */
> > > > > +struct packge_energy_stat {
> > > > > +    uint64_t e_start;
> > > > > +    uint64_t e_end;
> > > > > +    uint64_t e_delta;
> > > > > +    uint64_t e_ratio;
> > > > > +    unsigned int nb_vcpu;
> > > > > +};
> > > > > +
> > > > > +typedef struct thread_stat thread_stat;
> > > > > +typedef struct packge_energy_stat package_energy_stat;
> > > > > +
> > > > > +uint64_t read_msr(uint32_t reg, unsigned int cpu_id);
> > > > > +void delta_ticks(thread_stat *thd_stat, int i);
> > > > > +unsigned int get_maxcpus(unsigned int package_num);
> > > > > +int read_thread_stat(struct thread_stat *thread, int pid, int index);
> > > > > +pid_t *get_thread_ids(pid_t pid, int *num_threads);
> > > > > +double get_ratio(package_energy_stat *pkg_stat,
> > > > > +                        thread_stat *thd_stat,
> > > > > +                        int maxticks, int i);
> > > > > +
> > > > > +#endif /* VMSR_ENERGY_H */
> > > > > --=2


Reply via email to