Subject: Re: riscv64 system emulation maximum core count with virt machine
Date: Fri, 12 Nov 2021 10:16:34 +0000
References:
<CADJL7eegP=UajgcOxcb=cs1qfim+w3wt0ieq_on6sizneu-...@mail.gmail.com>
<d5150ee4-3e9e-43f3-828d-630ca3673...@getmailspring.com>
<CADJL7ecO089N_1GC6GPY5SJmRbVh81i7hE=8nspzaeduxhb...@mail.gmail.com>
User-agent: mu4e 1.7.4; emacs 28.0.60
In-reply-to:
<CADJL7ecO089N_1GC6GPY5SJmRbVh81i7hE=8nspzaeduxhb...@mail.gmail.com>
Mark Wyse <wy...@cs.washington.edu> writes:
> Thanks Rahul,
>
> After some digging I came to the same conclusions. I'll dig deeper to see if
> there are any impacts in raising max CPU count for RISC-V Virt
> machine, but I definitely agree that it should be set higher.
If you look at the commit that added the #define:
18df0b4695 (hw/riscv: virt: Allow creating multiple NUMA sockets)
You'll see amongst other things the number of vCPUs affected the size of
the one of the regions (IRQ controller?):
[VIRT_RTC] = { 0x101000, 0x1000 },
[VIRT_CLINT] = { 0x2000000, 0x10000 },
[VIRT_PCIE_PIO] = { 0x3000000, 0x10000 },
- [VIRT_PLIC] = { 0xc000000, 0x4000000 },
+ [VIRT_PLIC] = { 0xc000000, VIRT_PLIC_SIZE(VIRT_CPUS_MAX * 2) },
[VIRT_UART0] = { 0x10000000, 0x100 },
[VIRT_VIRTIO] = { 0x10001000, 0x1000 },
which I suspects limits the total number you can handle without messing
around with the MMIO memory map.
> FWIW, I was able to run a 32-core kernel after adjusting code and kernel
> appropriately and don't recall any warning or error messages
> popping up.
I should also point out from a emulation performance point of view you
will run into diminishing returns as you increases the number of CPUs as
the emulator has to spend more time ensuring cross-vCPU consistency when
things like the TLBs are flushed.
>
> Best,
> Mark
>
> On Wed, Sep 29, 2021 at 10:16 AM Rahul Pathak <rpathakmail...@gmail.com>
> wrote:
>
> If you want to boot with more than 8 harts then you also need to increase
> the CONFIG_NR_CPUS in the kernel. Along with change in
> QEMU - VIRT_CPUS_MAX in "include/hw/riscv/virt.h"
>
> But, the limit of 8 is low and should be increased. Don't know the reason
> why its 8.
>
> On Sep 22 2021, at 10:19 pm, Mark Wyse <wy...@cs.washington.edu> wrote:
>
> Hello all,
>
> Apologies if I missed the answer in the docs, but I am wondering why the
> virt machine is limited to only 8 cores when running
> RISC-V 64 system emulation? I am interested in booting a 16, 24, or even 32
> core SMP Linux kernel with OpenSBI as the
> bootloader on the generic virt machine before adding a custom machine
> definition. I was able to successfully boot Linux kernel
> 5.9.0 with OpenSBI v0.9 and 8 cores/harts with an ext4 filesystem provided
> by Buildroot.
>
> I poked around the docs and source code a bit, and found mention of the core
> count limit in the docs
> (https://qemu.readthedocs.io/en/latest/system/riscv/virt.html), but no
> reason as to why this is the limit. It also appears that
> providing a custom device tree blob and setting QEMU's -smp option to match
> might overcome this limitation, but it would be
> better if more than 8 cores was supported directly.
>
> Best,
> Mark
>
> Mark Wyse
> pronouns: he/him/his
> PhD Student
> Paul G. Allen School of Computer Science & Engineering
> University of Washington
--
Alex Bennée
